** sp4si Script package for Suck & INN **
This script package is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version. The package is distributed in the hope that
it will be useful but WITHOUT ANY WARRANTY.
(c) 1998-2000 Peter Sobisch, All rights reserved.
Because of many questions at the begin and the because of nobody wants
to read long and tedious manual, I decided to write this as a question-answer
like document. This text is written as WWW-Page and is also the regular
manual (included to sp4si package).
- What is sp4si ?
- What does sp4si and what it doesn't ?
- Why and when I need sp4si ?
- What I need to use sp4si ?
- How does it works ?
- Syntax and functions
- configuration: env and newshosts
- active - scripts: get.active and rebuild.active
- spool - scripts: nntp.up, nntp.down, uucp.batch and
- subscription - scripts: subscribe and wwwsubscribe
- Installation script: install.sh
- Configuration: first steps
- inserting of local newsgroups
- self-maintainance of sp4si
- extended configuration
- sp4si and CNFS
- Where to get sp4si ?
is a script package, which makes the management with INN
much easier. It keeps the configuration level be increased
number of newshosts on the same level. sp4si
is originated because
of misery and is designed for everybody who uses a dial-up internet connection
and wants to read the news offline on his own local INN
or for everybody who just wants to expand his INN
In the beginning sp4si was just a small script for auto-subscribing
of newsgroups on the local INN system and it becames an full automatical
multihost- and multitaskingable spool tool for INN and suck
sp4si is able to spool asynchronous all configured news hosts by
running only one script, optional spool reports will be sent with the result
of the spool.
sp4si can be set to do automatical download of active files from
each configured newshost, filter the active files, generate a new
file and replace the old active file regarding INN considerations.
sp4si is able to rebuild the suck config files in dependence
of each newsgroups subscribed in user .newsrc file.
sp4si is a replacement for get.news and put.news from
package. It's also able to spool a UUCP via TCP (if available).
sp4si isn't a configuration tool for INN, so INN must
be configured prior to install sp4si. sp4si will make the daily
work much easier than before.
if you often subscribe and unsubscribe often many newsgroups
if you want to have more control and synopsis about your newssystem and
want to know if the last spool was successfully done.
if you want to simply expand your newssystem with more than one newshost
if your bandwidth to your newshosts is smaller as your connection speed,
you can spool all hosts parallel and save you time and money.
if your news provider offers news by UUCP via TCP
You will need the following packages:
traditional and api (CNFS) storage are supported (CNFS using UUCP is
not yet tested)
(you should download the latest version)
(of course :)
UUCP if required
HTTP daemon to be able to use CGI script if required (e.g: apache)
It requires a configured and running INN
newssystem (local configuration)
(if needed). To get more information how to configure INN
system easily you can read in the german (sorry) manual from Carsten Voss
eines News-Systems unter Linux"
. Furthermore you must have suck package
installed and the suck command must be in your PATH
(for example you can copy it to /usr/local/bin directory).
sp4si consists of some shell scripts which can be divided into three main
parts and two configuration files: newshosts
file contains the site- and hostnames which should be spooled. The env
file contains the environment variables and will be included on the runtime
by the most scripts of sp4si
. The first part of the sp4si does the
active file management, the second part spools the news articles and the
last part takes care of newsgroup's auto subscribing.
Almost all scripts will evaluate the newshosts file and use its entries.
This is an advantage, if you want to add a new newshost, you have only
to do this in the newshosts and newsfeeds (INN) files. Because of the centralized
environment, sp4si makes the management much easier and increases the productivity
of your newssystem. If you want to use the spool frontend spool.news (is
strong recommended), all spool activities will be logged and you can send
optionally spool reports with the spool result by email or post it to a
it will be created during installation. It will be placed in the main
directory of sp4si. It contains variables definitions to influence
the environment and control in central point. All options are commented.
this file will be also created during the installation and placed in
the main directory of sp4si. For each site exists one-line entry, each
item of an entry must be saparated by a colon. Comments can be introduced
by a '#' at begin of each line. An entry has a following syntax:
<site> - has the same name as the correspondent entry in
the newsfeeds file
<newshost> contains the host's DNS name to reach the
host on internet
[up/down] - (optional) decribes the spool type, valid entries
- nntp/nntp (or just nntp): upload and download of articles by NNTP
- nntp/uucp: upload by NNTP and download by UUCP
- uucp/uucp (or just uucp): upload and download by UUCP
- uucp/nntp: upload by UUCP and download by NNTP
If no up/down parameter was given "nntp" will be used by default.
[port] - (optional) some newsservers offer their news on a other
TCP port by default. You can set it here. If no port given the default
port (119) will be used.
[login] - (optional) some newsservers need a authentification,
insert your login name here.
[pass] - (optional) this is the password for your login name.
[packer] - is the packer for outgoing batches to be packed by:
or cat (unpacked)
[gupmail] - is the mail address of an account, where gup(1)
serves the subscriptions.
[guppass] - the password for your own site, configured by your
news provider, the site name will be extracted with uuname -l.
Please look at UUCP-HowTo to configure your own UUCP site.
All options described before are commented in newshosts file too.
will be created during the installation and will be placed in the site
directory. Enter your local newsgroups regarding the syntax of the active(5)
file. These newsgroups will be regarded by the next run of rebuild.active
creating a global active file.
will be created during the installation and will be placed in the site
directory. Enter here the description for your local newsgroups as they
are in the newsgroups file. The file is not yet used.
Almost all scripts allow to invoke them giving the -h
parameters to show the syntax and online help.
Syntax: get.active [ -n ]
it downloads the active files from all newshosts which are in the newshosts
file and saves them to <site.active> files in the site subdirectory
(see env file). Additional the newsgroups description file could
be downloaded, up to now htere is no support for evalutae this file by
other sp4si scripts.
creates a new active file from the sites own active files regarding
the particular subscriptions in the newsfeeds file, stops INN, moves
the new active to the local active file, loads all INN's configuration
files and restart INN.
ATTENTION: before running these scripts you must have already configured
the newsfeeds file. Because of performance (the newsfeeds
file must be not interpretated for every run of subscribe) spool
files will be created and placed into site/<site> directory.
This files contain names of newsgroups which are to be spooled by a certain
host and are required for the autosubscribing procedure.
Syntax: spool.news [-s]
is a spool frontend with multitasking ability. Thereby the following
spool scripts will be invoked automatically: nntp.up, nntp.down,
and uucp.cico. The output of the scripts will be put together, a
report will be made and sent via email or posted it to a given newsgroup.
The -s parameter turn off multitasking spool and let run the spool
in synchronous mode, it is slower as the default multitasking mode but
it could be a little bit securer (no warranty) if there are problems with
Syntax: spool.down <site> [ logfile ]
it downloads the the news articles via NNTP protocol from the
suck(1) and post it using innxmit(1) to the
local host running INN.nntp.down expects some parameters:
wil be the same as used in newshosts configuration files, the stdout-output
could be optionally redirected to a file by giving the filename as the
nntp.down will be started
automatically by the spool.news script, but there is also a possibility
to start it separately (for debuging purposes). Normally the downloaded
articles will be posted using
innxmit(1). Because of many inqueries,
there is also a possibility to configure sp4si to run
by changing the LOCAL_POST variable in the env file.
Syntax: spool.up <site> [ logfile ]
it rewrites all outgoing articles using out.filter and posts
them via NNTP to remote host using rpost(1). All parameters
are of the same meaning as they be nntp.down are.
Syntax: uucp.batch <site> [ logfile ]
like nntp.up, uucp.batch rewrites all outgoing articles
using out.filter and put them as rnews(1) batches using batcher(1)
into site proprietary outgoing batch. If an alternate packer is specified
in newshosts this will be used (possible are: gzip, compress
and cat, whereby cat let the articles pass without any packing).
On unknown packer the default packer compress will be used. Other packer
will be implemented by the author by enough user's inquiry. To find out
the right packer ask your UUCP-news provider. Parameter <logfile>
has the same meaning as by nntp.* scripts. Because of no information
uulog(1) will be started and used to
get informations from the UUCP-Logfile how the spool ran. Running
uucp.batch the article will not be sent, just only prepared to be sent
out by the next run of uucp.cico.
Syntax: uucp.cico <site> [logfile]
initiates an UUCP call to the given site using uucico(1).
The baches which are prepared using uucp.bach before will be sent
out and new batches will be received and executed. Parameter <logfile>
has the same meaning as by uucp.batch script.
Syntax: out.filter [<infile> [<outfile>]]
this script has a special meaning sending out articles. All outgoing
articles will be rewritten using out.filter. There is no possibility
build in to let it out. out.filter uses awk(1) for filtering and
rewriting. The out.filter is configured to remove certain lines from the
header by the FILTER_STRIP variable. It is necessery to do this because
the most sites won't accept articles with certain predefined headers (e.g:
NNTP-Posting-Host oder Xref:). The host entra in the path can also be rewritten
by the string given by the FILTER_PATH parameter in the env file. out.filter
automatically if the filtered article is sending by uucp and turns into
the so called "rnews" mode using uucp.
Notice: the out.filter filters only header lines (until the
first empty line), not the message body. If the first line of a message
begins with '#! rnews nnn' the filter will switch into the mentioned
'rnews'-mode and recalculates the new article length.
subscribe scans all home directories which are defined in the /etc/passwd
file and reads out existing newsrc files on the local system (because
of the needed permissions it has to be started by root). subscribe
creates by comparing the subscribed newsgroups and the site/<site>/spool
files a new subscription (see below) config files. subscribe should
be started before running spool.news script. For the right interpretation
of newsrc files, they must be in newsrc format, because of
this many newsreaders would be supported, such as: slrn, tin
and similar and also Netscape Navigator and Communicator.
If the newsreader uses a different file name for his newsrc-file
then a link must be created. Basically, proprietary readers can also be
used, but prior to this, small changes at the subscribe script have to
be done. There is already support for one newsreader which uses a propriertary
newsrc file build in into subscribe. This ist the krn from KDE environment.
The subscriptions are divided into 2 diffent types:
nntp (for suck)
the current subscriptions from the newsrc files will be compared
with the site/<site>/sucknewsrc created running the previous
spool, these entries wil be taken for old subscriptions and new subscriptions
with be appended. An additional number will be attached after the name
of the newsgroup. This number is specified in the env file and can
be changed by setting of LAST parameter. Default value is -10, this mean
that on new subscriptions the last 10 articles will be sucked on the next
uucp (for gup)
UUCP requires a quite different management. A file site/<site>/uucpnewsrc
with last subscriptions will be created. At the next run of subscribe,
the new subscriptions will be compared with this file and then if this
two files are different a mail will be sent to remote gup account.
In this case we are sure that your news provider weren't bombed with your
Syntax: enter in the entry field of your browser: http://HOSTNAME/cgi-bin/wwwsubscribe
where HOSTNAME is the name of the machine where sp4si has been installed
and cgi-bin is the name of the CGI directory (in most cases it is cgi-bin
indeed). This script will be invoked by the HHTP-daemon. Therefore there
are restrictions about the access permissions of the HTTP-daemon account's.
In SuSE Linux distribution there is a special user, which exists only for
the HTTP daemon and uses for all thing made by the daemon his permissions.
The wwwsubscribe script maintains its own .wwwnewsrc file,
which is stored into home directory of user mentioned above. This allows
subscribe script (which is still required) to get the subscriptions
made via WWW-interface and regarding them (only if NEWSRC_WWW set
to "yes" in the env file) creating new SUCK configuration files.
Notice: wwwsubscribe doesn't replace the subscribe
script, the subscribe script is ever required.
the install script tries at first to get the environment definitions
of your INN news server. Then will ask you for the installation path of
sp4si, install the scripts and generates a new env and newshosts
config files in given directory. They are already default paths, so you
may have just to hit ENTER.
Notice: there will be two additional config files created in
the site subdirectory (if not exists): localhost.active and
which you can easy change to insert local newsgroups by changing their
contents. In this case your changes will be regarded by the next run of
rebuild.active. There are already two local newsgroups defined:
<hostname>.test (that may be useable) and <hostname>.spool
you can let post your spool-reports). <hostname> is the hostname
of the local host where sp4si has been installed. Because of usage
of local hostnames in the local newsgroups-names we will avoid, that local
postings they are posted outside because of wrong configuration of newsfeeds
CGI: the whole installation runs as user news, therefore there
are restrictions which not allow to write into the cgi-bin directory. Because
of this, you have to copy wwwsubscribe manually into the cgi-bin directory
and to set the right permissions. See capture 7. Installation.
before installing sp4si the INN must be already be right configured
and runable, this counts for UUCP too.
if you plan to use UUCP you have to install it before installing sp4si
if you want to use WWW interface to be able to do subscriptions, you
have to install a HTTP-daemon prior to sp4si
UUCP and HTTP-daemon installation and configuration description don't
belong to this package and you have to get them from other resources (e.g.:
HowTo). The sp4si
package consists of following parts:
install.sh - the install script
get.active- downloads active files from all configured newshosts
rebuild.active - generates a new active file
subscribe - for automatical newsgroup subscribing captured from
.newsrc compatible files.
wwwsubscribe - for manual newsgroup subscribing using the WWW-interface
spool.news - news spool frontend
nntp.down - spool articles down and posts it to the local news server
nntp.up - spools articles up to the remote newshost
uucp.batch - prepares the batches of outgoing articles
uucp.cico - (call-in-call-out) invokes the uucp call, spools outgoing
batches, receives and executes new batches
out.filter - just an outgoing filter for spool.up script
sp4si_de.html - german manual in HTML format
sp4si_en.html - english manual in HTML format
README - the small info file :-)
CHANGES - the history file
The archive must be unpacked first by typing tar xvfz sp4si-0.99.tar.gz
After that, you must change in the created subdirectory sp4si
typing cd sp4si
and start the install.sh
script expects no parameters. The innshellvars
file of your INN
installation will be found and parsed. Only the
destination directory for the sp4si
installation will be asked manually.
If the innshellvars
file couldn't be found, all othe required paths
will be asked too.
CGI Script installation:
on the installation you will be asked for the name of the account of
your HTTP-daemon. If you don't know about it, you can find it out as follows:
ps@comm:~ > ps -fC httpd
UID PID PPID C STIME TTY TIME CMD
root 10389 1 0 Dec08 ? 00:00:04 /usr/sbin/httpd -f /etc/httpd/httpd.conf -D PERL -D PHP
wwwrun 10392 10389 0 Dec08 ? 00:00:07 /usr/sbin/httpd -f /etc/httpd/httpd.conf -D PERL -D PHP
wwwrun 11064 10389 0 Dec08 ? 00:00:07 /usr/sbin/httpd -f /etc/httpd/httpd.conf -D PERL -D PHP
After the install.sh script finished, you will have to copy the wwwsubscribe script
manually into the cgi-bin directory and to set the right permissions.
You can set the permissions by typing: chmod 755 wwwsubscribe
After the installaion has successfully finished, the names of remote news
servers can be inserted into the newshosts
file (you will find this
in your destination directory), it could be as follows:
# example for NNTP sites
# example for UUCP sites
After you have added your remote news server names and saved into the newshosts
file, you can attempt the second step, the download if the remote active
files. By starting the script as shown below, all active files will be
downloaded and stored in site/<site>
After all active
file are successfully downloaded, you have to attempt
the difficultest step of the sp4si
configuration: the newsfeeds
(don't confuse with with newshosts
) entries. You have to take a
look at the (new downloaded) active files in
and have to decide which newsgroups from which news servers you will spool
to your local news server. You should decide so, that each newsgroup will
be spooled only by one host, you avoid therewith to propagate newsgroups
between your newsfeeds. Therefore you should read the man page for newsfeeds
You can look for an example in the german manual (sorry) too: "Erweiterung
von INN & suck auf mehrere Newsserver"
After the newsfeeds
file has been configured, a new active
file for your INN
can be created by typing:
On running this, a new active file will be created (regarding the localhost.active
file) and moved to the INN
's active file. For that, INN
be stopped, the config files reloaded and re-parsed (the changes on
file will be active) and finally the INN will be restarted.
So, we have finished our configuration of sp4si and you are able to spool
all your configured news hosts now.
For your synopsis:
enter <site> <newshost> and maybe <up/down> <packer> <gupmail> <password> too in newshosts file
connect to internet and run get.active, than you can continue offline
enter newsfeeds in newsfeeds file
sp4si is developed for decrease the news administration effort, here you
will find some useful tips which can help you by your daily work with sp4si.
to add a new newsgroup f.e: my.newsgroup
, it must be inserted in
localhost.test 0000000000 0000000001 y
localhost.spool 0000000000 0000000001 y
my.newsgroup 0000000000 0000000001 y
must be restarted:
after that all we have now 3 local newsgroups.
sp4si needs no special maintenance if once the right things are made. This
requires putting a few lines into the /etc/crontab file:
ACTIVE="su -l news -c /etc/news/sp4si/get.active"
REBUILD="su -l news -c /etc/news/sp4si/rebuild.active"
SPOOL="su -l news -c /etc/news/sp4si/spool.news"
ONLINE=/etc/ppp/ppp-on # how to get online
OFFLINE=/etc/ppp/ppp-off # how to get offline
0 6,16,21 * * * root $ONLINE; $SUBSCRIBE; $SPOOL; $OFFLINE
0 0 1 * * root $ONLINE; $ACTIVE; $OFFLINE; $REBUILD
As you can see, it the spool will be startet 3 times in the day, but prior
to spool, the subscribing will be checked first. Further every 1st day
of the month, the active file will be updated and makes it up-to-date (you
can do this often if you want). To get further information about the syntax
of crontab please refer the man page: man 5 crontab
As mentioned above, sp4si
can be configured by changing the env
file. This file is regular script, which will be executed at first by the
other scripts to make the settings be regarded. The env
of 2 parts:
the path settings area, where all needed paths are set. Theese are determined
during the run of installation script and they don´t have be changed:
# ** Script Package for Suck & INN (sp4si) **
# Read Manual for more
# Description: configuration file, will be set during installation
# You can change it manually.
# path environment
# DON'T REMOVE IT ! This is absolute necessery for running of sp4si.
# If you want to change any path values, you have to mov the correspondent
# files to the right directory.
BASE_DIR=/var/lib/news/sp4si # location directory of scripts
SITE_DIR=/var/lib/news/sp4si/site # location of site's active and newsgroups files
LOG_DIR=/var/lib/news/sp4si/log # directory for log files
SPOOL_DIR=/var/spool/news # spool directory
NEWSFEEDS=/var/lib/news/newsfeeds # path to newsfeeds file
ACTIVE=/var/lib/news/active # path to active file
OUTGOING_DIR=/var/spool/news/out.going # location of the newsfeeds, rpost need this
PATH=$PATH:/var/lib/news/sp4si:/usr/lib/news/bin:/usr/lib/uucp # path for binaries and scripts
# end of path environment
the configuration area, the spool variables are defined here, they could
be changed, but they may only contains predefined values (see comments).
If a variable is removed or wrong set, there a default value set in the
correspondent script will be taken.
# configuration settings
# Each entry may be changed. If you remove an entry, the default value will
# be used. If you want to change the following values, so you have to use
# the discriptions beside the options or the manual. All option's names are
# in upper and all values are in lower case. An exception are string options
# such a FILTER_STRIP parameter.
in the spool settings part the spool mode will be defined. With MULTI=yes
turns the musltitasking spool on, here the scripts will be invoked in background
and at the same time, with MULTI=no all spool scripts will be invoked one
after another. Because of performance is the first the better choice:
# spool.news settings
MULTI=yes # 'yes' = multitasking spool (asynchronous), 'no' = synchronous
after that the report settings are defined. Reports are the status outputs
of each of the spool scripts that are bundled to one file. sp4si sends
the reports via mail or posts it to a given newsgroup. There are two newsgroups
created automaticalle during the installation, one of this is $HOSTNAME.spool,
which will be taken by default. If you have subscribed this group by your
newsreade, you will be able to read the reports by using your newsreader.
For both type of reports (mail and posting) you can set the condition and
the destination for sending reports. As condition they are three cases:
"on error", "ever" and "never", the destination may be defined as follows:
be by mail report the destination address to send the reports to and by
news the name of the desired newsgroups to post the reports to:
MAIL_REPORT=error # 'error'=only error, 'yes'=ever send, 'no'=no mail
MAIL_TO=news # email address of the report recipient
NEWS_REPORT=yes # 'error'=only error, 'yes'=ever send, 'no'=no news
NEWS_TO=comm.spool # newsgroups to send the report to
the next settings belongs to out.filter script. Here are defined
the header lines, which should be removed from the outgoing articles. New
items can be included by inserting the headerline (with colon at the end)
to FILTER_STRIP variable and separating it by white space. The FILTER_PATH
parameter contains the string which shgould used instead of the locol host
name in the Path: header.
# filter settings
# the following header fields will be stripped from outgoing messages
FILTER_STRIP='NNTP-Posting-Host: Xref: X-Server-Date:'
FILTER_PATH="not-for-mail" # to leave original path set to ''
now follow the subscribe settings, you can set the NEWSRC parameters
to "yes" or "no" to enable or disable the checking the correspondent newsrc
or wwwnewsrc files for new subscriptions. The LAST parameter contains
the number of articles to be sucked from a newsgroup on new subscribed
groups. This number has to be negativ.
# subscribe settings
# 'yes' enables and 'no' disables an option :-)
NEWSRC=yes # for .newsrc file, it works with the most news reader
NEWSRC_KRN=no # for krn (kde) news reader
NEWSRC_WWW="yes" # for www interface (CGI)
LAST=-10 # count of articles to download if new subscribed
the last part specifies the spool with suck. You can add here other suck
parameters and options. By the LOCAL_POST variable you can set the tool
for local posting of downloaded articles. You can choose from two: rnews
and innxmit. rnews is prefered used automatically by UUCP
and is only implemented due to some inqueries to be used using NNTP, innxmit
belongs to INN distribution and should be used as possible. I don´t
know if if there are perfomance differences between of both tools.
# nntp.down settings
SUCK_OPTIONS="" # insert additional options or parameters (man suck)
LOCAL_POST="innxmit" # how to post articles locally: 'rnews' or 'innxmit'
# end of env file
9.4. sp4si and CNFS
sp4si has been adapted to be working with CNFS together. The sending articles
would be uploaded one after another, this cause poor perfomance on high
number of articles. This issue has to be improved in the future. It is
planed to make a batch file with the outgoing articles and to spool it
with the new version of rpost (>4.2.3 is able to handle rnews-batches).
How to setup your INN to CNFS have a look at the german (sorry) how-to
der INN Konfiguration uuf CNFS
Please use one of the links listed below:
If you found any bugs or have any suggestions to make sp4si better please
write me an email to: Peter Sobisch <firstname.lastname@example.org>.
I want to thank Carsten Voß,
whose manual helped me to install my first INN news system.