Monitoring SpamAssassin (Spamd) with MRTG

Concept

Like many users of spamassassin, we found that we had to offload spam processing to a separate machine (using spamc/spamd) for performance reasons. Being a stats freak, I wanted to use our existing MRTG system to follow spamd stats. I had already discovered the mailstats program for obtaining information about sendmail, so I decided to come up with a similar approach to spamassassin.

There are several steps you'll need to take, but it's fairly simple. Instructions assume that you're running MRTG and spamd on separate machines, since that's how I'm running it.  

Define custom port

On both the spamd server and on your MRTG you'll need to add the following (or similar) to /etc/services:

 spam-stats      7300/tcp                        # spam-stats

spamd machine

Install spam-stats.p_ on the spamd machine. This program will parse through your log file, and develop several totals. You will need to create a "/var/local/spam-stats/" directory, or change the location of $statfile and $offsetfile in order to allow spam-stats.l to store its state information between runs.

UPDATE: (7/30/03) I finally got around to writing a perl version. The old bash script is still available (spam-stats) if anyone cares. The old version is a simple bash script using wc. This version requires logtail, and will also need to create some state files (logtail's offset, and a dbm file for the stats). DBM is overkill for the ammount of data, but it was cleaner than parsing and updating a text file.

Then you'll need to get access to spam-stats through the network. Add the following line to /etc/inetd.conf on the spamd machine:

spam-stats      stream  tcp     nowait  root    /usr/local/sbin/spam-stats.pl

counts

...and restart inetd  

For xinet users, Paul Fries paul@cwie.net sent me a copy of his xinetd.d file. A copy can be found at xinetd.d. Thanks for the suggestion, Paul. It took me almost a year to post it, but the info is finally up here.  

MRTG machine: parser program

Install spamstats.p_ in /usr/local/mrtg/spamstats.pl (or wherever you prefer) on your MRTG station. This program will connect to the spamd machine and print the appropriate results, in the format that MRTG expects to see.

UPDATE: (3/8/09) reamed the .pl files to ".p_". This directory used to offer up .pl files as plain text, but it looks like my ISP wants to execute all .pl files now.

UPDATE: (7/30/03) spamstats.pl has been modified to include uptime and server name.  

MRTG machine: MRTG settings

Here are some suggested MRTG config file settings. You can also add the gauge option if you'd like to see the rate of spam processing. For some reason, I'm currently most interested in seeing totals. I also have things set to see relative percentage of spam/total processed, nonspam/total processed, and processed/total submitted messages.

#--------------------------------------------------------------- # # show number of messages that were spam, total number of #   messages _processed_ by spamd # Target[spam_spam]: `/usr/local/mrtg/spamstats.pl host.name.here spam` MaxBytes[spam_spam]: 100000 Options[spam_spam]: growright,nopercent,perminute Title[spam_spam]: Spam Statistics PageTop[spam_spam]: SA-identified spam, SA-processed email </H1> XSize[spam_spam]: 500 YSize[spam_spam]: 200 WithPeak[spam_spam]: dwmy YLegend[spam_spam]: No. of messages ShortLegend[spam_spam]: messages LegendI[spam_spam]: &nbsp;spam: LegendO[spam_spam]: &nbsp;total:

#--------------------------------------------------------------- # # show number of messages that were not spam and total number of #   messages _processed_ by spamd # Target[spam_clean]: `/usr/local/mrtg/spamstats.pl host.name.here nonspam` MaxBytes[spam_clean]: 100000 Options[spam_clean]: growright,nopercent,perminute Title[spam_clean]: Spam Statistics

PageTop[spam_clean]: SA-identified clean, SA-processed email</H1> XSize[spam_clean]: 500 YSize[spam_clean]: 200 WithPeak[spam_clean]: dwmy YLegend[spam_clean]: No. of messages ShortLegend[spam_clean]: messages LegendI[spam_clean]: &nbsp;nonspam: LegendO[spam_clean]: &nbsp;total:  

#--------------------------------------------------------------- # # show number of messages skipped (large messages) total # _submitted_ Target[spam_skipped]: `/usr/local/mrtg/spamstats.pl host.name.here skipped` MaxBytes[spam_skipped]: 100000 Options[spam_skipped]: growright,nopercent,perminute Title[spam_skipped]: Spam Statistics PageTop[spam_skipped]: Messages skipped due to large size, total messages

submitted</H1> XSize[spam_skipped]: 500 YSize[spam_skipped]: 200 WithPeak[spam_skipped]: dwmy YLegend[spam_skipped]: No. of messages ShortLegend[spam_skipped]: messages LegendI[spam_skipped]: &nbsp;skipped: LegendO[spam_skipped]: &nbsp;total:  

#--------------------------------------------------------------- # # show percentage of spam messages Target[spam_percent_spam]: `/usr/local/mrtg/spamstats.pl host.name.here spam` MaxBytes[spam_percent_spam]: 100000 Options[spam_percent_spam]: growright,gauge,nopercent,dorelpercent,noi,noo Title[spam_percent_spam]: Spam Statistics PageTop[spam_percent_spam]: Percent of messages tagged as spam</H1 XSize[spam_percent_spam]: 500 YSize[spam_percent_spam]: 200 WithPeak[spam_percent_spam]: dwmy YLegend[spam_percent_spam]: percent of messages ShortLegend[spam_percent_spam]: messages LegendI[spam_percent_spam]: &nbsp;skipped: LegendO[spam_percent_spam]: &nbsp;total:  

#--------------------------------------------------------------- # # show percentage of non-spam messages Target[spam_percent_nonspam]: `/usr/local/mrtg/spamstats.pl host.name.here

nonspam` MaxBytes[spam_percent_nonspam]: 100000 Options[spam_percent_nonspam]: growright,gauge,nopercent,dorelpercent,noi,noo Title[spam_percent_nonspam]: Spam Statistics PageTop[spam_percent_nonspam]: Percent of messages tagged as non-spam</H1 XSize[spam_percent_nonspam]: 500 YSize[spam_percent_nonspam]: 200 WithPeak[spam_percent_nonspam]: dwmy YLegend[spam_percent_nonspam]: percent of messages ShortLegend[spam_percent_nonspam]: messages LegendI[spam_percent_nonspam]: &nbsp;skipped: LegendO[spam_percent_nonspam]: &nbsp;total:  

Author

Rich Puhek rpuhek@2z.net>

Thanks to:

Tobias Oetiker <oetiker@ee.ethz.ch> and   Dave Rand <dlr@bungi.com>

...for MRTG itself

Petter Reinholdtsen <pere@td.org.uit.no>

...for the mailstats program that inspired this little idea.   last modified: 9/12/03 -- Added note to create /var/local/spam-stats directory. 08/15/2003 -- Added xinetd config info from Paul Fries. 07/30/2003 -- Updated spamstats.pl to include uptime and server name. New spam-stats.pl returns servername and uptime, also uses logtail and a DBM so we don't look at the whole log file all the time.

someday I'll clean up the page format...