exim, perl and snmp! oh my! ian norton shadowcat - agaton

Exim, Perl and SNMP!Oh my!

Ian NortonShadowcat Systems Ltd.

Exim, Perl and SNMP!Oh my!

Working with Net SNMP Extend.

Ian NortonShadowcat Systems Ltd.

What's this talk about?

● Using Net SNMP extend● Monitoring arbitrary things● Using Exim as an example● Cleverness with OpenNMS

Email systems are evil.

Exim is less evil :)

● Open source● Mail Transport Agent (MTA)● Mail Delivery Agent (MDA)● Cambridge University

In a previous life....

postmaster@lancaster.ac.uk

University infrastructure

InternetHub1

Hub1Hub1

Exchange

Unixmail

Dept 1 Dept 2

Standard service monitoring

● Connect TCP port 25● Check banner● How long did it take?

And so it begins...

● Messages delayed● 4xx temporary failures● Badly behaved senders don't retry

Everything is fine!

The monitoring says so!

idn facepalms

Everything is not fine

● One hub has an issue● Spam Assassin is dead● No problem identified● The monitoring is wrong

Service was affected.Messages were delayed.

So what's the problem?

● Spam Assassin process went away● Exim unable to scan messages● Temporary reject (safe thing to do)

How do we test?

Testing

● Spam Assassin provides spamc● Create a batch SMTP file:

helo testing.example.comMAIL FROM: "Person, A" <norton@exchange.example.com>RCPT TO: idn@unix.example.comDATASubject: Testing viagraTo: idn@unix.example.comFrom: "Person, A" <idn@exchange.example.com>

Test SMTP session.QUIT

Testing

● spamc -c -B < bsmtp-file● Exit status zero or one

Enter Perl

● List of files to test & expected return● Test files and check return

# Files to test along with expected exit valuemy $files = { "01spamassassinnormalmessage.txt" => 0, "02spamassassinspammymessage.txt" => 1,};

my $fail = 0;

# Scan each of the files in the hashforeach my $file (keys(%$files)) {

# Check the file system("/usr/bin/spamc c B < $file > /dev/null");

# Bitshift $? to get the exit code my $exitval = $CHILD_ERROR >> 8;

# Compare the exit value to that expected if($exitval != $files>{$file}) { $fail = 1; last; }}

# Files to test along with expected exit valuemy $files = { "01spamassassinnormalmessage.txt" => 0, "02spamassassinspammymessage.txt" => 1,};

my $fail = 0;

# Scan each of the files in the hashforeach my $file (keys(%$files)) {

# Check the file system("/usr/bin/spamc c B < $file > /dev/null");

# Bitshift $? to get the exit code my $exitval = $CHILD_ERROR >> 8;

# Compare the exit value to that expected if($exitval != $files>{$file}) { $fail = 1; last; }}

Files totest

Expectedexit

Write status to a file

open(my $fh, '>', '/tmp/sastatus') or die('Cannot open file');print($fh $fail);close($fh);

So we know #fail.

Now what?!

● Add SNMP!

● snmpd.confextend sastatus /bin/cat /tmp/sastatus

● snmpwalk$ snmpwalk v2c c public 127.0.0.1 .1.3.6.1.4.1.8072.1.3.2 NETSNMPEXTENDMIB::nsExtendNumEntries.0 = INTEGER: 1NETSNMPEXTENDMIB::nsExtendCommand."sastatus" = STRING: /bin/catNETSNMPEXTENDMIB::nsExtendArgs."sastatus" = STRING: /tmp/sastatusNETSNMPEXTENDMIB::nsExtendInput."sastatus" = STRING: NETSNMPEXTENDMIB::nsExtendCacheTime."sastatus" = INTEGER: 5NETSNMPEXTENDMIB::nsExtendExecType."sastatus" = INTEGER: exec(1)NETSNMPEXTENDMIB::nsExtendRunType."sastatus" = INTEGER: runonread(1)NETSNMPEXTENDMIB::nsExtendStorage."sastatus" = INTEGER: permanent(4)NETSNMPEXTENDMIB::nsExtendStatus."sastatus" = INTEGER: active(1)NETSNMPEXTENDMIB::nsExtendOutput1Line."sastatus" = STRING: 1NETSNMPEXTENDMIB::nsExtendOutputFull."sastatus" = STRING: 1NETSNMPEXTENDMIB::nsExtendOutNumLines."sastatus" = INTEGER: 1NETSNMPEXTENDMIB::nsExtendResult."sastatus" = INTEGER: 0NETSNMPEXTENDMIB::nsExtendOutLine."sastatus".1 = STRING: 1

Now we can monitor!

OpenNMS

● poller-configuration.xml <service name="SAStatus" interval="300000" userdefined="false" status="on"> <parameter key="retry" value="2"/> <parameter key="timeout" value="3000"/> <parameter key="port" value="161"/> <parameter key="oid" value=".1.3.6.1.4.1.8072.1.3.2.4.1.2.9.115.97.45.115.116.97.116.117.115.1"/> <parameter key="operator" value="="/> <parameter key="operand" value="0"/> </service>

OpenNMS

Commandoutput

OpenNMS

Externalcommandnumber

#fail.

+++OUT OF CHEESE+++

OpenNMS

Length ofidentifier

“sa-status”Thanks to roskens on

#opennms for pointing outthat this was wrong.

Cheers! :)

OpenNMS

“sa-status”

OpenNMS

Linenumber ofcommand

output

OpenNMS

Commandoutput

“sa-status”

Linenumber ofcommand

output

Length ofidentifier

“sa-status”

Provisioning

● Foreign source

● Node

That works!

But it sucks!

Issues

● Stale file?● Not SNMP tables - no way to instance map● Works for simple data● Can have multiple lines● Assumes ordering with multiple lines

Issues

● Want to add a service● Want to remove a service● Want to re-order my file● …...

Issues

● Want to add a service● Want to remove a service● Want to re-order my file (OCD attack)● …...

Mail queue size

● Want to track destination domains● Which are queueing● Which are local, remote, etc

Our single file approachdoes not scale.

Add cloud more files!

Output to two files

● Keys● Values

● The data can change● OpenNMS will map together for us● Add a timestamp

Output to two files

# Open the files to work withmy $file = $path . $counter . '_';my $keys_file = IO::File>new( $file . 'keys', 'w' ) or croak($OS_ERROR);my $stats_file = IO::File>new( $file . 'stats', 'w' ) or croak($OS_ERROR);

my $timestamp = time();

# Output the current timestamp to both files$keys_file>print("$timestamp\n");$stats_file>print("$timestamp\n");

# Output the keys and stats to the relevant files.foreach my $key ( sort( keys( %{$stats} ) ) ) { $keys_file>print("$key\n"); $stats_file>print( $stats>{$key} . "\n" );}

# Close the File::IO objects$keys_file>close();$stats_file>close();

Generate the data

● Exim has exipick● --flatq option is designed for parsing● Options for extra data

Generate the data# Set the exipick binary & CLI optionsmy $exipick_cmd = "/usr/sbin/exipick";my $exipick_opt = "flatq showvars message_size,deliver_freeze";

# Initialise interesting domain counters and the two catchall domains.my %domains = ('exchange.example.com' => 0, 'dept1.example.com' => 0, '*.example.com' => 0, # Catchall for other example.com 'internal' => 0, # Counter for all internal 'external' => 0, # Catchall for everything else);

# Run the exipick command and process the output.open(my $exipick, '|', "$exipick_cmd $exipick_opt");

# Loop through the command outputwhile(<$exipick>) { chomp;

my $line = $_;

# Split the showvars with spaces rather than semicolons $line =~ s/;/ /g;

# Send the exipick line to be processed. process_line($line);}

close($exipick);

Generate the datasub process_line { my $line = shift;

# Sample exipick output line with email address changed to anonymise: # 4d message_size='9265' deliver_freeze='' 1E9hWO0001AH4A <> sn@als.org

# Split the output line using a regexp if($line =~ m/.* message_size='(\d*)' deliver_freeze='(.*)' (.*.*.*) <(.*)> (.*)/){ my $msg_size = $1; my $msg_frozen = $2; my $msg_id = $3; my $msg_sender = $4; my $msg_recipient = $5;

# Send the recipient email address to process domain for a breakdown. process_domain($msg_recipient); }}

Generate the datasub process_domain { my $email = shift;

# Get the domain from the email address. my $domain = $email; $domain =~ s/^.*@//;

# Is this an internal domain? if($domain =~ m/example.com$/) { $domains{'internal'}++; }

# If this domain exists, increment the counter for that domain. if(defined($domains{$domain})) { $domains{$domain}++; }

# Increment the catchall counter for *.example.com addresses elsif ($domain =~ m/example.com$/) { $domains{'*.example.com'}++; }

# Increment the external counter for everything else else { $domains{'external'}++; }}

Generate the data

$VAR1 = { 'dept1.example.com' => 34, 'exchange.example.com' => 26, 'external' => 43, 'internal' => 110, '*.example.com' => 50 };

● Generates this data:

● Which writes our two files

/tmp/mailq_keys

/tmp/mailq_stats

● snmpd.conf

extend mailqkeys /bin/cat /tmp/mailq_keysextend mailqstats /bin/cat /tmp/mailq_stats

● snmpwalk$ snmpwalk v2c c public 127.0.0.1 NETSNMPEXTENDMIB::nsExtendOutLineNETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".1 = STRING: 1534221421134720NETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".2 = STRING: *.example.comNETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".3 = STRING: exchange.example.comNETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".4 = STRING: dept1.example.comNETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".5 = STRING: internalNETSNMPEXTENDMIB::nsExtendOutLine."mailqkeys".6 = STRING: externalNETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".1 = STRING: 1534221421134720NETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".2 = STRING: 50NETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".3 = STRING: 26NETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".4 = STRING: 34NETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".5 = STRING: 110NETSNMPEXTENDMIB::nsExtendOutLine."mailqstats".6 = STRING: 43

Now we can monitor!

OpenNMS

OpenNMS

<systemDef name="Exim monitoring"> <sysoidMask>.1.3.6.1.4.1.8072.3.2.10</sysoidMask> <collect> <includeGroup>eximmailq</includeGroup> </collect> </systemDef>

Graphs!

# Reports list for Eximreports=exim.eximMailQueueKey, exim.eximMailQueueStat

report.exim.eximMailQueueStat.name=Messagesreport.exim.eximMailQueueStat.columns=eximMailQueueStatreport.exim.eximMailQueueStat.propertiesValues=eximMailQueueKeyreport.exim.eximMailQueueStat.type=eximMailQueueInstreport.exim.eximMailQueueStat.command=title="{eximMailQueueKey}" \ DEF:messages={rrd1}:eximMailQueueStat:AVERAGE \ LINE2:messages#0000ff:"Messages" \ GPRINT:messages:AVERAGE:" Avg \\: %2.0lf %s" \ GPRINT:messages:MIN:" Min \\: %2.0lf %s" \ GPRINT:messages:MAX:" Max \\: %2.0lf %s\\n"

Graphs!

OpenNMS

● Now we get thresholding● And all the fun of the NMS!

Further expansion

● Antivirus● Total number of messages● Log file parsing● Message transit times● RBL monitoring

Questions?

Thanks!

Slides available at:http://agaton.scsys.co.uk/~iann/talks/

Ian Nortoni.norton@shadowcat.co.uk

idn on irc.perl.org and irc.freenode.net

exim, perl and snmp! oh my! ian norton shadowcat - agaton

Documents

snmp/web interface ups snmp card - … netys... ·...

explore exim

exim specification

snmp 1. snmp versions snmp version 1 (snmpv1) snmp version 2...

exim intro

snmp applied - sicheres anwendungs-monitoring mit snmp

snmp/web interface ups snmp card - socomec · snmp/web...

snmp informant-ms snmp-wmi comparison

c. incidents monitored - ndrrmc.gov.ph · at 5:00 pm, td...

exim spices

exim policies

snmp - cisco · snmp...

exim bank2

exim manual

snmp inform request - cisco · snmp inform request...

minuteman snmp-s and snmp-m - … ups/snmp-s and...

exim final

exim apples

exim submission

locus exim