mta performance comparison - eurobsdcon · –powerbook g3 “pismo” •os: macos x 10.2.1...
TRANSCRIPT
MTA Performance Comparison:sendmail vs. postfix on *BSD
Brad Knowles
Senior Consultant for Snow, [email protected]
http://www.shub-internet.org/brad/papers/mtacomparison/
Entire contents copyright © 2002 by Brad Knowles, all rights reserved
2002-11-16 Copyright © 2002 by Brad Knowles 2
Overview
• Goal• Meta Information
– Hardware Used– Software Tested– Tools Used– Methodology
• Test Results• Conclusions
2002-11-16 Copyright © 2002 by Brad Knowles 3
Goal
• Show you what it looks like to do MTAperformance tuning– On as many *BSD platforms as I could
• Focus on the process, not the numbersor expected/desired outcome– You get to see (most) everything, warts
and all
2002-11-16 Copyright © 2002 by Brad Knowles 4
Goal• Basically, chapter 3
from the booksendmailPerformance Tuning byNick Christenson– See
http://www.jetcafe.org/~npc/book/sendmail/
2002-11-16 Copyright © 2002 by Brad Knowles 5
Meta Information
• Hardware Used• Software Tested• Benchmark Tools• Methodology
2002-11-16 Copyright © 2002 by Brad Knowles 6
Hardware Used• NetBSD
– Twinhead “Twinstation 5G” (Sun SPARCstation 5clone)
– Not UltraSPARC, but the ancient SPARC 5• OS: NetBSD 1.6-RELEASE• CPU: microsSPARC-II @ 110MHz• RAM: 32MB real, 384MB virtual• NIC: On-board “Lance” 10Base-T Ethernet
& SBus QuadFastEthernet
• Unfortunately, disk drives died before testing could beperformed (bug in NetBSD regarding SCSI taggedcommand queueing for old drives?)
2002-11-16 Copyright © 2002 by Brad Knowles 7
Hardware Used
• FreeBSD– Compaq Armada 4131T
• OS: FreeBSD 4.6.2-RELEASE• CPU: Pentium 133• RAM: 48MB real, 384MB virtual• NIC: Asanté FriendlyNET AL1011
“Prism2” 802.11b WiFi PCMCIA• HD: 10GB IBM Travelstar 20GN
– 4200 RPM– 12ms avg. seek
2002-11-16 Copyright © 2002 by Brad Knowles 8
Hardware Used:FreeBSD
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16 Copyright © 2002 by Brad Knowles 9
Hardware Used
• MacOS X (Part 1)– PowerBook G3 “Pismo”
• OS: MacOS X 10.2.1• CPU: PowerPC G3 @ 400MHz• RAM: 1GB real, 2GB virtual• NIC: Apple AirPort 802.11b WiFi• HD: 48GB IBM Travelstar 40GH
– 5400 RPM– 12ms avg. seek
2002-11-16 Copyright © 2002 by Brad Knowles 10
Hardware Used:MacOS X (Part 1)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16 Copyright © 2002 by Brad Knowles 11
Hardware Used
• MacOS X (Part 2)– PowerBook G4 (DVI)
• OS: MacOS X 10.2.2• CPU: PowerPC G4 @ 800MHz• RAM: 1GB real, 2GB virtual• NIC: Apple AirPort 802.11b WiFi• HD: 40GB Toshiba MK4018GAS
– 4200 RPM– 12ms avg. seek
2002-11-16 Copyright © 2002 by Brad Knowles 12
Hardware Used:MacOS X (Part 2)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16 Copyright © 2002 by Brad Knowles 13
Software Tested
• Sendmail 8.12.6– ftp://ftp.sendmail/org/pub/sendmail/sendmail.8.12.6.tar.gz
• Postfix 1.1.11– http://www.postfix.org/ftp-sites.html
• Exim 4.10– Tried every way I could think of, but could
not get it configured to relay back to an IPaddress or port on the same machine
2002-11-16 Copyright © 2002 by Brad Knowles 14
Tools Used• Disk Benchmarking
– PostMark from Network Appliance• http://www.netapp.com/tech_library/3022.html
– Bonnie++ from Russell Coker• http://www.coker.com.au/bonnie++/
• SMTP Benchmarking– smtp-source/smtp-sink from Wietse Venema
• Included with postfix distribution– Postal from Russell Coker (with smtp-sink)
• http://www.coker.com.au/postal/
2002-11-16 Copyright © 2002 by Brad Knowles 15
Methodology
• Get disk subsystem baseline byperforming benchmarking– For each platform & filesystem
configuration• PostMark
– See http://www.shub-internet.org/brad/FreeBSD/postmark.html
• Bonnie++– Configured to test similar parameters
» File sizes, numbers, & subdirectories accordingto the four PostMark tests
2002-11-16 Copyright © 2002 by Brad Knowles 16
Methodology• For FreeBSD, we only test turning on or off “soft
updates”– We do not test (turned on by default)
• vfs.vmiodirenable• dirprefs• dirhash
– For data on all the possible combinations of the above, see“Recent Filesystem Optimisations in FreeBSD” athttp://www.usenix.org/events/usenix02/tech/freenix/dowse.html
– For comparison data of Journaling vs. Soft Updates, see“Journaling Versus Soft Updates: Asynchronous Meta-dataProtection in File Systems” athttp://www.usenix.org/publications/library/proceedings/usenix2000/general/seltzer.html
2002-11-16 Copyright © 2002 by Brad Knowles 17
Methodology
• For MacOS X– We test only
• HFS+– Apple’s Extended Hierarchical File System
• MFS– Memory-based filesystem created using
RamBunctious 2.0
2002-11-16 Copyright © 2002 by Brad Knowles 18
Methodology
• For MacOS X– We do NOT test
• UFS– IMO, Apple broke it while porting HFS+ to Mach– Many applications do not recognize it, in addition to
MacOS 9 itself– “Classic” requires that boot device be HFS+ and
MacOS 9 must be installed before MacOS X• HFS+ with Journaling
– Too new, introduced with MacOS X Server 10.2.2– Initial reports indicate ~20% performance loss
2002-11-16 Copyright © 2002 by Brad Knowles 19
Methodology• For MacOS X
– Does not have a native “poll.h” or “libpoll”(required by postal)
• Install manually from sourcehttp://www.clapper.org/software/poll/
• Install using “fink”– MacOS X package/port manager based on Debian
“dselect”– Unable to test Bonnie++
• Need further assistance from the author to find out why itis crashing
2002-11-16 Copyright © 2002 by Brad Knowles 20
Methodology• Perform SMTP benchmarking
– For each hardware & software configuration, test• Test direct source->sink connection
– Demonstrate maximum possible performance• Test source->relay->sink
– Show relay performance
– Note• Testing only relaying, not local delivery• Test generated on, relayed through, and terminated on
“loopback” network– Eliminate all question of network or NIC performance
2002-11-16 Copyright © 2002 by Brad Knowles 21
Test Results
• Chart data only– PostMark table data available from
http://www.shub-internet.org/brad/FreeBSD/postmark.html– Bonnie++ table data not yet available– smtp-source/smtp-sink data not yet
available• `hostname` must be legal (I.e., does not end
with “.”)– Postal data not yet available
2002-11-16 Copyright © 2002 by Brad Knowles 22
Test Results
• Dumb “Driver” Scripts– My “runpostmark” script available at
http://www.shub-internet.org/brad/runpostmark– My “runbon” script available at
http://www.shub-internet.org/brad/runbon– My “runsmtp” script available at
http://www.shub-internet.org/brad/runsmtp
2002-11-16 Copyright © 2002 by Brad Knowles 23
Test Results:PostMark Sample Run
% cd /var/tmp% postmarkPostMark v1.5 : 3/27/01pm>set number 1000pm>set transactions 50000pm>showCurrent configuration is:The base number of files is 1000Transactions: 50000Files range between 500 bytes and 9.77 kilobytes in sizeWorking directory: /var/tmpBlock sizes are: read=512 bytes, write=512 bytesBiases are: read/append=5, create/delete=5Using Unix buffered file I/ORandom number generator seed is 42Report format is verbose.pm>run
2002-11-16 Copyright © 2002 by Brad Knowles 24
Test Results:PostMark Sample Run
Creating files...DonePerforming transactions........DoneDeleting files...DoneTime: 497 seconds total 492 seconds of transactions (101 per second)
Files: 26014 created (52 per second) Creation alone: 1000 files (500 per second) Mixed with transactions: 25014 files (50 persecond) 24868 read (50 per second) 24880 appended (50 per second) 26014 deleted (52 per second) Deletion alone: 1028 files (342 per second) Mixed with transactions: 24986 files (50 persecond)
Data: 161.15 megabytes read (332.02 kilobytes per second) 168.38 megabytes written (346.92 kilobytes per second)
2002-11-16 Copyright © 2002 by Brad Knowles 25
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #1: 1000 Files & 50k Transactions
0.00
1,000.00
2,000.00
3,000.00
4,000.00
5,000.00
6,000.00
7,000.00
8,000.00
UFSTMPFS
ODS/R0
ODS/R5
NFS/F330
NFS/F630
PPro+SOFT
P3/Linux
P3+DPT
2xP3+UFS
2xP3+SOFT
2xP3+MFS
U5+UFS
U5+Log
U5+TMPFS
PBG3/HFS+
PBG3/MFS
PBG4/HFS+
PBG4/MFS
Compaq+UFS
Compaq+Soft
Platform
KB/Sec
0
500
1,000
1,500
2,000
2,500
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 26
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #2: 20k Files & 50k Transactions
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
3,000.00
UFSTMPFS
ODS/R0
ODS/R5
NFS/F330
NFS/F630
PPro+SOFT
P3/Linux
P3+DPT
2xP3+UFS
2xP3+SOFT
2xP3+MFS
U5+UFS
U5+Log
U5+TMPFS
PBG3/HFS+
PBG3/MFS
PBG4/HFS+
PBG4/MFS
Compaq+UFS
Compaq+Soft
Platform
KB/Sec
0
100
200
300
400
500
600
700
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 27
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #3: 20k Files & 100k Transactions
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
TMPFS
ODS/R0
ODS/R5
NFS/F330
NFS/F630
PPro+SOFT
P3/Linux
P3+DPT
2xP3+UFS
2xP3+SOFT
2xP3+MFS
U5+UFS
U5+Log
U5+TMPFS
PBG3/HFS+
PBG3/MFS
PBG4/HFS+
PBG4/MFS
Compaq+UFS
Compaq+Soft
Platform
KB/Sec
0
100
200
300
400
500
600
700
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 28
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #4: 20k Files, 100k Transactions, & 100 subdirectories
0.00
1,000.00
2,000.00
3,000.00
4,000.00
5,000.00
6,000.00
P3/Linux
2xP3+UFS
2xP3+SOFT
2xP3+MFS
U5+UFS
U5+Log
U5+TMPFS
PBG3/HFS+
PBG3/MFS
PBG4/HFS+
PBG4/MFS
Compaq+UFS
Compaq+Soft
Platform
KB/Sec
0
200
400
600
800
1,000
1,200
1,400
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 29
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #1: 1000 Files & 50k Transactions
268.32
1,566.72
332.02
3,174.40
235.40
489.66
280.35
1,638.40
346.92
3,317.76
245.96
511.62
82
485 492
980
72
149
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
3,000.00
3,500.00
PBG3/HFS+ PBG3/MFS PBG4/HFS+ PBG4/MFS Compaq+UFS Compaq+Soft
Platform
KB/Sec
0
200
400
600
800
1,000
1,200
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 30
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #2: 20k Files & 50k Transactions
82.00
737.57
99.14
1,495.04
38.95 66.57154.75
1,392.64
187.09
2,816.00
73.51125.63
29
349
36
666
15 23
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
3,000.00
PBG3/HFS+ PBG3/MFS PBG4/HFS+ PBG4/MFS Compaq+UFS Compaq+Soft
Platform
KB/Sec
0
100
200
300
400
500
600
700
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 31
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #3: 20k Files & 100k Transactions
90.20
819.89
94.42
1,628.16
42.68 69.72129.65
1,177.60
135.72
2,334.72
61.35100.21
31
310
33
613
15 22
0.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
PBG3/HFS+ PBG3/MFS PBG4/HFS+ PBG4/MFS Compaq+UFS Compaq+Soft
Platform
KB/Sec
0
100
200
300
400
500
600
700
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 32
PostMark Test Results
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
EastWestNorth
Postmark Test #4: 20k Files, 100k Transactions, & 100 subdirectories
65.43
708.74
77.21
1,484.80
38.81 40.1993.79
1,015.93
110.68
2,129.92
55.63 57.6124
277
28
574
14 130.00
500.00
1,000.00
1,500.00
2,000.00
2,500.00
PBG3/HFS+ PBG3/MFS PBG4/HFS+ PBG4/MFS Compaq+UFS Compaq+Soft
Platform
KB/Sec
0
100
200
300
400
500
600
700
Trans/Sec
Data ReadData WrittenTrans/Sec
2002-11-16 Copyright © 2002 by Brad Knowles 33
Test Results:Bonnie++ Sample Run
% bonnie++ -d /var/tmp/bonnie -n 1:10k:0 -m compaq-soft -r 48mWriting with putc()...doneWriting intelligently...doneRewriting…doneReading with getc()...doneReading intelligently...donestart 'em...done...done...done...Create files in sequential order...done.Stat files in sequential order...done.Delete files in sequential order...done.Create files in random order...done.Stat files in random order...done.Delete files in random order...done.Version 1.02d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CPcompaq-soft 300M 2407 67 4584 39 2052 21 1948 53 3644 18 57.8 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 1:10:0 402 31 +++++ +++ 832 49 564 46 +++++ +++ 1746 99compaq-soft,300M,2407,67,4584,39,2052,21,1948,53,3644,18,57.8,3,1:10:0,402,31,+++++,+++,832,49,564,46,+++++,+++,1746,99
2002-11-16 Copyright © 2002 by Brad Knowles 34
Bonnie++ ResultsBonnie++ Block I/O (Sequential Create)
181
150
76
402
340 331
50%
47%
23%
31% 32% 31%
0
50
100
150
200
250
300
350
400
450
compaq-ufs/1:10:0 compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/1:10:0
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesCreated/Sec
0%
10%
20%
30%
40%
50%
60%
%CPUBusy
Files Created/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 35
Bonnie++ ResultsBonnie++ Block I/O (Sequential Read)
704
165
670695
43%
10%
43%44%
0
100
200
300
400
500
600
700
800
compaq-ufs/20:10:0 compaq-ufs/20:10:0:100 compaq-soft/20:10:0 compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesRead/Sec
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
%CPUBusy
Files Read/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 36
Bonnie++ ResultsBonnie++ Block I/O (Sequential Delete)
215 188103
832
1,857
2,062
52%
48%
25%
49%
53%
60%
0
500
1,000
1,500
2,000
2,500
compaq-ufs/1:10:0 compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/1:10:0
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesDeleted/Sec
0%
10%
20%
30%
40%
50%
60%
70%
%CPUBusy
Files Deleted/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 37
Bonnie++ ResultsBonnie++ Block I/O (Random Create)
201
154
78
564
330350
57%
48%
24%
46%
33%35%
0
100
200
300
400
500
600
compaq-ufs/1:10:0 compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/1:10:0
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesCreated/Sec
0%
10%
20%
30%
40%
50%
60%
%CPUBusy
Files Created/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 38
Bonnie++ ResultsBonnie++ Block I/O (Random Read)
79
46
80 81
5%
2%
5% 5%
0
10
20
30
40
50
60
70
80
90
compaq-ufs/20:10:0 compaq-ufs/20:10:0:100 compaq-soft/20:10:0 compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesRead/Sec
0%
1%
2%
3%
4%
5%
6%
%CPUBusy
Files Read/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 39
Bonnie++ ResultsBonnie++ Block I/O (Random Delete)
170
40 39
1,746
1,8901,811
42%
7% 7%
99%
53% 53%
0
200
400
600
800
1,000
1,200
1,400
1,600
1,800
2,000
compaq-ufs/1:10:0 compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/1:10:0
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
FilesDeleted/Sec
0%
20%
40%
60%
80%
100%
120%
%CPUBusy
Files Deleted/Sec% CPU Busy
2002-11-16 Copyright © 2002 by Brad Knowles 40
Test Results:SmtpStone Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &% foreach I (1 2 3 4 5 6 7 8 9 10)foreach? /usr/bin/time smtp-source -l 10240 -f [email protected] \-t [email protected] -m 1000 -s 512 127.0.0.1:25foreach? end 8.06 real 0.22 user 1.65 sys 9.03 real 0.26 user 1.45 syssmtp-source: fatal: connect: Operation timed out 32.00 real 0.17 user 1.68 sys 13.08 real 0.19 user 1.43 sys 10.98 real 0.31 user 1.28 sys 9.02 real 0.24 user 1.52 sys 8.04 real 0.25 user 1.57 syssmtp-source: fatal: lost connection while reading server greeting 16.51 real 0.19 user 1.61 sys 10.99 real 0.19 user 1.53 sys 16.52 real 0.25 user 1.50 sys
2002-11-16 Copyright © 2002 by Brad Knowles 41
SmtpStone ResultsPowerBook G4 SmtpStone Performance (1000 messages)
68.19
84.20
154.16
208.29
150.78
79.10
39.98
19.989.98
5.00
7.58
8.05
0.980.64
0.420.15 0.10 0.07
0.46
0.000.00
50.00
100.00
150.00
200.00
250.00
512 256 128 64 32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
StandardDeviation
Msgs/Sec.Std.Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 42
SmtpStone ResultsPowerBook G4 SmtpStone Performance (20000 messages)
97.38 93.87100.96 101.69
93.57 96.08
123.55
188.79
237.28
155.99
79.20
39.83
29.34
38.23
32.92 32.42
21.14
35.78
19.76
9.34
5.10
2.581.24
2.56
0.00
50.00
100.00
150.00
200.00
250.00
16384 8192 4096 2048 1024 512 256 128 64 32 16 8
Number of Threads
MessagesperSecond
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
StandardDeviation
Msgs/Sec.Std. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 43
SmtpStone ResultsCompaq SmtpStone Performance (1000 Messages)
28.39
16.62
9.00
4.23
0.170.020.00
5.00
10.00
15.00
20.00
25.00
30.00
4 2 1
Number of Threads
MessagesperSecond
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 44
Test Results:Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &% postal -m 10k -p 64 -c 1 127.0.0.1 postal.user -time,messages,data(K),errors,connections,SSL connections23:05,7651,41262,0,7715,023:06,7795,41656,0,7795,023:07,7847,42305,0,7847,023:08,7616,41528,0,7616,023:09,7670,41659,0,7670,023:10,7413,39922,0,7413,023:11,7765,41886,0,7765,023:12,7752,41960,0,7752,023:13,7423,39835,0,7423,023:14,7475,39697,0,7475,023:15,7458,39981,0,7458,023:16,7462,40332,0,7462,023:17,7281,39294,0,7280,023:18,7408,40022,0,7409,023:19,7595,40882,0,7586,023:20,7737,41991,0,7746,023:21,7646,41122,0,7646,0^C
2002-11-16 Copyright © 2002 by Brad Knowles 45
Test Results:Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &% postal -m 10k -p 128 -c 1 127.0.0.1 postal.user -time,messages,data(K),errors,connections,SSL connections23:27,8283,43988,0,8411,023:28,8965,48165,0,8965,023:29,8952,48100,0,8952,023:30,8785,46803,0,8785,023:31,7948,42811,0,7933,0Server timed out on read.Server timed out on read.
[ .. .. Deletia .. .. ]
Server timed out on read.Server timed out on read.23:32,8016,43367,28,8059,023:33,8960,47748,0,8960,023:34,9054,48507,0,9022,023:35,8737,47175,0,8769,023:36,9057,48586,0,9057,023:37,8800,46722,0,8800,023:38,8933,48092,0,8933,023:39,8969,48155,0,8955,023:40,8984,48576,0,8998,0^C
2002-11-16 Copyright © 2002 by Brad Knowles 46
Test Results:Postal Sample Run
% sudo smtp-sink 127.0.0.1:25 1024 &% postal -m 10k -p 256 -c 1 127.0.0.1 postal.user -Usage: postal [-m maximum-message-size] [-p processes] [-l local-address] [-c messages-per-connection] [-r messages-per-minute] [-a] [-b [no]netscape] [-[z|Z] debug-file] [-s ssl-percentage] smtp-server user-list-filename conversion-filename
Postal Version: 0.61
% postal -m 10k -p 32 -c 1 127.0.0.1 postal.user -time,messages,data(K),errors,connections,SSL connectionsCan't connect to 127.0.0.1 port 25.Can't connect to 127.0.0.1 port 25.Can't connect to 127.0.0.1 port 25.Can't connect to 127.0.0.1 port 25.Can't connect to 127.0.0.1 port 25.
2002-11-16 Copyright © 2002 by Brad Knowles 47
Postal ResultsPowerBook G4 Postal Performance
145.77
126.46
79.69
39.92
19.97
9.98
378.02
164.94
27.5110.06 3.67 2.740.00
20.00
40.00
60.00
80.00
100.00
120.00
140.00
160.00
128 64 32 16 8 4
Number of Threads
MessagesperSecond
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
400.00
StandardDeviation
Msgs/SecStd.Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 48
Notes for Testing• When testing real MTAs, you need to watch more
than just the output from the testing program– Not unusual for MTAs to accept mail faster than they can
process it, especially under heavy load• Also watch
– Mail queue size• mailq | grep '[0-9][0-9]:[0-9][0-9]' | wc -l
– CPU load• iostat 5
– Memory utilization• vmstat 5
2002-11-16 Copyright © 2002 by Brad Knowles 49
Notes for Postfix
• Make sure that you modify/etc/postfix/main.cf to be suitable– Need to boost
• local_destination_concurrency_limit• default_destination_concurrency_limit
• Check /etc/postfix/master.cf to makesure there are no artificial limits on processeslike smtp, smtpd, bounce, cleanup,rewrite etc…
2002-11-16 Copyright © 2002 by Brad Knowles 50
Notes for MacOS X
• Unfortunately, Apple broke iostat– Still mostly okay for what we want to watch
(CPU user %, system %, and idle %)• Apple doesn’t even provide vmstat
– vm_stat is not the same program,although it is similar
2002-11-16 Copyright © 2002 by Brad Knowles 51
Postfix SmtpStone ResultsPowerBook G4 Postfix SmtpStone Performance (1000 messages)
57.22
52.26
39.50
19.98
9.92
4.99
0.64
0.84
0.13
0.08
0.85
0.29
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 52
Postfix SmtpStone ResultsCompaq+UFS SmtpStone Performance (1000 messages)
10.95
10.36
6.41
0.48
0.33
1.14
0.00
2.00
4.00
6.00
8.00
10.00
12.00
4 2 1
Number of Threads
MessagesperSecond
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 53
Postfix SmtpStone ResultsCompaq+Soft SmtpStone Performance (1000 messages)
10.65
9.40
6.14
1.55
0.95 0.94
0.00
2.00
4.00
6.00
8.00
10.00
12.00
4 2 1
Number of Threads
MessagesperSecond
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 54
Postfix SmtpStone ResultsCompaq Postfix SmtpStone Results (20000 messages/4 Threads)
7.53
11.08
303.20
17.72
0.00
2.00
4.00
6.00
8.00
10.00
12.00
compaq-ufs compaq+soft
Platform-Filesystem
MessagesperSecond
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 55
Sendmail SmtpStone ResultsPowerBook G4 Sendmail SmtpStone Results (1000 messages)
5.37
18.59
14.8914.36
9.69
4.97
0.37
18.19
4.75
1.921.47
0.680.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 56
Sendmail SmtpStone ResultsPowerBook G4 Sendmail+Async SmtpStone Results (1000
messages)
5.36
18.01
9.92
4.97
2.48
1.250.12
0.42
0.20
0.10
1.16
0.38
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 57
Sendmail SmtpStone ResultsCompaq Sendmail-UFS SmtpStone Results (1000 messages)
3.65
5.02
4.56
3.92
2.49
9.67
31.04
4.17 4.04
14.08
0.00
1.00
2.00
3.00
4.00
5.00
6.00
16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
StandardDeviation
Msgs/Sec.Std. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 58
Sendmail SmtpStone ResultsCompaq Sendmail+Soft SmtpStone Results (1000 messages)
5.91
6.61
4.10
2.81
2.35
82.20
72.11
10.93
62.62
20.37
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
StandardDeviation
Msgs/Sec.Std. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 59
Performance Tuning: Syslog• Problem
– Local syslog can cause excessive load on busy mailsystem
• Solutions– Configure syslog to write to logs asynchronously
• Perhaps via a “–” (dash) before the filename– Run syslog on a central log server, across the network
• Also allows you to perform more in-depth log processing,without interfering with mail services
• Generally logs are much easier to manage centrally• Can also combine with other central syslog processing
applications– E.g., Addamark Log Management System
2002-11-16 Copyright © 2002 by Brad Knowles 60
Performance Tuning: Syslog
• Problem– Classic syslog uses UDP, can lose 75%
or more of packets sent across the network• Solution
– Replace classic syslog with nsyslog,ssyslog, or syslog-ng and use TCPinstead of UDP
• Depending on your particular requirements forhardware, OS, environment, etc….
2002-11-16 Copyright © 2002 by Brad Knowles 61
Postfix SmtpStone ResultsPowerBook G4 (no syslog) Postfix SmtpStone (1000 messages)
26.78
60.89
52.04 52.59
36.32
19.30
9.96
4.98
1.41
1.67
1.23
0.80
2.06
2.80
0.350.47
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
128 64 32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
0.50
1.00
1.50
2.00
2.50
3.00
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 62
Sendmail SmtpStone ResultsPowerBook G4 Sendmail+Async+NoSyslog SmtpStone Results
(1000 messages)
5.24
17.20
9.92
4.97
2.50
1.24
7.10
0.100.36
1.76
0.12
5.50
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
32 16 8 4 2 1
Number of Threads
MessagesperSecond
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
StandardDeviation
Msgs/SecStd. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 63
Sendmail SmtpStone ResultsCompaq Sendmail-UFS-NoSyslog SmtpStone Results
(1000 messages)
5.64
4.46
3.63
2.30
24.92
2.31
8.44
11.43
0.00
1.00
2.00
3.00
4.00
5.00
6.00
8 4 2 1
Number of Threads
MessagesperSecond
0.00
5.00
10.00
15.00
20.00
25.00
30.00
StandardDeviation
Msgs/Sec.Std. Dev.
2002-11-16 Copyright © 2002 by Brad Knowles 64
Conclusions• This process can be difficult and time-
consuming to get right, but it pays off• Graphs & charts can help a great deal
– But they don’t tell the whole story• Software
– Postfix is a lot faster than even I had thought– Wasn’t able to get sendmail tuned for the
performance I’m confident it can deliver– Benchmarking tools (esp. Bonnie++ and postal)
need more work to be useful
2002-11-16 Copyright © 2002 by Brad Knowles 65
Conclusions
• Use the Scientific Method– Try to set aside personal bias– Trust only what you can prove– Process
• Generate Hypothesis• Generate test for Hypothesis• Test Hypothesis
– Prove or Disprove• If disproven, modify or generate new
Hypothesis
2002-11-16 Copyright © 2002 by Brad Knowles 66
Related Work
• Nick Christenson– sendmail Performance Tuning book
http://www.jetcafe.org/~npc/book/sendmail/– “Performance Tuning Sendmail Systems”
slideshttp://www.jetcafe.org/~npc/doc/performance_tuning.pdf
• Brad Knowles– “Sendmail Performance Tuning for Large
Systems” slideshttp://www.shub-internet.org/brad/papers/sendmail-tuning/
2002-11-16 Copyright © 2002 by Brad Knowles 67
Related Work
• Matthias Andreehttp://www-dt.e-technik.uni-dortmund.de/~ma/postfix/vsqmail.htmlhttp://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html
• Nakamura Motonorihttp://www.kyoto.wide.ad.jp/mta/eval1/eindex.html
• Ciprian Ascantehttp://www.benchmarks.dmz.ro/index.php?topic=LinMSB