lessons learned from real life
DESCRIPTION
Lessons Learned from Real Life. Jeremy Elson National Institutes of Health November 11, 1998. USC!. quick bio. Hi, I’m Jeremy. Nice to meet you. 1996: BS Johns Hopkins, Comp Sci Sep 96 - Sep 98: Worked at NIH full-time - PowerPoint PPT PresentationTRANSCRIPT
1
Lessons Learned from Real Life
Jeremy Elson
National Institutes of Health
November 11, 1998
2
quick bio• Hi, I’m Jeremy. Nice to meet you.
• 1996: BS Johns Hopkins, Comp Sci
• Sep 96 - Sep 98: Worked at NIH full-time– Led software development effort on a small team
developing an ATM-based telemedicine system called the Radiology Consultation WorkStation (RCWS)
• Sep 98: Decided to return to school full-time
• Nov 98: Gave a talk to dgroup about interesting lessons learned during development of the RCWS
3
my talk
• Very quick description of the RCWS
– In future dgroups, I can give a talk about the RCWS, or about ATM, if there is interest
• Some pitfalls and fallacies in networking I discovered while developing the RCWS
• Techniques for network problem solving
4
Radiology Consultation Workstation Network
5
RCWS Block Diagram
6
an unintended test
• Initial Configuration: 2 Sparc 20’s w/50MHz CPUs; Solaris 2.5.1; Efficient Networks ATM NICs @155MHz, LattisCell 10114-SM– TTCP memory-to-memory: 60 Mbps
• Upgrade to 75MHz chips, otherwise identical– TTCP now reports 90Mbps!
• 50% upgrade in CPU speed led to exactly 50% increase in network throughput
7
pitfall: infinite CPU• In many systems, the network is the bottleneck;
we have “infinite” CPU in comparison. We try to use CPU to save network bandwidth:– Compression– Multicast– Caching (sort of)
– Micronet design
• Pitfall: Assuming this is always true. In our ATM app, compression might slow it down!
8
a surprising outcome• There are various ways of doing IP over ATM
– “Classical IP” MTU ~9K
– “LANE” MTU 1500 bytes (for Ethernet bridging)
• Which would you expect would have better bulk TCP performance, and by how much?
• Classical IP did better -- by a factor of ~5! I didn’t believe it at first.
• Turned out that both were sending roughly the same packets/sec; CLIP: more bytes/packet
9
pitfall: networks run out of bandwidth first
• The number of bytes per second is only one metric; consider packets per second also. This is sometimes the wall you hit first.
• Fixed packet processing cost appears to far outweigh the incremental cost to transmit more bytes as part of the same packet
• This fits nicely with the previous observation: CPU is only fast enough for n packets/sec
• This is old news to Cisco, backbone ISPs, etc.
10
pathological networks
• We built an on-campus ATM network and bought access to a MAN (ATDnet), but the only WAN available was the ACTS satellite
• Our network was very long and very fat: OC3 (155 Mb/sec) over satellite (500ms RTT).
• We were expecting standard LFN-related problems; the solutions are fairly well-known (window scaling, PAWS, SACK, etc.)
• What surprised me was something else: interactive performance!
Earth
ACTS Satellite
Request
To perform actions suchas screen updates, requestsmust go through a server.Therefore the user responsetime will be ~RTT.
Reply1/8 of a second fromEarth to a geostationarysatellite; RTT ~1/2 second(plus ground switching delay & queuing delay)
12
the best laid plans• Requests are small messages (<100 bytes)
transmitted using TCP over ATM
• Everything seemed to work fine on-campus
• Over the satellite, we were expecting to see delays of 1/2 sec in command execution
• Instead we saw >1 second delays: much more than we were expecting & hard to use. Uh oh.
• My job (with 2 hours of satellite time ticking away…): figure out why this was happening
13
the answer: tcpdump
• ‘tcpdump’ is a packet-sniffer written by Steve McCanne, Craig Leres, and Van Jacobson at LBL
• Monitors a LAN in realtime; prints info about each packet (source/dest, sequence numbers, flags, acknowledgements, options)
• Runs on most UNIX variants
• The most spectacularly fantastically wonderful network debugging tool on planet Earth; my knee-jerk reaction whenever there is any problem is to fire this up first
14
tick, tock, tick, tock...
TimeClient Server
ACK 28 Server’s TCP stack ACKs that data
28 bytesClient sends data to server
ACK 42 Server ACKs the new 42 bytes
42 bytesClient sends more data as soon
as the ACK is received
28 bytes Server application has now received a complete 70 byte message; sends reply
42 bytes Server sends new data after it receives the client’s ACK
ACK 28Client TCP stack ACKs
RTT
At the application layer, messages are 70 bytes long.
ACK 42Client TCP stack ACKs
USER SEES RESPONSE HERE
15
the nagle finagle• Each application-layer message is split into 2
segments. Why?– Because the app was calling write() twice
• For some reason, the second half isn’t sent until the first half is ACKed! Why?– The Nagle Algorithm, which says “don’t send a
tinygram if there is an outstanding tinygram.”
• Users had to wait 3 RTTs instead of 1
• Short term fix: turn off the Nagle Algorithm (setsockopt TCP_NODELAY in Solaris)
• Long term fix: rewrite the message-passing library to use writev() instead of write().
16
pitfall: don’t care how TCP and app get along
• It’s easy to think of TCP as a generic way of getting things from Here to There; sometimes, if we look deeper, we find problems
• Good example: HTTP interactions with TCP study by Touch, Heidemann & Obraczka
• Of course, different TCP implementations react differently. (Maybe some TCPs wait before launching and would have hidden this.)
17
the big mystery
• Remember: 90 Mbps Sparc 20 to Sparc 20• Scenario: Two machines doing FTP (to /dev/null)
– Machine A: Sun Ultra-1 running Solaris 2.5.1, 155 Mbps fiber ATM NIC
– Machine B: Fast Pentium-II running Windows NT 4.0, 25 Mbps UTP ATM NIC
– Using LANE, 1500 byte MTU
• Transmitting from A to B: 23 Mbps
• Transmitting from B to A: 8 Mbps!! Why?
18
tcpdump to the rescue
Time
A B
data 1460 MSS-sized segment from sender
win 64KWindow advert. from receiver
data 1460 Another MSSdata 1176 Smaller segment from sender
… more segments (not shown)
data 1460 Another segment from sender
ACK/win 64K
Receiver finally ACKsCycle starts again
long quiet time - no activity
19
observations aboutour mystery
• Sending A to B (the 22Mbps case), machine generated only MSS segments; B to A did not. (Could account for some slowdown.)
• The ACKs from A all came at very regular intervals (~50ms)
• Data came quickly (say, all in about 20ms) followed by long quiet time (say, 30ms)
• What’s going on????
20
deferred ACKs
• When we receive data, we wait a certain interval before sending an ACK
• This attempts to reduce traffic generated by interactive (keystroke) activity by hoping a new window and/or data will be ready, too
• We don’t want to do this with bulk data (defined as 3 MSS’s in a row)
21
keystrokes: the worst case
Time
User Server
Assume both sides are initially advertising Win = 100
ACK ‘a’, win 99TCP stack sends ACK
win 100 telnet daemon wakes up; reads char
‘a’ (echoed) telnet daemon sends echoed char
ACK ‘a’, win 99
TCP stack sends ACK
win 100telnet client wakes up; reads char
‘a’User types a character
22
keystrokes: what we want
Time
User Server
Assume both sides are initially advertising Win = 100
ACKa, a, win100 telnet daemon sends ACK of receivedchar, echoed char, and open window
ACK a, win 100
telnet client wakes up; reads char
‘a’User types a character
Deferred ACK interval: don’tsend an ACK right away; wait,and hope that we have a newwindow and echoed char ready
23
another look at the trace
Time
A B
data 1460 MSS-sized segment from sender
win 64KDeferred ACK interval expires
data 1176 Smaller segment from sender - whichfools the receiver into thinking that weare not doing bulk data transfer
… more segments (not shown)
data 1460 Smaller segment from sender WINDOW IS NOW CLOSED
win 64KTimer expires; receiver sends ACK
Cycle starts again
long quiet time - no activity
24
the mystery unmasked
• Only observable because all of the following were true (take out 1, the problem vanishes)– Receiver using deferred ACKs– Sender not sending all MSS sized data– Bandwidth high enough and window small enough so
that the window can be filled before the deferred ACK interval expires (rare at 10mbps)
• When I turned off the deferred ACKs on the receiver, bandwidth jumped to 23 Mbps. (Under Solaris this can be done with ndd)
25
tcpdump: our best friend• Virtually impossible to figure out problems
like the previous one by just puzzling it out
• Reading about how protocols work is a good starting point; implementing them gives you even more. But…
• Nothing gave me more intimate knowledge of TCP than seeing it come alive. Not looking at high level behavior, but actually watching packets fly across the wire
• Different stacks have different personalities
• TCP/IP Illustrated v1 is great to learn how
26
other uses of tcpdump• Keeping my ISDN router from dialing• Widespread teardrop attack on NIH (I patched
tcpdump to make this easier)• Netscape SYN bug• Samba hitting DNS• Inoculan directed broadcasts• Diagnosing dead and/or segmented networks• Even rough performance measurement• The network people thought I was a magician!
27
summary:lessons learned
I. Thou shalt not assume that thy CPU is infinite in power, for thy network mayindeed be more plentiful.
II. Thou shalt take mind of the number ofpackets thou sendeth to thy network;
for, yea, a multitude thereof may wreak havoc thereupon.
28
summary:lessons learned
III. Thou shalt read the Word of Stevens in TCP/IP Illustrated, and become learnedin the ways of tcpdump, so that thy daysof network debugging shall be pleasant and brief.
IV. Thou shalt watch carefully the packets that thy applications create, so that TCP may
be thy servant and not thy taskmaster.
29
that’s all, folks!