Interactions Between Delayed Acks and Nagle’s Algorithm in HTTP and HTTPS: Problems

and Solutions

Arthur Goldberg

Robert Buff

New York University

March 1999

2

Client Browser Web ServerTCP/IPApplication

W1: write() less than MSS

W2: write() less than MSS

175

ms

(Win

32)

Ack

del

ay

Network

20 m

s

TCP segment with W1

TCP segment with W2

ACK for W1

Nagle - Delayed Ack Interaction

TCP/IP Application

read() W1

read() W2

write() W1/W2 response

read() W1/W2 response

segment with response

3

Observations

• Applies to request/response applications other than the Web

• Diagram to scale, and generous with message travel times

• Client blocks until receipt of response• ACK delay depends on TCP

implementation• Occurs with client/server roles reversed, too

4

Solution Opportunities: In Order of Increasing Implementation Difficulty and Impact

• Application: Change buffering, disable Nagle

• Sockets API: Broaden semantics

• TCP implementation: Decrease ACK delay

• TCP standard: Change Nagle

5

Network Communications Goals

• Application layer: minimize median response time - deliver data to a receiver as soon as it’s ready

• Transport and Network layers: avoid gross inefficiencies - collect data into the largest possible segments

6

Assumptions

• Network– modest delay– high bandwidth– minimal packet lost

• Application– client/server– many modules

7

The Silly Window Syndrome (SWS)

Sender Receiver

TCP window fullread 1 byte

read 1 byte

data

ack

Advertise 1 byte of available window

1 byte of data

ack

Advertise 1 byte of available window

1 byte of data

Overhead: 3 packets and 3*40 = 120 bytes of TCP/IP headers transport 1 byte of data!

write() some

8

TCP’s SWS Avoidance

• Receiver– avoid advertising small TCP window

advances– delay acknowledgements

• Sender– Nagle algorithm: delays transmission of

partially filled segments until all previously transmitted data has been acknowledged

9

Receiver SWS Avoidance in the TCP Spec, RFC 1122

• The receiver's SWS avoidance algorithm determines when the right window edge may be advanced;– For realistic receive buffers (greater than twice the

MSS) window advances are announced in increments of MSS.

• A TCP SHOULD implement a delayed ACK … the delay MUST be less than 0.5 seconds, and in a stream of full-sized segments there SHOULD be an ACK for at least every second segment.

10

Sender SWS Avoidance in the TCP Spec

• A TCP SHOULD implement … [Nagle 84] to coalesce short segments. However, there MUST be a way for an application to disable the Nagle algorithm on an individual connection.

• The Nagle algorithm is …: If there is unacknowledged data … then the sending TCP buffers all user data … until the outstanding data has been acknowledged or until the TCP can send a full-sized segment …

11

Test Application• Execute 100 iterations

System calls Typical

Client Server Applicationsemantics

TCPsegments

connect() listen(); accept() TCPhandshake

write() request (16 B) Dataread()write() First part of

response (10 B)

Data

read() ack(optional)

write() Second part ofresponse (10 B)

Data

read() ack(optional)

12

LabMachines connected by unloaded 100 Mbps Ethernet

* Win95 separated by 1 router

Name Processor Operating SystemNT4Wa Pentium, 100 MHz NT Workstation 4.00.1381NT4Wb Pentium, 200 MHz NT Workstation 4.00.1381NT4S Pentium, 233 MHz NT Server 4.00.1381, SP 3NT5S Pentium, 233 MHz NT Server 5.00.1671, beta 1Win95 Pentium, 100 MHz Windows 95Win98 Pentium, 90 MHz Windows 98Linux i486, 66 MHz Linux 2.0.31Solaris Sun SPARCstation 5 SunOS 5.6

13

Linux/NT, Nagle OffSegment,payload [bytes]

Delta[ms]

1 TCP handshake

2 0.5 0.5 ms cumulative

3 0.9 1.4

4 Request, 16 9.7 11.1

5 First response, 10 1.5 12.6

6 Second response, 10 1.2 13.8

7 (ack) 13.6 27.4

8 Request, 16 32.5 59.9

9 First response, 10 1.3 61.2

10 Second response, 10 1.2 62.4

and so on

14

NT/NT, Nagle OnSegment,payload [bytes]

Delta[ms]

1 TCP handshake

2 0.3 0.3 ms cumulative

3 0.3 0.6

4 Request, 16 9.1 9.7

5 First response, 10 3.6 13.3

6 (ack) 114.4 127.7

7 Second response, 10 0.2 127.9

8 Request, 16 8.3 136.2

9 First response, 10 3.5 139.7

10 (ack) 188.3 328.0

11 Second response, 10 0.2 328.2

and so on

15

Solaris/NT, Nagle OnSegment,payload [bytes]

Delta[ms]

1 TCP handshake

2 0.4 0.4 ms cumulative

3 0.7 1.1

4 Request, 16 3.1 4.2

5 First response, 10 0.8 5.0

6 (ack) 0.6 5.6

7 Second response, 10 0.3 5.9

8 Request, 16 2.1 8.0

9 First response, 10 0.8 8.8

10 (ack) 41.8 50.6

11 Second response, 10 0.3 50.9

and so on

16

Summary

Delay of ack [ms] Transaction [ms]

Client Server Nagle on Nagle off Nagle on Nagle off

NT4Wa NT4S 187.3 0.4 191.4 7.9

NT4S NT4Wa 187.0 0.2 191.6 3.0

NT4Wb Win95 193.1 194.1 193.1 199.0

Win95 NT4Wb 208.8 0.7 209.5 2.1

NT4Wb Win98 192.5 193.8 198.1 199.8

Win98 NT4Wb 192.5 1.0 192.9 1.5

NT4Wa NT5S 195.9 0.3 196.7 0.7

NT5S NT4Wa 197.4 0.2 198.6 1.6

NT4Wa Solaris 194.2 0.3 196.7 3.0

Solaris NT4Wa 46.2 0.6 47.5 4.3

NT4Wa Linux 188.7 0.3 191.8 3.9

Linux NT4Wa 15.8 –––– 21.5 4.4

17

ACK delays

Linux 2 Adjusted to interarrivalspacing, observed at 15 to20 ms

Solaris 50 ms mean (100 ms intervaltimer, according to Paxson)

Win32 Maximum 200 ms, increaserapidly during connection

18

HTTPS, new session keySegment,payload [bytes]

Delta[ms]

1 TCP handshake

2 0.2

3 0.4

4 SSL, 93 203.3

Initial 0.6 ms for TCP handshake and 203.3 ms for SSLcalculations at the server are omitted.

5 SSL, 726 1.4 205.3 ms cumulative

6 SSL, 204 88.5 293.8

7 SSL, 6 116.2 410.0

8 (ack) 156.1 566.1

9 SSL, 61 0.3 566.4

10 HTTPS Request, 46 10.7 577.1

11 HTTPS Resp 1, 306 3.3 580.4

12 HTTPS Resp 2, 1021 1.3 581.7

13 (ack) 1.5 583.2

14 (ack) 6.4 589.6

15 (ack) 0.2 589.8

19

HTTPS, session key reusedSegment,payload [bytes]

Delta[ms]

1 TCP handshake

2 0.2 0.2 ms cumulative

3 0.3 0.5

4 SSL, 108 11.1 11.6

5 SSL, 79 1.4 13.0

6 (ack) 133.1 146.1

7 SSL, 67 0.3 146.4

8 SSL, 67 4.5 150.9

9 (ack) 182.8 333.7

10 Request, 46 0.4 334.1

11 HTTPS Resp 1, 306 3.1 337.2

12 HTTPS Resp 2, 1021 1.3 338.5

13 (ack) 1.5 340.0

14 (ack) 6.0 346.0

15 (ack) 0.1 346.1

20

HTTPS Potential Savings

SSL handshake Entire HTTPS transactionSession key

Delay No delay Slowdown Delay No delay Slowdown

New 565 ms 409 ms 38 % 589 ms 433 ms 36 %

Reused 333 ms 17 ms 1826 % 346 ms 30 ms 1046 %

21

HTTP, GIF from Netscape

Segment,payload [bytes]

Delta[ms]

1 TCP handshake

2 3.0 3.0 ms cumulative

3 0.4 3.4

4 HTTP GET, 390 1.7 5.1

5 HTTP RESP, 1944 10.5 15.6

6 HTTP GET, 414 164.3 179.9

7 HTTP RESP, 354 8.8 188.7

8 (ack) 199.4 388.1

9 HTTP RESP, 3255 12.9 401.0

and so on

22

Application AlternativesStrategy Approx

Appdelay

IPpkts

Issues

Writev RTT/2 1 Ideal performance, butimpossible to implement if manymodules write

Coalescedwrites

RTT/2 1 Like writev, but more possiblycopying

Two writes,Nagle off

RTT/2 +

2 Easy to build; low delay; risky, aspacket count can surge withmany writes

Two writes ACKdelay +3RTT/2

3 slow, 3 IP packets

23

Alternative: Disable Nagle on the last write of an application message

• Advantages– easy to do– removes delayed ACK from critical path– eliminate risk of small packets flood

• Disadvantages– two extra system calls per application message– possibly one extra IP packet– complicated for a programmer

24

Related Work• Nagle - delayed ACK interactions widely

recognized, e.g. Stevens 98, Microsoft 98 and Sun 98

• P-HTTP recorded by Heidemann 97

• Nielsen 97, 98: test a typical HTTP/1.1 server page load

– (Client ran on a DEC Alphastation 400 4/233, UNIX 4.0a)

Situation Time (sec) Time (sec)Nagle 0.48 0.27NoNagle 0.45 0.21

25

Possible Solutions

• Sockets API– Include a flush

• TCP implementation– Decrease ACK delay, especially in Win32

• TCP standard– Desensitize SWS avoidance

• SWS avoidance clearly being triggered prematurely and introducing unnecessary delays

26

Nagle modification proposed

• Minshall, “A Suggested Modification to Nagle's Algorithm”, http://search.ietf.org/internet-drafts/draft-minshall-nagle-00.txt

• If a TCP has less than a full-sized packet to transmit, and if any previous less than full-sized packet has not yet been acknowledged, do not transmit a packet.

• Will not solve SSL problem above

27

Todos

• Measure production delayed ACK costs

• Examine other application protocols

• Develop high-performance sockets

Interactions Between Delayed Acks and Nagle’s Algorithm in HTTP and HTTPS: Problems

and Solutions

Arthur Goldberg

Robert Buff

New York University

March 1999