Владимир Кириллов-tcp-performance for-mobile-applications
TRANSCRIPT
TCP PerformanceFor Mobile Applications
Vladimir Kirillov@darkproger
Networking Stack
Data Link
Network
Transport
Application
Session
Data Link
Network
Transport
Application
Session
WiFi Edge 3G LTE
IP
TCP
HTTP
TLS
Data Link
Network
Transport
Application
Session
WiFi Edge 3G LTE
IP
TCP
HTTP
TLS
hardware
kernel
SOCK_STREAM
NSURLConnection
OpenSSL
Level Protocol API / Implementation
WiFi Edge 3G LTE
IP
TCP
HTTP
TLS
hardware
kernel
SOCK_STREAM
NSURLConnection
OpenSSL
ptrace
socket API
bpf(4)
dtrace
Protocol API / Implementation Introspection
capturing iPhone traffic
% udid=$(system_profiler SPUSBDataType \ | awk '/iPhone/{go=1} /Serial/ {if (go) print $3; go=0}')
276cb9530201bcehelloworldcd55560ed015d00
% rvictl -s $udidStarting device 276cb9530201bcehelloworldcd55560ed015d00[SUCCEEDED]
% ifconfig rvi0rvi0: flags=3005<UP,DEBUG,LINK0,LINK1> mtu 0
tcpdump -i lo0 -w t.pcap -s0 &nc -l 5000 &echo hello | nc localhost 5000kill %1
# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [S], seq 453038127, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0
2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [R.], seq 0, ack 453038128, win 0, length 0
2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [S], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 02012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [S.], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 303407352,sackOK,eol], length 02012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [P.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 62012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511354 IP (tos 0x0, ttl 64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511367 IP (tos 0x0, ttl 64, id 20879, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
17 packets captured
# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [S], seq 453038127, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0
2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [R.], seq 0, ack 453038128, win 0, length 0
2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [S], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 02012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [S.], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 303407352,sackOK,eol], length 02012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [P.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 62012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511354 IP (tos 0x0, ttl 64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511367 IP (tos 0x0, ttl 64, id 20879, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 02012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
17 packets captured
^^% stat -f %z t.pcap1306
% tcptrace t.pcap
17 packets seen, 17 TCP packets tracedelapsed wallclock time: 0:00:00.001344, 12648 pkts/sec analyzedtrace file elapsed time: 0:00:00.000305TCP connection info:
1: localhost:52132 - localhost:5000 (a2b) 1> 1< (reset)2: localhost:52133 - localhost:5000 (c2d) 8> 7< (complete) (reset)
% tcptrace -o2 -l t.pcap...
adv wind scale: 4 adv wind scale: 4 req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0 urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 16344 bytes mss requested: 16344 bytes max segm size: 6 bytes max segm size: 0 bytes min segm size: 6 bytes min segm size: 0 bytes avg segm size: 5 bytes avg segm size: 0 bytes
max win adv:146976 bytes max win adv: 146976 bytes
min win adv:146976 bytes min win adv: 146976 bytes
zero win adv: 0 times zero win adv: 0 times avg win adv: 146976 bytes avg win adv: 122480 bytes
initial window:6 bytes initial window: 0 bytes initial window: 1 pkts initial window: 0 pkts ttl stream length: 6 bytes ttl stream length: 1 bytes
missed data: 0 bytes missed data: 1 bytes truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 0.000 secs idletime max: 0.1 ms idletime max: 0.0 ms throughput: 27027 Bps throughput: 0 Bps
endpoint
SO_RCVBUF
SO_SNDBUF
endpoint
SO_RCVBUF
SO_SNDBUF
endpoint
SO_RCVBUF
SO_SNDBUF
endpoint
SO_RCVBUF
SO_SNDBUF
SEG
SEG
endpoint
SO_RCVBUF
SO_SNDBUF
endpoint
SO_RCVBUF
SO_SNDBUF
LATENCY
SEG
SEG
BANDWIDTH
2 * LATENCY = RTT
Latency
• Time from one endpoint to another• Each connection spans multiple links• edge• ethernet• fiber• wifi• latency = sum (lat foreach link)• RTT = 2 * latency
Bandwidth • Number of bytes a link can handle• bw = min (bw foreach link)
BandwidthDelay
ProductBDP = RTT * BANDWIDTH
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUF
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
SEG SEGSEG SEG
receiver windowSEGSEG SEG SEGSEG SEG
TCPbyte
stream
• stateful• ordered• reliable• managed
IP
TCP
HTTP
TLShas state
no state
paired
TLS
% openssl s_client -showcerts -connect google.com:443CONNECTED(00000003)depth=1 /C=US/O=Google Inc/CN=Google Internet Authorityverify error:num=20:unable to get local issuer certificateverify return:0---Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com i:/C=US/O=Google Inc/CN=Google Internet Authority... 1 s:/C=US/O=Google Inc/CN=Google Internet Authority i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority...SSL handshake has read 2388 bytes and written 316 bytes
TLS
% openssl s_client -showcerts -connect ciklum.com:443CONNECTED(00000003)
depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.netverify error:num=20:unable to get local issuer certificateverify return:1depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.netverify error:num=27:certificate not trustedverify return:1depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.netverify error:num=21:unable to verify the first certificateverify return:1---Certificate chain 0 s:/C=UA/OU=Domain Control Validated/CN=*.ciklum.net i:/O=AlphaSSL/CN=AlphaSSL CA - G2...Server certificatesubject=/C=UA/OU=Domain Control Validated/CN=*.ciklum.netissuer=/O=AlphaSSL/CN=AlphaSSL CA - G2
SSL handshake has read 1854 bytes and written 328 bytes
lol :)
IP
TCP
HTTP
TLS
1 RTT
3 RTTs
1 RTT
It takes 4 RTTs to serve a HTTPS request
IP
TCP
HTTP
TLS
1 RTT
3 RTTs
1 RTT
It takes 4 RTTs to serve a HTTPS request
TCP Reliability
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
SEG SEGSEG SEG
receiver windowACK
ACK
ACK
ACK
AirPortExpress
router
router
router
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
SEG SEGSEG SEG
receiver windowACK
ACK
ACK
ACK
AirPortExpress
router
router
router
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
SEG SEGSEG SEG
receiver windowACK
ACK
ACK
ACK
retransmit on timeout
TCP Congestion Control
SO_RCVBUF
sender window
receiver window
server
SO_RCVBUF
SO_SNDBUFSEG
client
SO_SNDBUF
SEGSEGSEG SEG SEGSEG
SEG SEG SEG SEG
sender windowreceiver window
AirPortExpress
overloadedrouter
router router
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG SEG SEGSEG SEG
ACK
ACK
ACK
^^^ What congestion control is actually designed for
AirPortExpress
router
router
router
sender window
receiver window
client
SO_RCVBUF
SO_SNDBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
SEGSEG
SEG SEGSEG SEG
receiver windowACK
ACK
ACK
ACK
SEG
SEG
SEG
SEG
^^^ What actually happens on mobile devices
TCP Artifacts
• Nagle algorithmwhile (1)write(fd, “5”, 1);
(telnet syndrom)
Delayed ACKhttp://www.stuartcheshire.org/papers/NagleDelayedAck/
TCP Artifacts
•SO_OOBINLINE•TCP URG
API Issues
API IssuesAsync NSURLConnectionUIScrollViewCFRunLoopAddCommonMode
SO_RCVBUF
sender window
receiver windowSO_RCVBUF
server
SO_RCVBUF
SO_SNDBUFSEGSEG
client
SO_SNDBUF
SEG SEGSEGSEG SEG SEGSEG
SEG
MoreCongestion
Control
• Additive Increase• Multiplicative Decrease• Slow Start
iOS / Unix
• setsockopt(2)• adjust window size• socket buffer sizes • TCP_NODELAY• etc•getsockopt(2)• monitoring
• low-latency responding to socket events• do not let the buffer stay full
Speedup
• Responsive UI• Bad example: Instagram• Bad example: VK• best UI• worst reliability
Speedup
• TCP Fast Open• Linux 3.6• HAProxy
• TCP/NC• TCP and math (maths)• http://dspace.mit.edu/openaccess-
disseminate/1721.1/58796• TCP Westwood+
Speedup
• TLS False Start• TLS NPN• Next Protocol Negotiation
• HTTP Pipelining• SPDY
kthxbai@darkproger
http://kirillov.im