Slide 1

Wyatt Lloyd, Michael J. Freedman Princeton University TRODS Transparent Recovery for Object Delivery Services Presented at DSN-DCCS 2011 in Hong Kong on 6/28/11

Slide 2

2

Slide 3

3

Slide 4

4

Slide 5

Client Server Service Server Connection Recovered! 5

Slide 6

Object Delivery Services Read-Only Static Content Webpages, Images, Videos 6

Slide 7

Work Now Cant Modify Clients 7

Slide 8

Key Idea Coerce client to help To identify connections that need recovery To reliably store information Yet client is unmodified and unaware Exploit TCP spec to control clients stack 8

Slide 9

Object Delivery Cluster 9 Server Service Server Liveness Monitor Load Balancer

Slide 10

Failure 10 Server Service Server Liveness Monitor Load Balancer

Slide 11

TRODS 11 Client Server Service Server Liveness Monitor Load Balancer ?

Slide 12

TRODS 12 Client Server Service Server Liveness Monitor Load Balancer ? Store

Slide 13

Road to Recovery StepTechnique Redirect to live server.Liveness monitor updates load balancer Induce client to send packetCoerce clients TCP stack Continue Connection Determine PhaseUse packet + stored info Identify Object.Stored Info Find Offset..Use packet + stored info 13 New

Slide 14

Coercing Clients Always Leave A Packet Unacknowledged SYNSYN/ACKACKRequestResponse 1 ACKResponse 2 ACKResponse 3 FIN/ACKFIN Client Server 14 SYNRequest Retransmit Queue SYN/ACKResponse 1 Response 2 Response 3 FIN/ACK FIN ACK Always Something Here Exploit TCP Spec for Recovery Initiation!

Slide 15

Continuing the Connection Determine Phase: 1)TCP Setup 2)HTTP Setup 3)HTTP Download 4)TCP Teardown 15 TRODS Saves Info

Slide 16

Continuing the Download HTTP ObjectID Offset 16 = TCP Ack HTTP ObjectISN HTTP ObjectISNTCP Ack TCP ISN HTTP Resp Header HTTP Object

Slide 17

Continuing the Download HTTP ObjectID Offset 17 = TCP Ack HTTP ObjectISN HTTP ObjectISNTCP Ack TCP ISN HTTP Resp Header HTTP Object

Slide 18

Persistent Store Key-Value Store + Corner Cases Handled + Unlimited Objects Still Efficient (1 save only) TCP Timestamp + Very Efficient (1 machine only) 1 Million Object Limit Corner Cases 18 IPTCPPayload TSTS TSTS KV Exploit TCP Spec for Persistence!

Slide 19

Recover the Connection Initiate New Connection GET ObjectID Range: bytes=Offset- Splice Connections Together Works with Unmodified Servers! 19

Slide 20

TRODS 20 1)Packet Manipulation Server TCP IP TRODS IPTCPIPTCP

Slide 21

Server TCP IP TRODS 21 1)Packet Manipulation 2)Protocol Inspection Request ObjISN ObjID Response 1

Slide 22

Server TCP IP TRODS 22 1)Packet Manipulation 2)Protocol Inspection 3)Blocks Connection Response 1 ObjISNObjID

Slide 23

Server TCP IP TRODS 23 1)Packet Manipulation 2)Protocol Inspection 3)Blocks Connection 4)State Injection IPTCPIPTCP TSTS TSTS

Slide 24

TRODS 24 1)Packet Manipulation 2)Protocol Inspection 3)Blocks Connection 4)State Injection 5)Recovery Initiation Server TCP IP TRODS Ack ?

Slide 25

Server TCP IP TROD S Server TCP IP TROD S Server TCP IP TROD S Failure Walkthrough 25 Client Service Liveness Monitor Load Balancer KV Store IS N ID SYN ACK Request SYN/ACK Response 1

Slide 26

Server TCP IP TROD S Server TCP IP TROD S Failure Walkthrough 26 Client Liveness Monitor Load Balancer KV Store ACK Response 2 ? ! ACK Response 3 ACK Response 4 FIN ACK IS N ID Service

Slide 27

Related Work New Transport Trickles, SCTP, TCP Migrate, TCP FT-TCP, ST-TCP, Backdoors, HTTP CoRAL, 27

Slide 28

Implementation Linux Kernel Module 3,000 lines of C ~CoRAL Optimistic subset of CoRAL 28

Slide 29

Experiments Additional Latency Normal Failure Throughput Lighttpd @ Princeton Apache @ Emulab Hybrid TS & KV Throughput Failure 29

Slide 30

Normal Case Latency TRODS-TimeStamp (TS) Median: + 0.009 ms 99 th : + 0.012 ms TRODS-Key-Value (KV) Median: + 0.137 ms 99 th : + 0.148 ms 30

Slide 31

Recovery Latency 31 Blink of an eye ~35% ~50% ~15%

Slide 32

ThroughPut Per Server 32 120 ops/s Raw 30 ops/s Frontend 30 ops/s/server20 ops/s/serverTPPS

Slide 33

33 KV/Server: 1/8KV/Server: 1/34KV/Server: 1/4KV/Server: 1/2 Lighttpd 9% 38% 7% 66%

Slide 34

34 Apache

Slide 35

Summary Recover Object Delivery Connections Exploit TCP Specification to Coerce Clients To send recovery-starting packets To provide persistent storage Evaluation Low Latency High Throughput Per Server 35 Unmodified ^ ^

Slide 36

Summary Recover Object Delivery Connections Exploit TCP Specification to Coerce Clients To send recovery-starting packets To provide persistent storage Evaluation Low Latency High Throughput Per Server 36 Questions? Unmodified ^