2004-04-28

Decomposing Overlay Applications

Yitzchak Gottlieb

Princeton University

Achieving Extensibility with High Performance

2

Constant Innovation

• 1725 RFCs since 1995– 50 Obsolete old RFCs

– 307 Update

• ~1400 New ideas– 897 Standards track

– 142 Experimental

3

Types of Innovations

• Similar to existing– More recipients

• Broadcast

• Multicast

– Any one recipient• Anycast

• Truly new– Scalable routing

– Scalable object location

– Content-based addressing

– Fault tolerance

– Automatic management

Hard to change current networks to enable these services

4

Overlay Networks

• On end hosts– Application-level networks

• Advantage– Ease deployment

• Problems – Limited performance

• User space

• Proxy path

5

Performance Limits: Proxy Path

100 Mbit/s

1500 B packets

10 Gbit/s800 000 pkt/s

8 000 pkt/s

6

PlanetLab

“Planetlab is an open platform for developing, deploying, and accessing planetary-scale services”

7

PlanetLab

• Nodes are PCs– Centrally managed– Distributed world-wide

• 366 Nodes

• 152 Institutions

• ~500 Projects

• Overlays are user-level programs– Executing in vservers on Linux

8

PlanetLab

• PlanetLab is for overlays

• Very successful overlays– e.g. Routing overlays– Traffic increases– Inefficiencies interfere

How to achieve high performance with overlays?

9

Inside Overlay Nodes

OS

Maintain network

Forward packets

Process queries

…

10

Computation Domains

• Where computation is• Separated by barriers

– Address space

– Protection level

– Processor

• Differences– Performance

– Trust Computation Barrier

11

Hypothesis

• We can decompose overlay applications

• We can map components onto different

computation domains

• This will improve performance and

efficiency

12

Outline

• Introduction

• Decomposition

• Software Architecture

• Examples

13

• Five forms

• Forms are:– Semantically equivalent

• All equally viable

– Refinement steps– Different

• Performance

• Trust

Decomposing Overlay Networks

14

Lessons of Internet

• High speed IP forwarding

• Separate concerns– Routing protocol

• Program forwarding table

– Packet forwarding• Decrement TTL

• Replace layer-2 address

15

Control and Data

• Control– Complex– Processed locally

• Data– Simple– Processed remotely

16

Decomposition—Form I

• State of the art

• Monolithic application

• Few network connections– All communication to application– Application sees all traffic

Application

17

Application

Decomposition—Form II

• Two code modules– Control

• Maintain network

• Offer service

– Overlay Forwarder• Forward to other hosts

• Two modes in overlay– Local delivery

– Forwarding

C

Forwarder

Control

18

• In separate domains– User-space control

– Overlay Forwarder• In kernel

• On line card

Decomposition—Form III

• Why trust forwarder?– SimpleVerifiably safe

• 100–200 instructions

• No backward branches

– Flow isolation

C

Forwarder

Control

19

Decomposition—Form IV

• Two modules– Overlay Daemon

• Connectivity• Routing

– Application• File serving• Query Processing

• Two control modes– Maintain network

– Offer service

Control

C

Forwarder

Application

Daemon

20

ForwarderFF

Decomposition—Form V

• Application-specific forwarding extension– Recording– Redirecting– Modifying

C Daemon

Application

• Applications need data stream

21

Decomposing Overlay Applications

Control

Data

General

Application

Application

Daemon

Overlay Daemon

F

Overlay Forwarder Application Forwarder

F

Application

22

High Performance Overlays

• Decompose overlays

• Place overlay forwarder in

• Leave rest in user space

Daemon Application

FF

FFKernel

User

FF

kernelrouterline card

Router

FFNIC

user space

23

Forwarders and ExtensionsSRAM

Read/WriteRegister

OperationsRegistersNeeded

IP-- 24 35 2

Pastry 20 <50 3

Chord 24 <200 8

TCP Splicer 24 45 7

Wavelet Dropper 8 28 4

ACK Monitor 12 15 4

SYN Monitor 4 5 0

Port Filter 20 44 4

24

Outline

• Introduction

• Decomposition

• Software Architecture

• Examples

25

Plug Board

C

26

Plug Board

• Software architecture for decomposition

• Features– Single powerful classifier– Multiple computation domains– Multiple forwarders

• Fixed forwarders

• Extensible forwarders

27

Plug Board

• Miscellaneous features– Modularity

• Functions grouped by protocol

– Encapsulation• Input functions

• Output functions

• Wrap forwarders

28

Implementation

• Point implementations

• Three domains– User space Linux– Kernel space Linux

• Leverage SILK

– Intel IXP1200 EEB• Leverage Vera architecture

29

Outline

• Introduction

• Decomposition

• Application

• Examples

30

Decomposition Examples

• Applications– Peer-to-peer networks– Other network

• Each is decomposable– Four components

31

Peer-to-Peer Networks

• Peers are clients, servers, and routers

• Examples– Pastry

• P2P application toolkit

– Chord• Scalable lookup service

– Gnutella• Unstructured, file-sharing network

32

Pastry

• Peer-to-peer application toolkit

• Nodes have 128 bit ID– Treated as number in base 2b

• Forward to node with closer ID– Digit by digit– Logarithmic forwarding

33

• Overlay Daemon– Connectivity

• Overlay Forwarder– Find next longest prefix match

• Application– PAST, Scribe, etc.

• Application Forwarder– Application dependent

Pastry

Application

Daemon

F

F

34

Pastry Implementation

• User-level library

• Kernel-level forwarding

• IXP Forwarding– ~50 instructions

Implementation Performance (pkt/s)Java Pastry 3000User Pastry 18000Kernel Pastry 25000IXP Pastry 90000

35

Chord

• Scalable object location

• 128 bit node addresses– Circular address space

• Forward to farthest 2-ith of circle– Logarithmic forwarding

• Iterative

• Recursive

35

17

26

4

0

36

Chord

• Overlay Daemon– Chord invariants

• Overlay Forwarder– Recursive lookup

• Application– Node location

• Application Forwarder

35

17

26

4

0

Application

Daemon

F

37

Gnutella

• Overlay Daemon– Connectivity

• Overlay Forwarder– Flood– Record route

• Application

• Application Forwarder– Redirects successful queries

38

Other Network Services

• IP Forwarding– Filters, Firewalls

• Proxies– Splicing TCP connections

• Ethernet bridging

39

Summary

• Overlays limit performance

• Improve by applying classic techniques– Decompose overlay

• Control/Data

• General/Application-specific

– Map onto computational domains

40

Conclusions

• Overlay applications can perform well

• Separate applications and networks– Optimize networks– Offload applications

• Some overlays are applications– Too complex for network– Limited performance

41

Credits

• SILK/Scout– Larry Peterson, Andy Bavier, Oliver Spatschek,

Ian Murdoch, Xiahu Qie, Aki Nakao

• Vera– Scott Karlin

• IXP1200– Scott Karlin, Tammo Spalink

And many more…

42

Thank you

43

URLs

• PlanetLab– http://www.planet-lab.org/

• Princeton CS Network Systems Group– http://www.cs.princeton.edu/nsg

44

Interfaces

• Overlay Daemon and Overlay Forwarder– Routing table interface (Add, Delete route)

• Application and Application Forwarder– Application specific– Use bit pipe or shared memory

• Overlay Daemon Application– Overlay specific

45

Function Types

• Predicates (P), Forwarders (F)

• Extensions (E)

• Extensible Forwarders (EF)

• Extensible Extensions (EE)

• Input (I) and Output (O)

• Encapsulation (F, I, OF)

• Installation (P,F())

46

A B

M

TCP Connection Splicing

• Overlay Daemon– Installs splicer

• Overlay Forwarder– Splicer

• Application

• Application Forwarder– URL rewriter

47

IP Forwarding

• Overlay Daemon– Routing protocols (e.g. BGP)

• Overlay Forwarder– IP forwarding

• Application– Many examples

• Application Forwarder

48

B3

Ethernet Bridging

• Overlay Daemon– Computes Minimum Spanning Tree

• Overlay Forwarder– Multicast to open ports

• Application– Control learning algorithm

• Application Forwarder– Record source and destination

B1

B5 B6

B2

B4