a programming model for application- defined multipath tcp ... · network stack sending queue...

© author(s) of these slides including research results from the KOM research network and TU Darmstadt; otherwise it is specified at the respective slide Template all v.3.4

A Programming Model for Application-

defined Multipath TCP Scheduling

ACM/IFIP/USENIX Middleware 2017

Alexander Frömmgen

Alexander Frömmgen Amr Rizk

Tobias Erbshäußer Max Weller

Boris Koldehofe Alejandro Buchmann

Ralf Steinmetz

Application

Programmable

Scheduler

Userland

Kernel

Network Stack

Sending Queue Subflows

Send Data

Load MPTCP Scheduler

KOM – Multimedia Communications Lab

Multipath TCP in a Nutshell

Server Sending Queue

p5

Scheduler

Subflow 2 LTE

Subflow 1 WiFi

p6

Receive Queue

p1

Application

p2

Client

0x01


Multipath TCP in a Nutshell


p5

Scheduler

Subflow 2 LTE

Subflow 1 WiFi

p6

Receive Queue

p1

Application

p2

Client

MPTCP Packet

Trigger Subflow Establishment

Path Management

Control Information, e.g., Data Sequence Numbers

TCP Options for MPTCP

Fairness on Joint Paths

Congestion Control

Map Packets on Subflows

Scheduler

1 2

3 4

0x01


Multipath TCP Scheduling

Server

Scheduler

Client

Packet 2

Subflow 2 LTE

Subflow 1 WiFi

Packet 1

0x02

RTT = 40ms

BW = …

P_IN_FLY = 0

CWND = 10

RTT = 10ms

BW = …

P_IN_FLY = 0

CWND = 10

Images: https://www.youtube.com/watch?v=U2T1_VrTcoo



Server

Scheduler

Client Subflow 2 LTE

Subflow 1 WiFi

Packet 2

Packet 1

0x02

RTT = 40ms

BW = …

P_IN_FLY = 0

CWND = 10

RTT = 10ms

BW = …

P_IN_FLY = 0

CWND = 10

Schedule on Subflow with minimum round-trip time (RTT). Intuition:




Server

Scheduler


Subflow 1 WiFi

Packet 5 Packet 4

0x02

RTT = 40ms

BW = …

P_IN_FLY = 0

CWND = 10


Schedule on Subflow with minimum round-trip time (RTT).

Packet 1

RTT = 10ms

BW = …

P_IN_FLY = 10

CWND = 10

Packet 11



Server

Scheduler


Subflow 1 WiFi

Packet 5 Packet 4

0x02

RTT = 40ms

BW = …

P_IN_FLY = 0

CWND = 10


Packet 1

RTT = 10ms

BW = …

P_IN_FLY = 10

CWND = 10

Packet 11

Schedule on Subflow with minimum round-trip time (RTT).

which is not saturated (Congestion window larger than packets in flight).



Motivating Example

Home Network WiFi and LTE Germany

Constant Bitrate Stream

0x03


p5

Scheduler

Subflow 2 LTE

Subflow 1 WiFi

p6

Receive Queue

p1

Application

p2

Client ~RTT 40ms

~RTT 10ms


Multipath TCP Scheduler Overview

Name Domain Pref. Available in Linux

Implementation

Min RTT (default) General Purpose Binary

Round-Robin Academic Binary

Redundant Thin Flows Binary

Compensate Loss Short Datacenter Flows ? No

Video and Energy ? No

DASH Video Yes No

...

0x04



Great Concepts without MPTCP Implementation

H. Xu and B. Li. RepFlow: Minimizing flow completion times with replicated flows in data centers.

In IEEE INFOCOM, pages 1581–1589, 2014

X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, and G. Simon. Cross-layer scheduler for video streaming over MPTCP. In ACM MMSys, 2016

N. Kuhn, E. Lochin, A. Mifdaoui, G. Sarwar, O. Mehani, and R. Boreli. DAPS: intelligent delay-aware packet scheduling for

multipath transport. In IEEE ICC, 2014.


Research Questions

How can we systematically specify and execute MPTCP schedulers?

How can we enable application-defined MPTCP scheduling?

0x06


Part I: Towards a Programmable Scheduler in

the Network Stack

Application

Programmable Scheduler

Userland

Kernel

Network Stack

Sending Queue Subflows

Send Data

Load MPTCP Scheduler

0x07


Model of the Scheduler Environment

Sch

ed

ule

r

Sending Queues Subflows

PUSH

op*

*operation, e.g., POP

PUSH

Application

Userland

Kernel

0x08


Model of the Scheduler Environment

Subflow sbf1

Queue Qsbf1

Sending Queue Q

Reinjection Queue RQ Sch

ed

ule

r

Prop.: rtt, cwnd, …

Subflow sbf2

Queue Qsbf2

Prop.: rtt, cwnd, …

p2 p3 p4

p1

Packets in Flight QU

p1

Sending Queues Subflows

PUSH

PUSH

op*

op*

*operation, e.g., POP

PUSH

op*

Application

Userland

Kernel

0x08


Specifying Multipath TCP Schedulers

Expressiveness

Isolation

Timely Execution

1. Modelled Elements as Entities: Set of Subflows with their properties, Queues of Packets with their Properties 2. Declarative Packet and Subflow Selection (Filter, Min, Max) 3. No Recursion, No Functions, Limited Loops 4. Variables with Single Assignment, Implicit and Static Type System 5. No, One, or Multiple Packets per Scheduler Execution

Design Decisions

Requirements

0x09


0000000000001010

Systematically Specify MPTCP Schedulers

Domain Specific Specification Language

1 VAR sbfCandis = SUBFLOWS.FILTER(

2 sbf => sbf.CWND > sbf.SKBS_IN_FLIGHT + sbf.QUEUED

3 AND !sbf.TSQ_THROTTLED AND !sbf.LOSSY);

4

5 VAR backSbf = sbfCandis.FILTER(

6 sbf => sbf.IS_BACKUP).MIN(sbf => sbf.RTT);

7 VAR nonBackSbf = sbfCandis.FILTER(

8 sbf => !sbf.IS_BACKUP).MIN(sbf => sbf.RTT);

9

10 IF (nonBackSbf.RTT_MS > 100 AND backSbf.RTT_MS < 80) {

11 backSbf.PUSH(Q.POP());

12 } ELSE {

13 nonBackSbf.PUSH(Q.POP());

14 }

Example: Preference-aware RTT-sensitive Scheduler

0x0A


0000000000001010

Systematically Specify MPTCP Schedulers

Domain Specific Specification Language

1 VAR sbfCandis = SUBFLOWS.FILTER(

2 sbf => sbf.CWND > sbf.SKBS_IN_FLIGHT + sbf.QUEUED

3 AND !sbf.TSQ_THROTTLED AND !sbf.LOSSY);

4

5 VAR backSbf = sbfCandis.FILTER(

6 sbf => sbf.IS_BACKUP).MIN(sbf => sbf.RTT);

7 VAR nonBackSbf = sbfCandis.FILTER(

8 sbf => !sbf.IS_BACKUP).MIN(sbf => sbf.RTT);

9

10 IF (nonBackSbf.RTT_MS > R1 AND backSbf.RTT_MS < R2) {

11 backSbf.PUSH(Q.POP());

12 } ELSE {

13 nonBackSbf.PUSH(Q.POP());

14 }

Example: Preference-aware RTT-sensitive Scheduler

0x0A


Systematically Specify and

Execute MPTCP Schedulers

Specified schedulers are executable in the Linux Kernel

ProgMP

Lexer

Parser

Kernel

Control Flow Graph Scheduler

Optimizer

Native Code

Userland Write to /proc/…/schedulers

Interpreter

eBPF Compiler

Compiler Frontend

Network Stack

Uses

Ahead of Time Compiler

0x0B


Application-aware Scheduling in

Multi-Tenancy Cloud Environments

Application, e.g., Webserver

Ext. API Socket API

Isolated Scheduler

1

2

3

4

5

6

7

8

9

10

11

12

13

import socket

from progmp import ProgMp

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.connect(("10.0.0.2", 8080))

try:

ProgMp.loadScheduler("python_api_example...")

ProgMp.setScheduler(s, "python_api_example")

except:

print "Scheduler loading error."

ProgMp.setRegister(s, Progmp.R1(), 50)

s.send("Multipath is awesome!")

0x0C


Abstraction vs. Overhead

0x0D


Abstraction vs. Overhead

The runtime environment induces a small overhead, which is acceptable for most application scenarios.

0x0D


Part II:

Design of Novel Multipath TCP Schedulers

0x0E

Round-trip Time-aware

Constant Bitrate Stream Scheduling

Redundant Scheduling

HTTP-aware Scheduling

More in Paper and Under Review*

* Multipath TCP Scheduling for Thin Streams: Active Probing and One-way Delay-awarness by Alexander Frömmgen,

Jens Heuschkel and Boris Koldehofe, IEE ICC 2018 (to appear).


A Close Look at Redundant Schedulers

Seq 2

Seq 3 Seq 4

Seq 5

Q

Subflow 1 (RTT 50ms)

In Flight

Seq 3 Subflow 2 (RTT 150ms)

Receiver Sender

Ack 2

Seq 2

Seq 5

Seq 4

Seq 7

Seq 6

Seq 3

Should we send the fresh packet or the old packet when the acknowledgement arrives at the sender? Ack 2

Seq 6 Seq 5

0x0F



Seq 2

Seq 3 Seq 4

Seq 5

Q

Subflow 1 (RTT 50ms)

In Flight

Seq 3 Subflow 2 (RTT 150ms)

Receiver Sender

Ack 2

Seq 2

Seq 5

Seq 4

Seq 7

Seq 6

Seq 3

Should we send the fresh packet or the old packet when the acknowledgment arrives at the sender? Ack 2

Seq 6 Seq 5

1

2

3

4

5

6

7

8

FOREACH(VAR sbf IN sbfCandidates) {

VAR skb = QU.FILTER(s => !s.SENT_ON(sbf).TOP;

IF(skb != NULL) {

sbf.PUSH(skb);

} ELSE {

sbf.PUSH(Q.POP());

}

}

1

2

3

4

5

6

IF(!sbfCandidates.EMPTY) {

FOREACH(VAR sbf IN sbfCandidates) {

sbf.PUSH(Q.TOP);

}

DROP(Q.POP());

}

Prefer Global Fresh Packets

Prefer Per Subflow Fresh Packets

0x10

ProgMP enables rapid specification and evaluation of schedulers.



ProgMP enables novel redundant schedulers, which outperform established approaches.

Better

0x11


Side Note: How can we systematically compare

and evaluate scheduler design decisions?

0x12

Configurations

Environ.

Single Experiment Result Add Config. and Env. Variations

Fix / Improve Implementation

Add Protocols and Algorithms

Inspect Results: Interactive Data Analysis and Exploration

Web Frontend to Manage Exp.

Exp. Study

Exp.

Config.

Env.

x

Scalable Experiment Execution

Iterative Refinments in the Research Process

Exp.


Side Note: How can we systematically compare

and evaluate scheduler design decisions?

0x12

Configurations

Environ.

Single Experiment Result Add Config. and Env. Variations

Fix / Improve Implementation

Add Protocols and Algorithms

Inspect Results: Interactive Data Analysis and Exploration

Web Frontend to Manage Exp.

Exp. Study

Exp.

Config.

Env.

x

Scalable Experiment Execution

Iterative Refinments in the Research Process

Exp. A Framework for the Management, the Scalable Execution and Interactive Analysis of Extensive Network Experiments

https://maci-research.net


HTTP/2-aware Scheduling

0x13

ProgMP enables HTTP/2-aware Scheduling.


HTTP/2-aware Scheduling

0x14

Note that the large variance is partly caused by a high variance of the underlying real world environment.


Conclusion

• Specification and execution of MPTCP schedulers • Application-defined MPTCP scheduling

The presented programming model enables:

• RTT-aware scheduler • Constant bitrate schedulers • Flavors of redundant schedulers • HTTP-aware scheduler • …

We presented the first programming model for Multipath TCP scheduling.

We proposed and evaluated sophisticated novel schedulers.

0x15


Conclusion

• Specification and execution of MPTCP schedulers • Application-defined MPTCP scheduling

The presented programming model enables:

• RTT-aware scheduler • Constant bitrate schedulers • Flavors of redundant schedulers • HTTP-aware scheduler • …

We presented the first programming model for Multipath TCP scheduling.

We proposed and evaluated sophisticated novel schedulers.

0x15

Try it! https://progmp.net


Questions

KOM – Multimedia Communications Lab http://www.tessares.net/highlights-from-advances-in-networking-part-1/


Enabling per Application-aware Scheduling in

Multi-Tenancy Cloud Environments

Queue Q

Packet Scheduler p2 p3 p4

Extended API

Choose Scheduler

Registers

Set Register Set Packet Property

proc fs

p5

PROP

Parser / Optimizer / Repository

Application, e.g., Webserver

Socket API

R1 R2 …

Python Application

Application B JRE

Ext. API Socket API

Isolated Scheduler Runtime Environment Isolated Scheduler …

Users

pace

K

ern

el

a programming model for application- defined multipath tcp ... · network stack sending queue...

Documents