![Page 1: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/1.jpg)
mOS: A Reusable Networking Stack for
Flow Monitoring Middleboxes
M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim,
Dongsu Han, KyoungSoo Park
![Page 2: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/2.jpg)
47%
63%
67%
0% 25% 50% 75%
Web security gateway
Mail security gateway
Web application firewall
Virtual Appliances Deployed in Service Provider Data Centers
Most Middleboxes Deal with TCP Traffic
• TCP dominates the Internet
• 95+% of traffic is TCP
• Top 3 middleboxes in service providers rely on L4/L7 semantics
2
[1] TCP
UDP
etc
[1] “Comparison of Caching Strategies in Modern Cellular Backhaul Networks”, ACM MobiSys 2013.
95.7%
[2] IHS Infonetics Cloud & Data Center Security Strategies & Vendor Leadership: Global Service Provider Survey, Dec. 2014.
[2]
![Page 3: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/3.jpg)
• Custom middlebox application
• No open source solution
3
Data Accounting System
Cellular Core Network
Internet
Example: Cellular Accounting System
Client
$ $
![Page 4: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/4.jpg)
Challenges in Building Flow-level Middleboxes
4
• The main logic for a cellular accounting system
• No charge for TCP retransmission, only if payloads match.
For every IP packet, p p is retransmitted
no yes
p’s payload == original payload charge for p
yes
skip accounting TCP tunneling attack!
no
Core logic itself is straightforward!
payload A seq# = 10
seq# = 10
payload A seq# = 10
payload B seq# = 10
$
payload B
![Page 5: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/5.jpg)
Challenges in Building Flow-level Middleboxes
5
• Requires handling complex flow-level states and events
• The accounting system requires:
• Reassembly buffer that holds the original payload
• Non-contiguous fragments that holds the original payload
• Event notification on TCP retransmission
• Storage for per-flow accounting metadata and statistics
![Page 6: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/6.jpg)
Challenges in Building Flow-level Middleboxes
• How to implement flow-processing features beneath its core
logic?
6
• 50K~100K code lines tightly coupled with their IDS logic
Borrow code from open-source IDS (e.g., snort, suricata)
• Designed for TCP end host
• Different from middlebox semantics
Borrow code from open-source kernel (e.g., Linux/FreeBSD)
• Complex and error-prone
• Repeat it for every custom middlebox
Implement your own flow management code
![Page 7: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/7.jpg)
Difference from End-host TCP Applications
• Typical end-host TCP applications
• Typical flow-processing middleboxes
7
TCP application
Berkeley Socket API
TCP/IP stack
→ Nice abstraction that separates TCP/IP stack
from application
Middlebox application +
Flow-processing logic
Packet I/O stack
→ Developers build own flow-processing logic
from scratch (e.g., on top of PCAP, DPDK, PF_RING)
Our Goal
Build a reusable flow-processing networking stack
for modular development of middleboxes
![Page 8: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/8.jpg)
mOS Networking Stack
8
• A reusable stack for flow-processing middleboxes
• Abstraction for sub-TCP layer middlebox operations
• Exposes programming abstractions
• Monitoring sockets abstracting TCP flows
• Flexible event system
• Fine-grained resource usage
• Benefits
• Clean, modular development of stateful middleboxes
• Developers focus on core logic rather than flow management
• Highly scalable on multi-10Gbps networks
Middlebox application
mOS programming API
Flow-processing logic
![Page 9: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/9.jpg)
Key Programming Abstractions in mOS
9
• For better reusability, mOS encourages decomposing a
complex application into a set of <event, event handler> pairs
• One can share a well-designed set of event definitions
• mOS provides two key programming abstractions:
• mOS events for expressing custom flow-level conditions
• mOS sockets for retrieving comprehensive flow-level features
mOS event
Event handler
mOS socket
invokes
retrieve flow state
Flow-processing logic
![Page 10: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/10.jpg)
• Notable condition that merits middlebox processing
• Built-in event (BE)
• Events that happen naturally in TCP processing
• e.g., packet arrival, TCP connection start/teardown, retransmission
• User-defined event (UDE)
• User can define their own event (= base event + filter function)
Key Abstraction: mOS Events
10
New data
arrival
Packet
arrival
Filter (HTTP request)
Built-in event
Filter (ACK packet)
HTTP request
arrival
ACK packet
arrival
User-defined event
Filter (counter)
3 duplicate
ACK arrival
User-defined event
![Page 11: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/11.jpg)
• Abstracts a non-terminating midpoint of a ongoing connection
• Simultaneously manages the flow states of both end-hosts
• For every incoming flow, a new mOS monitoring socket is created
• To monitor fine-grained TCP-layer operations and metadata
• e.g., abnormal packet retransmission, out-of-flow packet arrival,
abrupt connection termination, employment of weird TCP/IP options
• Read flow-reassembled data or non-contiguous fragments
• Modify/drop the last packet that raised the event
Key Abstraction: mOS Monitoring Socket
11
peek TCP-layer buffer Application buffer
modification drop
![Page 12: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/12.jpg)
mOS Flow Management
12
• Dual TCP stack management
• Infer the states of both client and server TCP stacks
mOS stack emulation
TCP server
Server side
TCP stack
Server side
TCP stack
TCP
state
TCP
state
SYN
LISTEN
CLOSED SYN_SENT
Client side
TCP stack
Client side
TCP stack
TCP
state
TCP
state
SYN_RCVD
ESTABLISHED
ESTABLISHED
TCP client
![Page 13: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/13.jpg)
mOS Flow Management
13
• Dual TCP stack management
• Infer the states of both client and server TCP stacks
mOS stack emulation
TCP server
Server side
TCP stack
Server side
TCP stack
TCP
state
TCP
state
SYN
SYN/ACK
LISTEN
CLOSED SYN_SENT
Client side
TCP stack
Client side
TCP stack
TCP
state
TCP
state
SYN_RCVD
ESTABLISHED
ESTABLISHED
TCP client
![Page 14: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/14.jpg)
mOS Flow Management
14
• Dual TCP stack management
• Infer the states of both client and server TCP stacks
mOS stack emulation
TCP server
Server side
TCP stack
Server side
TCP stack
TCP
state
TCP
state
SYN
SYN/ACK
LISTEN
CLOSED SYN_SENT
Client side
TCP stack
Client side
TCP stack
TCP
state
TCP
state
SYN_RCVD
ESTABLISHED
DATA/ACK
Receive
buffer
Receive
buffer
ESTABLISHED
TCP client
![Page 15: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/15.jpg)
Scalable mOS Event Management
15
• Each flow can register/change its own set of events dynamically
• Some flows may add or delete events
• Some flows may change event handlers for registered events
• Scalability problem
• How to efficiently manage event sets for 100K+ concurrent flows?
• Naïve approach suffers from expensive copying of event sets
• Observation: the same event sets are shared by multiple flows
• Reduces management overhead
Challenge
How to efficiently find/share the same event set?
![Page 16: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/16.jpg)
Data Structures for Event Management
16
• Each socket points to an event invocation forest that records
a set of flow events to wait on
s1 Socket
e1 ON_CONN_NEW_DATA
YouTube_event
http_event ftp_event
OnYouTubeRequest()
OnFTPEvent()
Event invocation forest
e2
e4 e5
e3
f1
f4 f5
IF1
e6
e7
f7
built-in event UDE event handler socket
![Page 17: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/17.jpg)
Dynamic Event Registration Process
17
Naïve way
1. s1 registers a new event <e3, f3> IF1 is created
2. s2 also registers the same event <e3, f3> IF2 is created
s1
built-in event UDE event handler socket
s2
e1
e2 e3
f2 f3
e4
e6
f6
IF2
e5
e1
e2 e3
f2 f3
e4
e6
f6
IF1
e5
e1
e2
f2
e4
e6
f6
IF0
e5
Problem
IF1 and IF2 are redundant!
Alternative
To reuse IF1 for s2
How does s2 find IF1?
![Page 18: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/18.jpg)
Efficient Search for Dynamic Registration
18
• Each event invocation forest has an ID (searchable via hashtable)
• id (invocation forest) = XOR sum of hash (event + event handler)
• New invocation forest id after adding or deleting <e, f> from t
• id (new forest) = id (old forest) ⊕ hash (e + f)
s1 s2
e1
e2 e3
f2 f3
e4
e6
f6
IF1
e5
e1
e2
f2
e4
e6
f6
IF0
e5
![Page 19: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/19.jpg)
Efficient Search for Dynamic Registration
19
• Each event invocation forest has an ID (searchable via hashtable)
• id (invocation forest) = XOR sum of hash (event + event handler)
• New invocation forest id after adding or deleting <e, f> from t
• id (new forest) = id (old forest) ⊕ hash (e + f)
s1 s2
e1
e2 e3
f2 f3
e4
e6
f6
IF1
e5
e1
e2
f2
e4
e6
f6
IF0
e5
s1 registers a new event <e3, f3>
id(IF0) ⊕ h(e3+f3) = id(IF1) shared
![Page 20: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/20.jpg)
Efficient Search for Dynamic Registration
20
• Each event invocation forest has an ID (searchable via hashtable)
• id (invocation forest) = XOR sum of hash (event + event handler)
• New invocation forest id after adding or deleting <e, f> from t
• id (new forest) = id (old forest) ⊕ hash (e + f)
s1 s2
e1
e2 e3
f2 f3
e4
e6
f6
IF1
e5
e1
e2
f2
e4
e6
f6
IF0
e5
shared
s2 unregisters the event <e3, f3>
id(IF1) ⊕ h(e3+f3) = id(IF0)
![Page 21: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/21.jpg)
Efficient Search for Dynamic Registration
21
• Each event invocation forest has an ID (searchable via hashtable)
• id (invocation forest) = XOR sum of hash (event + event handler)
• New invocation forest id after adding or deleting <e, f> from t
• id (new forest) = id (old forest) ⊕ hash (e + f)
s1 s2
e1
e2 e3
f2 f3
e4
e6
f6
IF1
e5
e1
e2
f2
e4
e6
f6
IF0
e5
s1 unregisters the event <e3, f3>
id(IF1) ⊕ h(e3+f3) = id(IF0)
shared
![Page 22: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/22.jpg)
Fine-grained Resource Management in mOS
• Not all middleboxes require full features
• Some middleboxes do not require flow reassembly
22
TCP
client
TCP
server
Client side
TCP stack
Client side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
P
P
Server side
TCP stack
Server side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
![Page 23: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/23.jpg)
Fine-grained Resource Management in mOS
• Not all middleboxes require full features
• Some middleboxes do not require flow reassembly
• Some middleboxes monitor only client-side data
23
TCP
client
TCP
server
Client side
TCP stack
Client side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
P
P
Server side
TCP stack
Server side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
![Page 24: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/24.jpg)
Fine-grained Resource Management in mOS
• Not all middleboxes require full features
• Some middleboxes do not require flow reassembly
• Some middleboxes monitor only client-side data
• No more monitoring after handling certain events
24
TCP
client
TCP
server
Client side
TCP stack
Client side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
P
P
Server side
TCP stack
Server side
TCP stack
Receive
buffer
Receive
buffer
TCP
state
TCP
state
Global or per-flow
manipulation
![Page 25: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/25.jpg)
• Per-thread library TCP stack
• ~26K lines of C code (mTCP [1] : ~11K lines)
• Shared nothing parallel architecture
mOS Stack Implementation
25
NIC
Packet I/O
Application
mOS core
Receiver
TCP stack
Sender
TCP stack
Rx
Application
mOS core
Tx
....
CPU core N CPU core 1
Symmetric RSS
[1] “mTCP: a highly scalable user-level TCP stack for multicore systems”, NSDI'14
![Page 26: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/26.jpg)
Evaluation
26
1. Does mOS API support diverse middlebox applications?
2. Does mOS promise high performance?
![Page 27: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/27.jpg)
mOS API Evaluation
27
• Does the API support diverse range of middleboxes?
• Snort3 (strip ~10K lines)
• Snort with mOS flow management
• Replaces HTTP/TCP inspection module
• nDPI
• L7 protocol parsing over flow content
• PRADS
• Signature pattern matching on flow content
• Lessons learnt
• mOS simplifies code
• mOS patches vulnerabilities (nDPI/PRADS)
• Detects signature that spans multiple segments
• mOS does not degrade performance
• Perform on par with respective vanilla (DPDK) versions
2104
765
615
0K
20K
40K
60K
80K
100K
Snort3 nDPI PRADS
Lines Modified
Total Lines
![Page 28: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/28.jpg)
mOS API Evaluation (cont.)
28
• Does the API support diverse range of middleboxes?
• Halfback proxy (128 lines)
• Low latency proxy with proactive TCP retransmissions
• Abacus (561 lines vs 4,091 lines)
• Secure cellular data accounting system
• Parallel NAT
• High performance NAT
• Midstat
• netstat for middleboxes
• L4 firewall
• Etc.
• Applications ported to mOS: ~9x code line reduction
![Page 29: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/29.jpg)
Performance Evaluation
• mOS applications in inline mode
• Flow management and forwarding packets by their flows
• 2 x Intel E5-2690 (16 cores, 2.9 GHz), 20 MB L3 cache size,
• 132 GB RAM, 6 x 10 Gbps NICs
• Six pairs of clients and servers: 60 Gbps max
• Intel E3-1220 v3 (4 cores, 3.1 GHz), 8 MB L3 cache size
• 16 GB RAM, 1 x 10 Gbps NIC per machine
29
• Does mOS provide high performance?
mOS
applications
6 x 10Gbps 6x 10Gbps
Clients Servers
![Page 30: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/30.jpg)
Performance Scalability on Multicores
30
1.4 1.2 4.1 3.2 5.0 4.5
16.7 11.6
22.8 21.7
53.0
42.5
0
10
20
30
40
50
60
1 4 16 1 4 16
Th
rou
gh
pu
t (G
bp
s)
(# of CPU cores)
Counting packets Searching for a string
64B file 8KB file
File download traffic with 192K concurrent flows
• Each flow downloads an X-byte content in one TCP connection
• A new flow is spawned when a flow terminates
Two simple applications
• Counting packets per flow (packet arrival event)
• Searching for a string in flow reassembled data (full flow reassembly & DPI)
1 4 16 1 4 16
![Page 31: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/31.jpg)
Performance Scalability on Multicores
31
1.4 1.2 4.1 3.2 5.0 4.5
16.7 11.6
22.8 21.7
53.0
42.5
0
10
20
30
40
50
60
1 4 16 1 4 16
Th
rou
gh
pu
t (G
bp
s)
(# of CPU cores)
Counting packets Searching for a string
64B file 8KB file
File download traffic with 192K concurrent flows
• Each flow downloads an X-byte content in one TCP connection
• A new flow is spawned when a flow terminates
Two simple applications
• Counting packets per flow (packet arrival event)
• Searching for a string in flow reassembled data (full flow reassembly & DPI)
1 4 16 1 4 16
Performance linearly scales as
# of cores are increased.
![Page 32: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/32.jpg)
Latency overhead by mOS applications
32
58.4
93.8 93.5
117.4
191.9 193.2
0
50
100
150
200
250
Direct connection Counting packets Searching for a string
Flo
w c
om
ple
tio
n t
ime (
us) 64B file 8KB file
76us
35us
![Page 33: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/33.jpg)
Dynamic Event Registration Evaluation
33
• Monitor 192K concurrent flows
• Flow size: 4KB
• Searching for a string in flow reassembled data
• Dynamically register a new event when target string found
• 50% client flows have target strings
18.1
12.5 8.9
5.9 3.2
0.6
34.1 33.7 33.7 32.8 31.5
27.4
0
10
20
30
40
32 64 128 256 512 1024
Naïve mOS
# of event nodes in the tree
Thro
ughput
(Gbps)
![Page 34: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/34.jpg)
• Software-based middleboxes have:
• Modularity issues
• Readability issues
• Maintainability issues
mOS stack: reusable networking stack for middleboxes
• Programming abstraction with socket-based API
• Event-driven middlebox processing
• Efficient resource usage with dynamic resource composition
• mOS stack/API available @:
https://github.com/ndsl-kaist/mOS-networking-stack
Conclusion
34
![Page 35: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/35.jpg)
Thank You
35
http://mos.kaist.edu/
Questions?
https://github.com/ndsl-kaist/mOS-networking-stack
![Page 36: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/36.jpg)
Appendix
![Page 37: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/37.jpg)
Extra Slides
![Page 38: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/38.jpg)
Performance under Selective Resource Consumption
38
19.67
23.22
35.47
39.22
56.68
29.6 34.18
46.43
51.9
0
10
20
30
40
50
60
64 256 1K 4K 16K
Th
rou
gh
pu
t (G
bp
s)
File size (B)
full flow management
w/o client buf management
w/o buf management
w/o client side
w/o client side, w/o server buf mgmt.
59.97
![Page 39: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/39.jpg)
Real applications performance
39
Application original + pcap original + DPDK mOS port
Snort-AC 0.57 Gbps 8.18 Gbps 9.17 Gbps
Snort-DFC 0.82 Gbps 14.42 Gbps 15.21 Gbps
nDPIReader 0.66 Gbps 28.92 Gbps 28.87 Gbps
PRADS 0.42 Gbps 2.03 Gbps 1.90 Gbps
• Workload: real LTE packet trace (~67 GB) • 4.5x ~ 28.9x performance improvement • mOS brings code modularity & correct flow management
![Page 40: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/40.jpg)
Events & Available Hooks
40
• Stream monitoring socket
• Raw monitoring socket
Built-in event MOS_HK_SND MOS_HK_RCV
MOS_ON_PKT_IN O O
MOS_ON_CONN_START O O
MOS_ON_CONN_END O O
MOS_ON_TCP_STATE_CHANGE O O
MOS_ON_REXMIT O O
MOS_ON_CONN_NEW_DATA X X
MOS_ON_ORPHAN X X
Built-in event MOS_HK_SND MOS_HK_RCV
MOS_ON_PKT_IN X X
![Page 41: mOS: A Reusable Networking Stack for Flow ... - KAIST · mOS: A Reusable Networking Stack for Flow Monitoring Middleboxes M. Asim Jamshed, YoungGyoun Moon, Donghwi Kim, Dongsu Han,](https://reader033.vdocuments.us/reader033/viewer/2022060918/60aae6a4c460aa10364f5ba9/html5/thumbnails/41.jpg)
Cellular Accounting with mOS Networking Stack
41
Core Logic + Flow Mgmt
For every IP packet, p p is retransmitted
no yes
p’s payload == original payload account for p
yes
skip accounting report abuse
no
Event-action
eREX MOS_ON_REXMIT
eNEW MOS_ON_CONN_NEW_DATA
eREX eNEW
FFAKE
eFAKE
freport faccnt
Filter
Built-in events
User-defined event
Event handler (action)
FFAKE IsFakeRexmit()
eFAKE UDE_FAKE_REXMIT
freport ReportAbuse()
faccnt AccountDataUsage()
4,639 LoC
561 LoC
built-in event UDE event handler filter function FFAKE