measurement flow architecture in oml
DESCRIPTION
Jolyon White GEC9, 4 th November 2010. Measurement Flow Architecture in OML. OML = Measurement Flows. Rutgers University, New Jersey. Parking Discovery Rutgers Marco Gruteser. Deutsche Telekom Labs @ TU Berlin BOWL Testbed. National Broadband Network 100Mbs FTTH VoD Trial. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/1.jpg)
Jolyon WhiteGEC9, 4th November 2010
Measurement Flow Architecture in OML
![Page 2: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/2.jpg)
OML = Measurement Flows
2
Rutgers University, New Jersey
NICTA, Sydney
Deutsche Telekom Labs@ TU Berlin
BOWL Testbed
National Broadband Network100Mbs FTTH
VoD Trial
IREELNetwork EducationTeaching Platform
Rail Bridge Monitoring Sensors
NSW Road Traffic Authority
Parking DiscoveryRutgers
Marco Gruteser
![Page 3: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/3.jpg)
Current OML data pipeline
Application or
Service
Measurement points Filters Measurement streams
OML Server
Database
(SQL)
Database tables
File
OML client library3
![Page 4: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/4.jpg)
Schemas
• Schemas enable:– Provenance– Processing in the pipeline (data crunching)
• Measurement Stream schema == Combination of schemas of filter outputs
• Each MS stored in its own DB table
4
MP (A, B, C) A
B
C
(S, T)
(U, V, W)
(X, Y)
(S, T, U, V, W, X, Y)
MS Schema
![Page 5: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/5.jpg)
Schemas
• Example: app name is “otr2”
• SQL issued to the database:
• Schema names + metadata define provenance5
avg avg : DOUBLEmax : DOUBLEmin : DOUBLE
ts : DOUBLEflow_id : INT32seq_no : UINT32pkt_length : UINT32src_host : STRINGsrc_port : STRING
MP udp_in:
CREATE TABLE otr2_udp_in ([METADATA COLS], pkt_length_avg REAL, pkt_length_max REAL, pkt_length_min REAL);
![Page 6: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/6.jpg)
Measurement Collection Graph
• Modularize producers + consumers• Measurement Point (MP) – data source• Processing Point (PP) – buffer, select, filter, join,
forward• Termination Point (TP) – persistent storage
6
MP
MP
MP
PP
PP
TP
TP
TP
PP Metadata Store
ServicesAPI
MDA(Measurement Data Archive)
![Page 7: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/7.jpg)
Resource provisioning
• OML has no concept of resource provisioning• Experimenter obtains resources for I&M identically
to experimental resources– i.e. no distinction between I&M and experiment resources
• User has full control over how resources used• Useful defaults, but allow more if experimenter
wants it• Can’t always cleanly separate I&M from
experiment– Mobile wireless testbeds where I&M must share compute
+ network with experiment– E.g. Parknet
• Almost all wireless traffic was measurement flows7
![Page 8: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/8.jpg)
Transports
• OML currently supports two custom procotols– Text version– Binary version
• Standard transports are good!• We like IPFIX, aiming to support it (near future)• Why? Several reasons but:
– Template support self-describing measurement streams
8
Metadata headers(schemas) Measurement flow
Metadata headers(schemas) Measurement flow
![Page 9: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/9.jpg)
Processing Point Applications
9
![Page 10: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/10.jpg)
Proxy Server
• Buffer measurements on command– Don’t transmit to remote server
• Same protocol as server– Transparent to client applications
Proxy server OML ServerApplication
CMD_BUFFERCMD_REPLAY
10
![Page 11: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/11.jpg)
Hierarchical Measurement Collection
• High-resolution measurements lose value over time
• Local storage may be limited• Measuring at different granularities• Inspired by existing research in Streaming
Databases– Numerous VC-backed startups in financial data feed
processing space
11
![Page 12: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/12.jpg)
Context-Driven Experiment Steering
• Dynamic experiments need measured context feedback
• E.g. Geographic trip lines, link state feedback
12
![Page 13: Measurement Flow Architecture in OML](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816525550346895dd7ade4/html5/thumbnails/13.jpg)
Context-Driven Measurement
• Environment feedback can be used to influence the measurement process itself
13