the measurement manager - cs.umd.edupavlos/papers/pavlos-thesis-slides.pdf · november 17, 2008...

November 17, 2008 Ph.D. Defense 1

Ph.D. DefenseDepartment of Electrical and Computer Engineering

University of Maryland, College Park, MD

The Measurement Manager:Modular and Efficient

End-to-End Measurement Services

Pavlos [email protected]

Network Measurement Is Needed


• Applications need to monitor end-to-end path– Due to the design of the Internet network routers do not

provide any feedback about network conditions

– Applications need to adapt based on path conditions

• Application Layer Overlays need to select paths– Based on capacity, available bandwidth, delay, loss– Needed for construction and maintenance of overlay

• Network services need to select servers– Download managers need to select the best server– BitTorrent needs to select peers to download/upload

Network Measurement


So many applications need to measure the paths on which they already send data For example: applications need to measure the residual

available bandwidth on the paths they use to select the best nodes to useServer Selection Problem Overlays: MediaNetBitTorrent: Peer-to-Peer

Problem DescriptionCurrent measurement techniques are inefficient,

inflexible, not modular and do not scale

• Active Measurement– Injects probes to measure the network– Examples: available bandwidth, capacity, loss– Inefficient and Disruptive

• Passive Measurement– Uses existing traffic to estimate network properties– Can estimate limited network properties

• Custom Measurement– Uses application’s own traffic to measure the network (Skype)– Works only for that application, not reusable


Example: Available Bandwidth

• Pathload: An active measurement tool• Measures end-to-end available bandwidth• By Jain & Dovrolis at Georgia Tech

– Self Loading Periodic Streams methodology (SLoPS)• One way delays of a periodic stream show increasing trend when

the stream rate is higher than the available bandwidth

• Operates in rounds: sends packet trains of 100 UDP probes

• Each packet train corresponds to a probing rate

• Uses equally sized packets and uniform packet gaps

– Good candidate for our evaluation• Available bandwidth is a very useful network property• Quite accurate (even for GigE speeds, PAM05)

• Non-trivial overhead (we can test probe reuse)


Pathload: Sample Run



Our ThesisReusing empty padding inside probes to piggyback

application payload is feasible and enables us to build a new Measurement Architecture that is

flexible, efficient and modular.

Our ApproachA new Measurement Manager Architecture

Combine best properties of Active, Passive and Custom approaches and develop a new transport protocol,

MGRP, that combines applications packets and probes.

The Measurement Manager


Motivation

Why do we need theMeasurement Manager?

MGRP

ArchitectureImplementation

Step-by-Step Examples

Experimental Results

Piggybacking is feasibleand improves

network performance

The Estimation Overlay

Case Study: MediaNet

MGRP in the Network Stack


MGRP: Step by Step Example




Prober sends MGRP probesApplication sends TCP over MGRP

MGRP Sender piggybacks payload on probes





MGRP packets traverse the network



MGRP Receiver reconstitutes probes and payload

Pseudocode: Sending Probes


Pseudocode: Receiving Probes


Piggybacking Effects on Applications

• MGRP may affect application data– Payload buffer adds constant delay

• May affect the response of TCP since it increases RTT

– May increase the chance of loss• Piggybacked portions go out at a higher burst rate• Some probes are prone to loss (tail of long train)

• MGRP addresses these issues– Payload delay set to 10 millisec, much lower than typical RTTs

(40-70millisec) and enough for full piggybacking– Turn off piggybacking for high risk probes– Allow the application and measurement tool to turn off

piggybacking selectively


Piggybacking Effects on Tools

• MGRP reduces the packets on the path– Tools reach an estimate faster; more room to probe– But tools must now account for piggybacking– With MGRP, packets from local traffic have much less influence– Tools measure the path as if the application was sending less

• How to compensate for piggybacking– MGRP computes which portion of the piggybacked packets

would have competed with the probes just sent out– Since we use a payload buffer to increase piggybacking, not all

piggybacked packets need to be counted– Some of the piggybacked packets would have been sent after

the probes, so they do not need to be counted– Tools like pathload, use this information to adjust their estimates


MGRP Implementation

• Implementation– MGRP is implemented as a Linux kernel module (2.6.25)– MGRP registers as a transport protocol– Has interfaces to socket and IP layer– Applications access MGRP through the BSD socket API

• MGRP Protocol– Encapsulates two packets into one– Piggybacks transparently application data into probe padding– Performs fragmentation and reassembly– Generates precise timing for probes inside the kernel– Provides instrumentation to assess its precision– Fully functional protocol for the Linux kernel


MGRP Packet Format


MGRP Header

MGRP Packet Format


MGRP Header MGRP Packet



Motivation


MGRP





network performance



Experiments

• Objective– Show that MGRP piggybacking is feasible and

improves network performance

• Metrics– Throughput

• Source Traffic: MGRP increases throughput• Cross Traffic: With MGRP, probing is more fair

– Completion Times• Measurement Tool (Pathload): Completes Faster


Experiments: Topology


• UDP or TCP cross traffic between c1 and c2• TCP traffic from m3 to m5, rate limited at 4mbps• Running pathload continuously between m3,m5

• Two versions of pathload: pSLOW and pFAST• Two modes: With and Without MGRP

Experiments: Cross Traffic


Experiments: Probe Traffic


• Pathload– Running continuously

between m3,m5– Two versions:

• pSLOW (original)• pFAST

– Two modes:• Over UDP (original)• Using MGRP and Probe

Transactions

• Fixed Packet Trains– Emulates pathload but without

probe rate adaptation

– Makes it easier to compare since it injects always the same number of probes

pk1 pk2 pk3

Packet length 300 500 300

Packets per train 10 20 10

Packet gap (usec) 200 1000 0

Train gap (msec) 20 20 10

Inst. Throughput (mbps) 12.0 4.0 Line rate

Avg. Througput (mbps) 1.1 2.0 2.3

STEP Experiment:pk1, pk2, pk3, pSLOW, pFAST


STEP Experiment:Details for pk2, pFAST and pSLOW


STEP: Bandwidth Timeseries with pk2

November 17, 2008 Ph.D. Defense 39STEP: probe train pk2

Piggybacking reduces the probing overhead

November 17, 2008 Ph.D. Defense 40STEP: probe train pk2

STEP: Bandwidth Timeseries with Pathload

November 17, 2008 Ph.D. Defense 41STEP: pathload pFAST

Effective Probe Overhead is Minimal


Pathload Completes Faster

November 17, 2008 Ph.D. Defense 43STEP: pathload



72%

STEP: pathload



72%

45%

STEP: pathload



72%

45%

66%

23%

STEP: pathload

Pathload Completes More Often


95%

77%

STEP: pathload

Pathload Completes More Often


95%

77%

66%

52%

STEP: pathload

Adjustment for Pathload is Accurate(pFAST)


Adjustment for Pathload is Accurate(pSLOW)

November 17, 2008 Ph.D. Defense 50STEP: pathload pSLOW

WEB Experiment with Pathload(pSLOW and pFAST)


Pathload Completes Faster (especially for pSLOW)

November 17, 2008 Ph.D. Defense 53WEB: pathload

Pathload Completes Faster (especially for pSLOW)


81%

65%

71%

27%

WEB: pathload

Pathload Completes More Often (especially for pSLOW)


96%

88%

91%

45%

WEB: pathload

But the Source Traffic is Affected


First it Falls Behind


For this portion, the source cannot sustain the fixed 4mbps rate and falls behind.

Then it Catches Up


For this portion, the source cannot sustain the fixed 4mbps rate and falls behind.

For this portion, the source tries to catch up since there is available bandwidth.

But we care how long we can sustain 4Mbps


60%

MGRP sustains 4Mbps for shorter time


30% 60%

Similar Results for pSLOW


30% 60% 65%50%

Piggybacking may be too Aggressive

November 17, 2008 Ph.D. Defense 62WEB: pathload pFAST

Piggybacking may be too Aggressive


Too many piggybacked packets get lost


Solution: Reduce the Piggybacking Ratio


So that High Risk Probes are Avoided


Investigate Different Piggybacking Ratios


A Piggybacking Ratio of 50% is sufficient


Pathload Completion Times Still Better(50% piggybacking)


Lessons Learned

• Blind piggybacking is not always good– Application packets and probes share the same (risky) fate– May lead to unexpected losses that hurt our source

• We tried different policies to address this issue– Piggyback only the initial portion of a probe transaction (50%)– Duplicate all piggybacked packets to mask losses

• To try: Hybrid piggybacking and duplication– Piggyback portion of probe train and duplicate– Automatically determine the threshold for partial piggybacking


Experiment Summary

• Our Micro-Benchmarks showed:– MGRP piggybacking is feasible and improves

network performance– Throughput

• Source Traffic: MGRP increases throughput• Cross Traffic: With MGRP, probing is more fair

– Completion Times• Measurement Tool (Pathload): Completes Faster

• We proceed to show how MGRP works with an existing Application Layer Overlay




Motivation


MGRP





network performance



Case Study: MediaNet Overlay


Overlaynode

Sender Receiver

(a) MediaNet can modify the streaming rate

(b) MediaNet can reroute the streams

• Forwards media streams based on path conditions

– Controlled by a Global Scheduler– Multiple Local Schedulers– Small number of long-term TCP

connections between peers– Only passive measurements

used in original operation– Needs to measure all paths that

it sends data on

• MGRP Modifications– Added active measurements – Pathload estimates the available

bandwidth on active paths– MGRP not used directly but

through the Estimation Overlay

Estimation Overlay


• Separate Measurement Overlay Service– Provides measurement as a service– Performs active measurements between pairs of

nodes using MGRP so that overhead is minimized– Currently provides the Pathload algorithm

• MediaNet and the Estimation Overlay– The Estimation Overlay mirrors the MediaNet overlay

– The MediaNet Global Scheduler queries the Estimation Overlay to acquire per-link statistics

– MediaNet Local Schedulers need no modifications

MediaNet Experiment


MediaNet and Cross Traffic Starts

MediaNet Experiment


MediaNet Local Schedulers start sending MPEG streams

MediaNet Experiment


The Estimator Overlay mirrors the MediaNet Overlay

MediaNet Experiment


MediaNet queries the Estimator for Available Bandwidth

MediaNet adjusts the ratesof the MPEG Streams


• MediaNet loops three MPEG video streams– MediaNet monitors conditions in two ways– The Local Schedulers report if they can send at desired rate

• If the streaming rate cannot be sustained, MediaNet reconfigures and

lowers the rate by filtering lower priority frames (first B, then P frames)

– The Estimation Overlay continuously monitors the paths• If available bandwidth becomes available again, MediaNet reconfigures

and sets the highest possible rate for each stream

Frame Type

Frame Size

(bytes)

Frequency (frames/sec)

High Rate

(Kbps)

MediumRate

(Kbps)

LowRate

(Kbps)

I 13500 2

1200700

200

P 7625 8 Dropped

B 2850 20 Dropped Dropped

MPEG Streaming Rates




Highest Stream QualityAll three types of frames are being forwarded (I, P, B)(effective rate is 1.2 Mbps)



Local Scheduler reports not enough bandwidth. MediaNet drops to lowest rate by droppingP and B frames (effective rate 200Kbits/sec)



Pathload reports that bandwidth is again available. MediaNet restores highest rate using all frames.



After a brief drop to the lowest rate, MediaNet reverts to the medium rate by dropping only B frames (effective rate 700Kbits/sec)

MediaNet Original OperationCannot reclaim freed bandwidth

November 17, 2008 Ph.D. Defense 87MediaNet:

mgrpOFF/pOFF

MediaNet with Active MeasurementBetter use of available bandwidth


mgrpOFF/pFAST

MediaNet with MGRPBest use of available bandwidth


mgrp10/pFAST

MediaNet Original OperationMPEG Streams Remain at Lowest Rate


mgrpOFF/pOFF

MediaNet with Active MeasurementMPEG Streams Upgrade to Higher Rates


mgrpOFF/pFAST

MediaNet Original OperationMPEG Streams Upgrade to Highest Rates


mgrp10/pFAST

MGRP Improves MediaNetThe aggregate MPEG streaming rate is higher


Experiment Runs AverageRun Duration (sec)

Aggregate Streaming

Rate (Mbps)

Improvement over

non-MGRP

Original MediaNet(mgrpOFF/pOFF) 14 337 1.84

PathloadpSLOW

mgrpOFF 22 336 1.96

mgrp10 32 336 2.05 4.40%

PathloadpFAST

mgrpOFF 10 335 1.86

mgrp10 22 336 2.28 22.52%

Related to the quality of the playback

MGRP Improves MediaNetThe number of decoded MPEG frames increases



Aggregate Frames per

Second

Improvement over

non-MGRP


PathloadpSLOW

mgrpOFF 22 336 39.58

mgrp10 32 336 43.42 9.69%

PathloadpFAST

mgrpOFF 10 335 39.10

mgrp10 22 336 52.08 33.19%

Directly affects the quality of the playback

MGRP Improves MediaNetThe number of bad MPEG frames decreases



Aggregate Ratio of Bad

Frames

Decrease over

non-MGRP


PathloadpSLOW

mgrpOFF 22 336 0.0080

mgrp10 32 336 0.0079 -0.76%

PathloadpFAST

mgrpOFF 10 335 0.0114

mgrp10 22 336 0.0100 -12.07%

Bad frames result in playback glitches. MGRP reduces them.

Related Work:Measurement Services

• Perform Measurements– MAD: kernel level generation of probes based on discrete clock– Periscope: programmable probing framework– Scriptroute: conduct measurement from multiple vantage points– Pktd: packet capture and injection agent

– NIMI: large-scale measurement infrastructure

• Query for Measurements– iPlane: overlay service that predicts path properties– Sophia: information plane for networked systems– Routing Underlay: shared measurement infrastructure

• Congestion Manager– Inspiration for Measurement Manager


Related Work: Inline Measurements

• Reuse TCP packets as probes– TCP Probe: uses back-to-back TCP packets as probes

(integrates TCP with CapProbe algorithm)– ImTCP: spaces TCP packets and uses them as packet

trains (to estimate bandwidth algorithm)

• Inject probes in a TCP connection– TCP Sidecar: masks probes as TCP retransmissions

within a TCP stream– Sting: manipulates a TCP connections to measure

packet loss rates– Sprobe: manipulates a TCP connection to send packet

pairs to estimate bottleneck bandwidth


Future Work:An Estimation Service

• Current State– Estimation algorithms

reside outside theMeasurement Manager

• New Estimation Service– Natural addition to our architecture– Based on the Estimation Overlay


– Measurement Manager is extended with a new Estimation Service– Applications query the Estimator for measurements– Estimator uses MGRP to send out probes– Estimator uses best suited algorithm under current conditions

– High overhead when it has data to use as piggybacking– Low overhead when no payload available

– Applications reuse their own traffic transparently and unknowingly – Estimation and Probing continue to be separate

Future Work:New Probing Paradigms

• Full power of MGRP not leveraged– So far we have retrofitted existing tools to use MGRP

• But current tools try to minimize average overhead• By sending specially crafted probe sequences with precise timing• Probe sequences can be quite long and very bursty• Which do not match well with application traffic and piggybacking• We consider this Inelastic Probing

– What if we change the way we do probing?– MGRP gives us access to all application packets

• MGRP enables new probing techniques– Opportunistic Probing Turn every packet into a probe– Elastic Probing Minimally shape application bursts– Semi-Elastic Probing Combine application packets with probes


Thanks!


How we stand out• Efficiency and Accuracy

– MGRP balances the accuracy of active measurement and the efficiency of passive measurement

• Flexibility– The Probe API is sufficient to implement a range of active measurement

algorithms accurately.

• Modularity– Estimation algorithms can be designed as if active, but transparently

implemented as if passive, independent of applications


The ability of MGRP to piggyback any data packet on any probe turns out to be pivotal in making our measurement

system unique, in the sense that any measurementalgorithm can now be written as if active,

but implemented as passive

ContributionsNew Architecture: Designed the Measurement Manager to provide a systematic and transparent way for applications to reuse their own traffic for network measurement.

New Protocol: Designed MGRP, a new transport protocol, that implements piggybacking in the Linux network stack.

New Probe API: Analyzed a range of active measurement tools and derived a generic API for sending probes. Tools use this API to send probes using MGRP.

Experiments: Provided experimental evidence that piggybacking data packets on measurement probes is feasible and improves the source traffic, the cross traffic and the performance of measurement algorithms, without affecting their accuracy.

Case Study and Estimation Overlay: Demonstrated how to build a generic Estimation Service using the Measurement Manager, and how to use this service to improve the performance of MediaNet, an existing Application Layer Overlay.



Additional Slides

Pathload: Burst Profile


Pathload

• Good candidate for our evaluation– Available bandwidth is a very useful network property– Quite accurate (even for GigE speeds, PAM05)– Non-trivial overhead (we can test probe reuse)

• 100 probes per train contain a lot of wasted padding

• Modified Operation– Uses the Probe API to schedule probes with MGRP– Probes are generated inside the kernel

• Timestamps are fixed appropriately

– MGRP may piggyback TCP payload on the probes• Estimates need to be adjusted because of piggybacking


Active Measurement• What is it?

– Injects special packets (probes) into the network

– Sends packets in groups with accurate inter-packet gaps

– Uses estimation algorithms to deduce network properties

– Can be fast, accurate– Many standalone active

measurement tools

• The Problem– Not efficient: probes carry

empty padding– Intrusive: probes interfere

with packets on the path we are measuring

– Usability: not clear how standalone tools can be integrated

– Does not scale

ProbesProbes

Source Destination


Passive Measurement

• What is it?– Observes existing

packets and applies estimation algorithm

– Typically tightly coupled with each application

• TCP RTT estimation• RTP delay monitoring

– Efficient, no overhead

• The Problem– Unpredictable:

cannot generate packets when no packets available

– Inflexible: no control over the packet sizes and packet gaps

– Inconsistent: accuracy cannot be controlled


The Life of a Probe Transaction


May 21, 2007 Ph.D. Research Proposal 115

Piggybacking: Delay the payload


– TCP packets are buffered in a FIFO buffer– Increases chances that probes will find payload for piggybacking– Impact on TCP: appears as increased RTT

Piggybacking: Delay the probes


– Probes are delayed waiting for TCP payload– No impact on TCP– Algorithms can compensate for delay

MGRP Piggybacking


MGRP Fragmentation Header


MGRP Fragmentation


MGRP Reassembly


the measurement manager - cs.umd.edupavlos/papers/pavlos-thesis-slides.pdf · november 17, 2008...

Documents