Enabling a “RISC” Approach for Software-Defined Monitoring

using Universal Streaming

Vyas Sekar

Zaoxing Liu, Greg Vorsanger,

Vladimir Braverman

Network Management:Many Monitoring Requirements

SDN Controller (OpenDayLight etc.)

Traffic Engineering

Analyze new user apps

Anomaly DetectionNetwork Forensics

Worm Detection

Accounting

Botnet analysis …….

“Heavy-hitters”“Flow size distribution”

“SuperSpreaders”

“Entropy”, “Traffic Changes”

1

Traditional: Packet Sampling

3

1613111

Flow reports1

Not good for fine-grained analysisExtensive literature on limitations for many tasks!

11316111131611

12

Sample packets at random, aggregate into flows

FlowId CounterFlow = Packets with same patternSource and Destination Address and Ports

Estimate: FSD, Entropy, Heavyhitters, Changes, SuperSpreaders ….

4

Application-Specific Sketches

Packet Processing

Counter Data

Structures

Application-LevelMetric

Heavy Hitter Entropy Superspreader

Complexity: Need per-metric implementationRecent Example: OpenSketch [NSDI’13]Trend: Many more applications appear!

….Monitoring(on router)

Bloom-filter,Count-min Sketch,reversible sketch, etc.

Packet Processing

Counter Data

Structures

Application-LevelMetric

Packet Processing

Counter Data

Structures

Application-LevelMetric

….

Traffic

Computation(off router)

5

Packet Processing

Counter Data

Structures

Application-LevelMetric

Support many applications

Holy Grail of Flow Monitoring?

Results with high accuracy

Traffic

6

Our Solution: Universal Monitoring

Recent theory advances: Universal Streaming

Packet Processing

UniversalSketch

Traffic

App 1

Application-specific Computation

App n…...

UnivMon Control Plane

UnivMon Data Plane

One sketch does it ALL

Theory of Universal Streaming

1. Vladimir Braverman, Rafail Ostrovsky: Zero-one frequency laws. STOC 20102. Generalizing the Layering Method of Indyk and Woodruff: Recursive Sketches for Frequency-Based Vectors on Streams. APPROX-RANDOM 2013

1331511 2 4 6 5 …... (A stream of length m with n unique items)

‘Universal’ Sketch

Estimated G-sum

frequency vector is <f1,f2 … fn>

G-sum =

As long as does not grow asymptotically faster than2,Universal Sketch can do it!

6

8

Universal Sketch Data Structure

1331511 2 4 6 5

11511

25

2

L2 Heavy Hitter Algorithms

(1,4), (3,2),(5,2)

Heavy Hitters

(1,4), (5,2),(2,1)

…...

(2,1)

(5,2), (2,1)

0

1

log(n)

…...

Generate k=log(n) pairwise ind. zero-one hash functions:

H1 …. Hk

2 5

5

Similar to counting bloom filter

H1(1)=1, H1(5)=1, H1(2)=1

H2(5)=1, H2(2)=1

H3(2)=1

LevelsHeavy Hitter Alg

Heavy Hitter Alg

Heavy Hitter Alg

Heavy Hitter Alg

Count Sketch Alg+4 +2 -2

-2 -4 +2

+2 +4 -2

+4 -2 -1

+4 -2 +1

+1 -2 -4

+1

-1

+1

…...

Count-Sketch, Pick-and-drop etc.

In Parallel

9

Estimating G-sum

(1,4), (3,2),(5,2)

Counters from Universal Sketch

(1,4), (5,2), (2,1)

…...

(2,1)

(5,2),(2,1)

Levels0

1

log(n)

…...

Apply arbitrary g()

(1,g(4)), (3,g(2)),(5,g(2))

(1,g(4)), (5,g(2)), (2,g(1))

(5,g(2)),(2,g(1))

(2,g(1)) Y3=g(1)Sum of the g()s

Y2=g(1)+g(2)

Y1=g(1)+g(2)+g(4)

Y0=2g(1)+2g(2)+g(4)

Estimated G-sum

Recursive Steps:Yi-1 = 2Yi + new counters – repeated counters

10

Putting it together: UnivMon

Universal Sketch Offline Recursive Computation

11

Comparison with custom sketches via OpenSketch

Preliminary Evaluation

N/A

12

• Distributed universal streaming

• Multidimensional data

• Dynamically change monitoring scope

• Feasibility of hardware implementations?

Future Directions

13

Conclusions• Network management needs many traffic metrics• Today’s solutions offer undesirable extremes• Generic but low fidelity (e.g., sampling)• High fidelity but high complexity (e.g., specific-sketches)

• Holy grail: Universal Monitoring• Decouple monitoring control and data plane like SDN!

• This work: Can be viable via Universal Sketches• Several open questions• e.g. dynamic, multidimensional, distributed, hardware viability