Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap Algorithms for Bitmap Algorithms for Counting Active Flows on Counting Active Flows on

HighHighSpeed LinksSpeed Links

Cristian Estan, George Varghese, Mike Fisk

Computer Science and Engineering Department,University of California, San Diego

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Why count flows?Why count flows?

• Detect port/IP scans• Identify DoS attacks• Estimate spreading rate

of a worm• Packet scheduling

Dave Plonka’s FlowScan

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Existing flow counting solutionsExisting flow counting solutions

ServerNetFlow data

Analysis

Traffic

reports

Network Operations Center

Router

Fast link

Memory

Network

Memory sizeMemory size

& bandwidth& bandwidth

NetworkNetwork

bandwidthbandwidth

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Motivating questionMotivating question

• Can we count flows at line speeds at the router?

– Wrong solution – counters

– Naïve solution – use hash tables (like NetFlow)

– Our approach – use bitmaps

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting algorithmsBitmap counting algorithms

• A family of algorithms that can be used as building blocks in various systems

• Algorithms can be adapted to application

• Low memory and per packet processing

• Generalize flows to distinct header patterns– Count flows or source addresses to detect attack– Count destination address+port pairs to detect scan

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(green)=10001001

Set bits in the bitmap using hash of the flow ID of incoming packets

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(blue)=00100100

Different flows have different hash values

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(green)=10001001

Packets from the same flow always hash to the same bit

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(violet)=10010101

Collisions OK, estimates compensate for them

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(orange)=11110011

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(pink)=11100000

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

HASH(yellow)=01100011

As the bitmap fills up, estimates get inaccurate

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

Solution: use more bits

HASH(green)=10001001

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – direct bitmapBitmap counting – direct bitmap

Solution: use more bits

Problem: memory scales with the number of flows

HASH(blue)=00100100

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – virtual bitmapBitmap counting – virtual bitmap

Solution: a) store only a portion of the bitmap

b) multiply estimate by scaling factor

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – virtual bitmapBitmap counting – virtual bitmap

HASH(pink)=11100000

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – virtual bitmapBitmap counting – virtual bitmap

HASH(yellow)=01100011

Problem: estimate inaccurate when few flows active

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multiple bmpsBitmap counting – multiple bmps

Solution: use many bitmaps, each accurate for a different range

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multiple bmpsBitmap counting – multiple bmps

HASH(pink)=11100000

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multiple bmpsBitmap counting – multiple bmps

HASH(yellow)=01100011

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multiple bmpsBitmap counting – multiple bmps

Use this bitmap to estimate number of flows

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multiple bmpsBitmap counting – multiple bmps

Use this bitmap to estimate number of flows

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multires. bmpBitmap counting – multires. bmp

Problem: must update up to three bitmaps per packet

Solution: combine bitmaps into one

OR

OR

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

HASH(pink)=11100000

Bitmap counting – multires. bmpBitmap counting – multires. bmp

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting – multires. bmpBitmap counting – multires. bmp

HASH(yellow)=01100011

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Basic estimatesBasic estimates

Direct bitmap

Virtual bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Multiresolution bitmap estimateMultiresolution bitmap estimate

Find most accurate Find most accurate componentcomponent

Estimate number of Estimate number of flows hashing to itflows hashing to it

Apply scaling factorApply scaling factor

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Relative error in estimatesRelative error in estimates

Direct bitmap

Virtual bitmap

Multiresolution

bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Error of virtual bitmapError of virtual bitmap

Flow density (flows/bit)

Ave

rage

(re

lati

ve)

erro

r

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Memory requirementsMemory requirements

Direct bitmap < N / ln (Nε2+1)

Virtual bitmap 1.5441/ ε2

Multiresolution bitmap 0.9186 ln (Nε2) / ε2+ct.

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

100 million flows, error 1%100 million flows, error 1%

Hash table* 1.21 Gbytes

Direct bitmap 1.29 Mbytes

Virtual bitmap* 1.88 Kbytes

Multiresolution bitmap 10.33 Kbytes

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Triggered bitmapTriggered bitmap

• Need multiple instances of counting algorithm (e.g. port scan detection)

• Many instances count few flows

• Triggered bitmap– Allocate small direct bitmap to new sources– If number of bits set exceeds trigger value, allocate

large multiresolution bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Related workRelated work

• Flajolet, Martin (1985) probabilistic counting – Memory use similar to multiresolution bitmap

• Whang et al (1990) introduce direct bitmap• You, Chang (1996) use virtual bitmap• Chauduri, Motwani, Narasayya (1998)

– Counting flows without bias impossible from sampled data

• Duffield, Lund, Thorup (2002) – Accurate solutions based on counting TCP SYN flags

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Multires. bmp. vs. prob. countingMultires. bmp. vs. prob. counting

Number of flows (log scale)

Ave

rage

(re

lati

ve)

erro

r

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Scan detection memory usageScan detection memory usage

Interval

length

Snort

(naïve)

Probabilistic

counting

Triggered

bitmap

12 seconds 1.94 M 2.42 M 0.37 M

600 seconds 49.60 M 22,34 M 5.59 M

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Talk structureTalk structure

• Per packet processing for bitmap algorithms

• Computing flow count estimates from bitmaps

• Variance analysis of estimates

• Derived algorithms

• Related work

• Measurements

• Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

A family of counting algorithmsA family of counting algorithms

Setting Algorithm Applications

General counting Multiresolution bmp. Track infections

Narrow range Virtual bitmap Triggers (e.g. DoS)

Small counts common Triggered bitmap Port scans

Stationarity Adaptive bitmap Measurement

Add and delete Increment-decrement Scheduling

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Bitmap counting algorithmsBitmap counting algorithms

• A family of algorithms that can be used as building blocks in various systems

• Algorithms can be adapted to application

• Low memory and per packet processing

– With 2Kbytes error around 1%

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

The endThe end

Bitmap algorithms will be available at:

http://ial.ucsd.edu/bitmaps/

Any questions?

Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Adaptive bitmapAdaptive bitmap

• Virtual bitmap measures accurately number of flows if range known in advance

• Often number of flows does not change rapidly• Measurement repeated• Can use previous measurement to tune virtual

bitmap• Combine a large virtual bitmap with a small

multiresolution bitmap used for tuning

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Adaptive bitmap accuracyAdaptive bitmap accuracy

Number of flows (log scale)

Ave

rage

(re

lati

ve)

erro

r

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

With 2 kilobytes of memoryWith 2 kilobytes of memory

Adaptive bitmap

(min avg max)

Probabilistic counting

(min avg max)

Trace1 -4.4% 1.1% 4.7% -9.5% 2.8% 13.3%

Trace2 -1.9% 0.7% 2.0% -6.9% 2.8% 7.6%

Trace3 -1.8% 0.6% 1.8% 2.4% 10.2% 17.7%

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

Increment-decrement algorithmsIncrement-decrement algorithms

• Active flow defined as flow with packets in queue

• Must support additions and deletions

• Replace bits of bitmap with counters– Increment when packet arrives– Decrement when packet leaves– Estimate number of flows based on zero counters