towards a meaningful mra for traffic matrices

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices1/36

Towards a Meaningful MRA for Traffic Matrices

D. Rincón, M. Roughan, W. Willinger

IMC 2008


Outline

Seeking a sparse model for TMs

Multi-Resolution Analysis on graphs with Diffusion

Wavelets

MRA of TMs: preliminary results

Open issues


Context: Abilene 12 nodes (2004)

Abilene topology (2004)

STTL

SNVA

DNVR

LOSA

KSCY

HSTN

IPLS

ATLA

CHINNYCM

WASH

ATLA-M5


Example: Abilene traffic matrix

STTL

SNVA

DNVR

LOSA

KSCY

HSTN

IPLS

ATLA

CHIN

NYCM

WASH

ATLA-M5

ST

TL

SN

VA

DN

VR

LOS

A

KS

CY

HS

TN

IPLS

AT

LA

CH

IN

NY

CM

WA

SH

AT

LA-M

5

80 Mbps

0 Mbps

20 Mbps

40 Mbps

60 Mbps

Traffic matrix from Abilene (March 2nd 2004, 12:00-12:05)


Traffic matrices

Open problems Good TM models

• Synthesis of TMs for planning / design of networks• Traffic prediction – anomaly detection • Traffic engineering algorithms

Traffic and topology are intertwined • Hierarchical scales in the global Internet apply also to traffic

Time evolution of TMs How to reduce the dimensionality catch of the inference

problem?

Our goals Can we find a general model for TMs? Can we develop Multi-Resolution machinery for jointly

analyzing topology and traffic, in spatial and time scales?


Can we find a general model for TMs?

Our criterion: the TM model should be sparse Sparsity: energy concentrates in few coefficients (M << N2) Tradeoff between predictive power and model fidelity Easier to attach physical meaning Could help with the underconstrained inference problem

Multiresolution analysis (MRA) “Classical MRA”: wavelet transforms observe the data at

different time / space resolutions Wavelets (approximately) decorrelate input signals

• Energy concentrates in few coefficients• Threshold the transform coefficients sparse representation

(denoising, compression) Successfully applied in time series (1D) and images (2D)


How to perform MRA on TMs?

Traffic matrices are 2D functions defined on a graph

2D Discrete Wavelet Transform of TM as images• Uniform sampling in R2

• TMs are NOT images! – the intrinsic geometry is lost

Graph wavelets (Crovella & Kolaczyk, 2003)• Spatial analysis of differences between link loads -anomaly

detection

• Drawbacks of the graph wavelets approach

– Non-orthogonal transform - overcomplete representation

– Lack of fast computation algorithm



Diffusion Wavelets (Coifman & Maggioni. 2004)

MRA on manifolds and graphs Diffusion operator “learns” the

underlying geometry as powers increase – random walk steps

Amount of “important” eigenvalues -vectors decreases with powers of T

Those under certain precision are related to high-frequency details, while those over are related to low-frequency approximations

Operator T

W1 V1

W2 V2

W3 V3



Diffusion Wavelets (Coifman & Maggioni. 2004) Operator T

W1 V1

W2 V2

Eigenvalues (low to high frequency)

Cv2 5 CW2 3 CW1 2


Diffusion Wavelets and our goals

Unidimensional functions of the vertices F(v1) can be projected onto the multi-resolution spaces defined by the DW.

Network topology can be studied by defining the right operator and representing the coarsened versions of the graph.

But Traffic Matrices are 2D functions of the origin and destination vertices, and can also be functions of time: TM(V1,V2,t)


2D Diffusion wavelets

Extension of DW to 2D functions defined on a graph

F(v1,v2) Construction of separable

2D bases by “projecting twice” into both “directions”

• Tensor product

• Similar to 2D DWT Orthonormal, invertible,

energy conserving transform

Operator T

VW1WW1 WV1 VV1

VW2WW2 WV2 VV2

VW3WW3 WV3 VV3


2D Diffusion wavelets

Extension of DW to 2D functions defined on a graph Operator T

VW1WW1 WV1 VV1

VW2WW2 WV2 VV2


MRA of Traffic Matrices

More than 20000 TMs from operational networks Abilene (2004), granularity 5 mins GÉANT (2005), granularity 15 mins Acknowledgments: Yin Zhang (UTexas), S. Uhlig (Delft),

Diffusion operator: A: unweighted adjacency matrix “Symmetrised” version of the random walk – same

eigenvalues Double stochastic (!) Precision ε = 10-7

2

1

2

1

ADD


2D Diffusion wavelets – Abilene example

W1 V1

W2 V2

W3 V3

W4 V4

V0

W5 V5

W6 V6

W7 V7

W8 V8

V4

# eigenvalues at each subspace Wj = WVj + VWj + WWj

12

120

120

102

64

6

5 1

32

21

11



Original TM - 12 eigenvaluesApprox3 - 10 eigenvaluesApprox4 - 6 eigenvaluesApprox

5 - 5 eigenvaluesApprox

6 - 3 eigenvaluesApprox

7 - 2 eigenvaluesApprox8 - 1 eigenvalue

STTL

SNVA

DNVR

LOSAKSCYHSTN

IPLS

ATLA

CHIN

NYCM

WASH

ATLA-M5



DW coefficients Abilene 14th July 2004 (24 hours)

Coefficient index (high to low freq)Time (5 min intervals)



How concentrated is the energy of the TM? Wavelet coefficients for the Abilene TM

12 x 12 = 144 coefficients, low- to high-frequency

0 20 40 60 80 100 120 1400

2

4

6

8x 10

10

Coefficients - low to high frequency

Sq

ua

red

co

effi

cie

nts

(e

ne

rgy)

Coefficients – high to low frequency


Compressibility of TMs


Stability of DW coefficients


Rank signature

Coefficient index

Tim

e (5

min

inte

rval

s)

Coefficient rank – Abilene March 2004


Rank signature – anomaly detection?


Conclusions and open issues

Representation of TMs in the DW domain TMs seem to be sparse in the DW domain Consistency across time and different networks

Ongoing work Develop a sparse model for TMs How the sparse representation relates to previous models (e.g.

Gravity) ? Exploit DW’s dimensionality reduction in the inference problem Exploring weighted / routing-related diffusion operators Exploring bandwidth-related diffusion operators Introducing time correlations in the diffusion operator Diffusion wavelet packets – best basis algorithms for compression DW analysis of network topologies


Thank you !

Questions?


Extra slides


Géant

23 nodes (2005)


Context: topology

Spatial hierarchy

AS1

AS2

AS3

PoPs

Access Networks

Network/AS


Multi-Resolution Analysis

Intuition: “to observe at different scales”



Approximations: coarse representations of the original data



Mathematical formalism

Set of nested scaling subspaces (low-frequency approximations)

generated by the scaling functions

The orthogonal complement of Vi inside Vi+1 are called detail (high-frequency) or wavelet subspaces Wi,

generated by wavelet functions

0121 ...... VVVVV nn

)(xi

)(xiW1 V1

W2V2

W3 V3

V0


-0.5 0 0.5 1 1.5-1.5

-1

-0.5

0

0.5

1

1.5

-0.5 0 0.5 1 1.5-1.5

-1

-0.5

0

0.5

1

1.5


Scaling functions: averaging, low-frequency functions

Wavelet functions: differencing, high-frequency functions

0 5 10 15-1.5

-1

-0.5

0

0.5

1

1.5

0 5 10 15-1.5

-1

-0.5

0

0.5

1

1.5


Multi-Resolution Analysis (2D)

Separable bases: horizontal x vertical Example: 2D scaling function


Wavelet transform example

2D wavelet decomposition of the image for j=2 levels Vertical/horizontal high/low frequency subbands


Our approach

Can we develop Multi-Resolution machinery for analyzing topology and traffic, in spatial and time scales?

Classical 1D or 2D wavelet transforms are not an option

We need a new graph-based wavelet transform! Graph wavelets for spatial traffic analysis (Crovella &

Kolaczyk 03) Diffusion wavelets (M. Maggioni et al, 06)


The tools: Graph wavelets

Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Exploit spatial correlation of traffic data Sampled 2D wavelets



Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Link analysis Definition of scale j: j-hop neighbours



j=1

j=3

j=5

traffic

Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Anomaly detection in Abilene

towards a meaningful mra for traffic matrices

Documents