Multi-Scale Analysis for Network Traffic

Prediction and Anomaly DetectionLing Huang

Joint work with Anthony Joseph and Nina Taft

January, 2005

January 2005

Problem Statement Accurate network traffic modeling and prediction are

important for: Network provisioning Problem diagnosis

But, network traffic is highly dynamic Exhibits multi-timescale properties (temporal domain) Anomalies (failures, attacks) interfere with analysis

Need to isolate anomalies from normal traffic variation to achieve better modeling and prediction

January 2005

Our Work

We decompose traffic signals into two parts Normal variations: follow certain law, can be modeled and

are predictable Anomalies: consist of sudden changes and are not

predictable Method: Multi-scale signal analysis and modeling for

Network traffic prediction [almost done] Volume anomaly detection and identification [in progress]

January 2005

Properties of Real Life Data Streams

Three real-world data traces and a random trace

January 2005

Observation: Multi-Scale Property

Data source: bandwidth measurements for the CUDI network interface on an Abilene router with 5-minute average.

Multi-scale property of traffic data in a weekly measurement

Long-term trend

Transient Variation

Sudden Changes (anomalies)

January 2005

Multi-Scale Prediction of Future Data Traditional Approaches

Last seen data as approximation for current estimation Linear Prediction

Exploit and leverage statistical properties of data stream on temporal domain in a window of size T

Exploit temporal correlation in different time scale Long term trend: B-spline estimation High frequency residual: ARMA modeling ARMA stands for AutoRegressive and Moving Average

model, which is a standard time series technique to model chaotic data stream

January 2005

Two-Level Modeling and Prediction B-spline modeling for long term trend

Piecewise continuous, low-degree B-spline can represent complex shapes

Least-square B-spline regression for two-level decomposition

B-Spline extension for future forecasting ARMA forecasting for transient oscillation

System Identification to determine the order of the model Parameter estimation by optimization algorithm Low complexity recursive equation for future forecasting

Statistical properties for the calibration of prediction results

January 2005

Complexity of Prediction Algorithms

Legend T: the window size of history data m: the order of the linear predictor K: the order of the ARMA model d: the degree of B-spline curve c: the increase in storage due to multi-level data

representation

January 2005

Performance of Prediction Algorithms

Performance of Prediction Algorithms On Network Traffic

Mean Relative Increment

January 2005

Unpredictability: Anomalies in Data Signal

The data stream can be decomposed into two layers: the long-term trend, which is the modeled pattern; the residual, high frequency with anomalies

Monday Data

Long-term trend (modeled)

Residual with anomalies

January 2005

Anomalies Detection and Identification Volume anomaly: Sudden change in link/flow’s

traffic count Network failures, attacks, flash crowds Measurement anomalies

Anomalies are not normal variations of network traffic and are not predictable Worse yet, anomalies skew the prediction models! For better modeling and prediction, need to detect and

isolate anomalies from data The rest of the talk focuses on anomaly detection

algorithms Existing algorithms: single-link vs. network-wide

analysis New directions

January 2005

Single-link Anomalies Detection Multi-scale analysis to capture temporal correlation

Use wavelets for multi-scale data decomposition Isolate characteristics of traffic signal on different timescales Expose the details of both ambient (modeled) and

anomalous traffic Detection of sharp increase in the local variance in a moving

time window on different time-scale Disadvantages

Diagnose on single link or at single router, and is impractical to do analysis for large network

Many anomalies are across multiple links, and is not obvious on single link

January 2005

Network-Wide Anomalies Detection Diagnose traffic anomalies spanning multiple links

Capture the spatial correlation cross links Analyze origin-destination flows with known traffic matrix Principle Component Analysis (PCA) for dimension

reduction and signal decomposition Separate traffic signal into modeled and residual space Scoring/prediction method to detect abnormal changes in

residual space Disadvantages

Need ISP support Single time scale analysis Centralized algorithm

January 2005

New Direction (1): Multi-Scale PCA PCA analysis on wavelets representation of traffic

data Wavelets capture correlation within a single link PCA captures the correlation across links and transforms

the multivariate space into a subspace which preserves maximum variance of the original space

PCA analyzes data on single time-scale and can not utilize the information pertaining to the frequency

Multi-scale PCA combines two extremes of wavelets and PCA based analysis of multivariate data

Benefits: detection of multi-timescale anomalies

January 2005

New Direction (2): Distributed Algorithm Distributed anomalies detection based on partial

flow information No ISP support, no information about OD flows and traffic

matrix Diagnose volume anomalies based on network monitoring

data – flow and link information from a subset of places in the network Network-wide traffic modeling, inference and prediction

improve measured data Distributed algorithms for network-wide data reduction,

decomposition and anomaly detection Collective PCA

Benefit: local analysis and low cost

January 2005

Conclusion and Future Work Summary

Apply statistical algorithms to network traffic analysis Multi-scale analysis is effective in traffic modeling and

prediction Contribution: using multi-scale network-wide analysis to

capture both temporal correlation within single link and spatial correlation cross links

Future Work Develop distributed algorithms based on multi-scale PCA Exploit tradeoff between detection accuracy, false positive

and computation cost Build real system for applications in traffic analysis and

network health monitoring