Choosing an Accurate Network Modelusing Domain Analysis

Almudena Konrad, Mills College

Ben Y. Zhao, UC Santa Barbara

Anthony Joseph, UC Berkeley

The First International Workshop on Performance Modelling

in Wired, Wireless, Mobile Networking and Computing

(PMW2MNC, July 2005)

Modeling Network Measurements

• Model-driven traces as experimental tool

– Real measurements costly to perform

– Produce models by extracting statistics from measurements

– Use model to generate traces w/ “similar” characteristics

• New networks (wireless) difficult to model

– Traditional models require stationary statistics

– Gilbert model, Higher order DTMC, Hidden Markov Models (HMM)

– Artifacts disrupt stationarity of wireless traces• Bursty error or delays due to signal interference or loss

– Result: traditional models generate inaccurate results

• Our solution:

– Data preconditioning: preprocess traces before extracting model

– Result: MTA, MMTA accurately model wireless traces

A New Challenge

• Many mathematical models are available– Gilbert, Higher order DTMC, HMM, MTA, MMTA…

– Each optimized for certain characteristics

– Each with different computational and memory costs

• Different networks exhibit different characteristics– IP networks, wireless (802.11), sensor nets

• The challenge– Match up network type with more accurate, lowest cost model

– No tools or techniques to do this

• Our solution:– Domain analysis

Outline

• Motivation

• Data preconditioning and its models

– The Markov-based Trace Analysis model (MTA)

– The Multiple states MTA model (MMTA)

• Quantifying modeling accuracy

• Domain Analysis

• Conclusion

Modeling using Data Preconditioning

• Collect network characteristic trace

– Binary traces show occurrence of an event, e.g. lost or delayed frame

• Identify data patterns or “states” (regions with stationary behavior)

• Precondition the data to fit traditional models

– Calculate probability distribution for each state

– Determine transition probabilities among states

Collected Trace

Subtrace 1

Subtrace 2

Subtrace 3

Preconditioning Model (MTA)

• MTA: Markov-based Trace Analysis Algorithm

– Two states: lossy & error-free states

– Create two subtraces lossy & error-free subtraces

– Algorithm to optimize the change-of-state variable, C • For a given trace, executes stationary test for large range of

values C

• Choose the highest C that yields a stationary lossy states

– Model subtraces as Exponential distributions

C …10001110011100….0 0000…0000 11001100…00 00000..000...

Lossy Lossy Error-free Error-free

C Trace:

Lossy sub-trace:

Error-free sub-trace: ...10001110011100….0 11001100…00 ...

… 0000…0000 00000..000...

States:

Preconditioning Model (MMTA)

• MMTA: Multiple states MTA Algorithm

– Allows for multiple states in the data

– Identifies states in original trace

– Concatenates similar states to create subtraces

– Uses a DTMC to model subtraces & transition between states

Network Path Traces

• A selection of end to end & wireless layer path traces

…0000000 10101111 00000000000…

– End-to-end IP traces (IP-1)

– End-to-end WLAN delay trace (WLAN-D)

– Wireless WLAN trace (WLAN-E)

– Wireless GSM trace (GSM-E)• Trace collected at the Radio Link Protocol (RLP)

• Traces can be decomposed into two stationary states

– Lossy & error-free states

• Characterize traces with three parameters (Lexp, EFexp, Lden)

– Exponential length distributions of lossy and error-free states

– Lossy error density

Quantifying Accuracy of Models

• Previously used metrics

– FER and simulated transmission time – No actual correlation to error distribution

• How accurate is a particular model?

– Synthetic traces comparison (Lexp, EFexp, Lden)

– Compute correlation coefficient between error-burst distributions

Relationship between cc

and accuracy:

CC > 0.96 indicates

an accurate model

Reference trace:

(Lexp, EFexp, Lden)=

(0.006, 0.1, 1)

Example of Model Accuracy:GSM Error Burst Distributions

• GSM, MTA & MMTA experience similar error burst characteristics

• Gilbert, 3rd order and HMM don’t reproduce large error burst

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 5 10 15 20 25 30 35 40 45 50

Error Burst Length (Frames)

Cu

mu

lati

ve D

ensi

ty F

un

ctio

n

GSM GilbertMTA3rd MarkovMMTAHMM

CC values Gilbert: 0.74 HMM : 0.89 MTA : 0.99 MMTA : 0.99

Choosing the Right Network Model:Domain Analysis

• Preconditioning models also work well for traditional networks

– There could be a “bursty” nature to congestion loss in IP networks

– Key is the presence of “bursty” behavior

• Need a way to choose optimal model for given network

– Solution: domain analysis

• Domain analysis

– Create Domain Analysis Plots (DAPs) for wide range of parameters

– Collect empirical packet trace: T= …00110110100…• 1: corrupted/delayed packet, 0: normal packet

– Calculate parameters (Lexp, EFexp, Lden) for T

– Use DAPs to lookup optimal model

Domain Analysis Plots (DAPs)

• Generate reference traces (T1..Tx)

– Range of values (Lexp, EFexp, Lden)

• For each reference trace Ti create

– mathematical models

– artificial traces from models

• Plot error and error-free distribution

• Calculate CC between reference and artificial distributions

• Optimal model for Ti => models that yield highest CC value

networkmodel

artificial network metric trace

trace analysis algorithm

real network metric trace

DAPs for (Lexp,EFexp) 0.001 -> 0.1

0

0.02

0.04

0.06

0.08

0.1

0 0.02 0.04 0.06 0.08 0.1

Lossy State Exp

Err

or-

free

Sta

te E

xp

Gilbert HMM MMTA MTA

0

0.02

0.04

0.06

0.08

0.1

0 0.02 0.04 0.06 0.08 0.1

Lossy State Exp

DAP 1 Lden=0.2

• Gilbert region CC values (Gilbert = 0.99)

• MMTA = 0.98• MMTA region CC values (MMTA =0.97)

• Gilbert = 0.96

DAP 2 Lden = 0.4

• Gilbert region CC values (Gilbert = 0.99)

• MTA = 0.97• MTA region CC values (MTA = 0.98)• MMTA region CC values (M3=0.97)

DAP for (Lexp,EFexp) 0.001 -> 0.1

0

0.02

0.04

0.06

0.08

0.1

0 0.02 0.04 0.06 0.08 0.1

Lossy State Exp

Err

or-

free S

tate

Exp

DAP 3 Lden=0.7

• MMTA region CC values (MMTA=0.98)

• MTA=0.97•MTA region CC values (MTA=0.99)

• MMTA=0.99

Conclusions

• The challenge

– How to choose the most accurate, least cost model for each network type

• The solution

– Key metric for modeling accuracy by error burst distributions

– Domain analysis plots show optimal model for different networks

• Using domain analysis

– Take your own traces for network X

– Calculate the three parameter values

– Use DAP to determine optimal model for your trace

Thank you

Questions?

akonrad@mills.edu