parapet research group, princeton university ee ieee international symposium on workload...

Parapet Research Group, Princeton University EE

IEEE International Symposium on Workload Characterization

IISWC ’05, Austin, TX

Oct 06, 2005

Detecting Recurrent Phase Behavior under Real-System Variability

Canturk ISCI

Margaret MARTONOSI

Canturk Isci - Margaret Martonosi2

Detecting Recurrent Phase Behavior under Real-System Variability [IISWC ’05]

Phase Analysis & Real Systems Phases:

Self-similar, mostly recurrent, execution regions

How to identify phase recurrences when real-system effects make them inexact replicas?

Useful for characterization, dynamic-adaptive management

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40 45Time [s]

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40 45Time [s]

E1 E2 E3 E4 E5



Underlying Research Questions What are the types and extent of system-induced variations?

How do phases manifest themselves with real-system effects?

Can we extract recurrent behavior in spite of these variations?If so, how?



Background: Power and Phases

Runtime processor power monitoring and estimation [Micro’03] Sample PMCs to estimate powers for 22 chip components

Real measurement feedback for tuning and verification

Workload power phase behavior with power vectors [WWC’03] Consider power estimations as power vectors

Characterize “power phases” based on vector similarity



Variability in Real-System Runs Initial idea was to look at phase distributions of apps and use

some signature analysis to detect/predict phases

HOWEVER: Multiple runs inevitably exhibit different behavior

Quantities & durations vary Phase distributions vary

Metric Variability Time Variability

0

10

20

30

40

50

60

0 5 10 15 20 25 30Time [s]

Po

wer

[W

]

Gcc Run1Gcc Run2Gcc Run3

Time VariabilityMetric Variability




Metric variability

Time variability





Real-System Variability Effects on Phases

t

Metric

A B C BIdeal

A B C B D BGlitch

A B C B D E BGradient

A B C B D E BShift

A B C B D E FMutation

A B C B D E FTime Dilation



Real-System Variability Effects on Phases

A direct apples to apples comparison of phase signatures is not very relevant in real world!

A B C BIdeal

A B C B D BGlitch

A B C B D E BGradient

A B C B D E BShift

A B C B D E FMutation

A B C B D E FTime Dilation

FINAL



Improving Phase Analysis Using Transitions

t

Metric

A B C B

t

Metric

A B C B D E F

Ideal

Final



Value based phase representations do not show good correlation

t

Value Based Phases (VBP)

A B C B

t

A B C B D E F

12

32

12

4

6

23

5

Improving Phase Analysis Using Transitions



Our Proposed Solution with Transitions

Tracking phase transitions rather than phase sequences is more useful in detecting recurrent behavior

t

Transition Based Phases (TBP)

A B C B

t

A B C B D E F

1 1 100…0 00…0 00…0 00…0

1 1 100…0 00…0 00…0 00…000…0

1 1 1



Our Transition-Guided Detection FrameworkBenchmark run #1

Sample PMCs to form 12D vectors

Benchmark run #2

Vector stream #1

Identify Transitions

Vector stream #2

TBPinit #1

Apply glitch/gradient filtering

TBPinit #2

TBPgg #1 TBPgg #2

Apply near-neighbor blurring

TBPggN #1

Match ⇒ Peak at best alignmentMismatch ⇒ No observable peak

Apply cross correlation



Sampling Effects: Glitches & Gradients Nothing happens without disturbances Glitches

Glitch: Instability where before & after are same Spurious transitions

Nothing happens instantaneously Gradients Gradient: Instability where before & after are different A single true trans-n

Glitch/Gradient Filtering: Very simple: no consecutive transitions

Initial Transitions:

GLITCHES:

0 00 0 00 110 000 0 0 0 00 0 11 1 10 000 00 00 0 11 11 1 111 100 00

Refined Trans-ns: 0 00 0 00000 000 0 0 0 00 000000 000 00 00 000000000000 00

Initial Transitions:

GRADIENTS:

0 00 0 11 110 000 0 0 0 00 0 11 1 10 000 00 00 0 11 11 1 111 100 00

Refined Trans-ns: 0 00 010000 000 0 0 0 00 010000 000 00 00 010000000000 00



Time Shifts Cross-correlation of binary sequences shows the highest

matching of signatures at the best alignment

Ex: Matching and Mismatch cases, and “The Peak”

Matching case: Gcc1-Gcc2 Mismatch case: Gcc-Equake

Strong peak indicates good match! Low peak

signifies mismatch!



Observation: Dilations exist as small jitters (few samples)

Proposed Solution: “Near-Neighbor Blurring” Blur edges slightly Consider transitions as distributions around

their actual locations

Tolerance: Spread of this distribution, [t-x, t+x] samples

Ex: Matching improvement with tolerance=2:

Time Dilations

run1

Mism

atch!

t

1 1 1

00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

run21 1 1

00 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t

1



.7.3

.7.3

.7.7.3

.7.3

.7

Observation: Dilations exist as small jitters (few samples)

Proposed Solution: “Near-Neighbor Blurring” Blur edges slightly Consider transitions as distributions around

their actual locations

Tolerance: Spread of this distribution, [t-x, t+x] samples

Ex: Matching improvement with tolerance=2:

Time Dilations

run1

Match!t

1 1 1

0 0 0 0 0 0 0 0

run21 1 1

00 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t

1

0 0 0 000000000



Our Transition-Guided Detection FrameworkBenchmark run #1

Sample PMCs to form 12D vectors

Benchmark run #2

Vector stream #1

Identify Transitions

Vector stream #2

TBPinit #1

Apply glitch/gradient filtering

TBPinit #2

TBPgg #1 TBPgg #2

Apply near-neighbor blurring

TBPggN #1

Match ⇒ Peak at best alignmentMismatch ⇒ No observable peak

Apply cross correlation



Results How do we quantify phase recognition quality?

Matching Score:

Range of values ≥ 0

Higher is better



Results Detection Results:

(green: highest match; red: highest mismatch)

bzip2 equake gap gcc gzip mcf vortex convert lamebzip2 0.44 0.05 0.07 0.05 0.15 0.18 0.08 0.09 0.15equake 0.15 0.39 0.28 0.06 0.26 0.25 0.09 0.04 0.08gap 0.20 0.22 0.79 0.07 0.10 0.33 0.04 0.05 0.12gcc 0.05 0.04 0.05 0.19 0.03 0.05 0.16 0.04 0.12gzip 0.10 0.10 0.19 0.05 1.08 0.16 0.10 0.03 0.07mcf 0.18 0.18 0.23 0.04 0.16 6.14 0.17 0.08 0.08vortex 0.23 0.10 0.12 0.01 0.11 0.08 1.93 0.03 0.05convert 0.21 0.17 0.26 0.06 0.14 0.25 0.09 0.22 0.13lame 0.12 0.11 0.12 0.04 0.13 0.20 0.06 0.02 0.21

BESTTOLERANCE

1 2 1 3 1 0 1 7 2



Receiver Operating Characteristics

Best detection scheme (tolerance=1) achieves 100% hit detection with <5% false alarms.

(Using the same threshold for all apps!)

Very high detect threshold

P{hit} = 0P{false alarm} = 0

0 detect threshold

P{hit} = 1P{false alarm} = 1

Desired operating point

P{hit} ~ 1P{false alarm} ~ 0



Comparison: TBP Outperform VBP10.818.4 18.4 11.3

0

1

2

3

4

5

bzip2 equake gap gcc gzip mcf vortex convert lame AVE

Mat

chin

g S

core

/ H

igh

est

Mis

mat

ch

Value-Based Phases Refined Trans-nsNear-Nbr Blurred Trans-ns

Break-evenvalue

(VBPs)(Tgg)

(TggN)

In all cases transitions perform better In almost all cases near-neighbor blurring improves detection



Conclusions Detecting phase behavior on real systems has interesting

challenges resulting from system induced variability

Phase transition information improves detection capabilities TBP show 6X better detection capabilities than VBP

Supporting methods, such as Glitch/Gradient Filtering and Near-Neighbor Blurring improve detectability of transition signatures Near-neighbor blurring with tolerance=1 achieve 100% recurrence

detection with <5% false alarms

Resulting infrastructure can enable a range of phase-oriented system adaptations!



Thanks!



BACKUPS 0.5) How much noise, how much variation? 1) Variation in time sequences of phase distributions for two

gcc runs; recurrent phases with ammp 2) Refined transition counts for different thresholds 3) Advantages with Power/PMC Vectors 4) Threshold vs. Hits & Misses with Tolerance=1 5) How about instr-n based sampling/control flow-based

approach? 6) What’s the source of variability? 7) Glitches/Gradients vs. sampling frequency? 8) Use of this framework? 9) Multithreaded / OLTP like benchmarks? 10) SMT/CMP/multiprogramming environment?



0.5) Noise vs. Variations

Gcc Gzip Vpr Vortex Gap

Crafty

Measured Modeled

Stable Apps Vpr/Crafty change very little, Variable ones change much more



1)Phase Distributions Along Execution Timeline for 2 Runs of Gcc



0

10

20

30

40

50

60

70

80

90

100

1330 1530 1730 1930 2130 2330 2530

1) Recurrence Example with Ammp

Although obvious to the eye, comparing phase sequences directly does not reveal the recurrence clearly!

0

1

2

3

4

5

6

PM_FPU_FMOV_FEST

PM_INST_CMPL

PM_FPU_FSQRT

PM_FPU_FIN

PM_FPU_FEST

PM_FPU_FMA

PM_FPU_FDIV0

1

2

3

4

5

6

0.01 4729.66 9471.93 14213.98 18972.01 23736.07 28478.26 33221.45 37963.79 42706.74

Ammp (POWER4) Performance Vectors (As Individually Normalized Rates)

0

1

2

3

4

5

6



2) Refined Transitions for Different Thresholds

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50Threshold

Fin

al T

ran

sit

ion

Co

un

t

0

100

200

300

400

500

Ori

gin

al T

ran

sit

ion

Co

un

t

Final transitions Original transitions

0

10

20

30

40

50

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7Threshold

Fin

al T

ran

sit

ion

Co

un

t

0

120

240

360

480

600

Ori

gin

al T

ran

sit

ion

Co

un

t

Final transitions Original transitions

Gcc

Equake



3) Advantages with Power/PMC Vectors Direct relation to actual processor power consumption

Acquired at runtime

Identify program phases with no programmatical knowledge of application



4) Threshold vs. Hits & Misses with Tolerance=1

100% hits with < 5% false alarms, for threshold:

3/14=0.21 – 4/14=0.29



5) How about inst-n based sampling / control flow-based approaches? We have tried 3 methods:

OS/USR counting with PMCsDoesn’t eliminate variability

Binding to threads in sampling Didn’t solve variability/registration problems

Dynamic instrumentation with PinGot back to perfect repeatability

Lost actual benchmark execution behavior that flows thru the processor

PC sampling doesn’t solve variability if we simply sample PCs every 1ms or so. (Application execution time varies)

Sampling at fixed instruction counts is for a specific PID makes it deterministic Has its downsides with uncontrolled timing behavior and not being

able to bind to flow thru processor



6) What’s the Source of Variability? We don’t have perfect, classified answer yet.

Maybe Pin/atom can help

- Different locality at different runs

- Intensity of spontaneous system processes

- Inexact memory access patterns / swaps

- Different cache/tlb/bht etc states



7) Glitches/Gradients vs. Sampling Frequency

Reducing frequency smoothes glitches, BUT dithers gradients More sluggish, LPF’ed response Also smoothes actual phase changes

We use 100ms to meet limitations of high frequency corner: No observable perturbation to actual execution

Limited by RS232 speed

Close lower bound to acquire 3-4 DMM samples



8) What’s the Use of This? First, this is a GENERIC recurrence detection under variability

system!! Can use to detect/predict phases with specific features:

Memory boundness Hotspots

Can be stretched to security/reliability: Matching signatures with PIDs

Specific promising avenues: CMP workload balancing by signatures power Activity migration in the case of hotspot signatures **DVFS at experienced signatures**

Need help from BBVs under phase behavior changes with taken actions!!



9) Multithreaded/OLTP Like Benchmarks?

No fundamental analysis problem as we don’t try to bind to processes

Some of the experimented ones: Mozilla, Xmms, Mplayer

FLAT power behavior Not interesting

Need more infrastructure work to get OLTP like applications running on our platform

Interesting follow-on to see variability of these apps



10) SMT/CMP/Multiprogramming Environments

Don’t have the SMT/CMP platforms hooked up for multimeter (yet)

SMT should be similar, as long as the multi-app behavior is somewhat repeatable

CMP less clear, one PMC set & power measurement per core? Overall per chip?

We have tried multiprogramming on our P4: Memory intensive apps create too much swapping/thrashing for

the behavior to be somewhat repeatable.

Not useful for phase detection

How deterministic is Task switching?



OLD/EXTRA Slides

parapet research group, princeton university ee ieee international symposium on workload...

Documents