cyclic tests unleashed: large-scale rt analysis with jitterdebugger · 2019. 12. 21. · cyclic...

Cyclic Tests Unleashed:Large-Scale RT Analysis with

JitterdebuggerOpen Source Summit Japan 2019

Prof. Dr. Wolfgang MauererTechnical University of Applied Sciences RegensburgSiemens AG, Corporate Research

Daniel WagnerSUSE Linux GmbH(Work done while at Siemens AG)

W. Mauerer and D. Wagner 17. July 2019 1 / 31

About: WMI Siemens Corporate Technology: Corporate Competence Centre Embedded Linux

I Civil Infrastructure PlatformI Technical University of Applied Science Regensburg

I Theoretical Computer ScienceI Head of Digitalisation Laboratory

About: DWI SUSE Linux GmbHI Primary Author of JitterdebugerI Stable-RT Maintainer


Introduction & Overview

About: WMI Siemens Corporate Technology: Corporate Competence Centre Embedded Linux

I Civil Infrastructure PlatformI Technical University of Applied Science Regensburg

I Theoretical Computer ScienceI Head of Digitalisation Laboratory

About: DW


Introduction & Overview

1. Measuring Real-Time Systems2. Jitterdebugger: Cyclictest for Dummies

2.1 Measuring2.2 Archiving

3. Analysis Examples3.1 Comparing Distributions3.2 Time-Resolved Analysis3.3 Estimating Upper Bounds/WCET


1. Measuring Real-Time Systems

Outline

Why Measure RT Systems?I CPUs & systems: Effectively indeterministic these daysI Development/Debugging vs. Deployment Guarantees

Debugging/DevelopmentI Functional Correctness

I Locking etc.I Odd use of system functionalitiesI Functional correctness

I Eliminate large outliersI Triggers and Tracing

Verification/DeploymentI Characterising System Behaviour in

fieldI Reference Distributions (regression

testing)I Satisfy Certification Criteria


1. Measuring Real-Time Systems

Measuring Real-Time Systems I





2. Jitterdebugger: Cyclictest for Dummies

Outline

Key PointsI Few tuneable knobs (deliberately)I PostprocessingI Control load/stress generationI Mass deployments (network)


2. Jitterdebugger: Cyclictest for Dummies

Jitterdebugger: Cyclictest for Dummies I

Do one thing, and do that wellJitterdebugger→ Jittersamples→ Statistical Software

OutputCPUID;Timestamp;Latency


2.1 Measuring

Jitterdebugger II

Do two things, and do/dispatch that wellJitterdebugger + Stress→ Jittersamples→ Statistical Software



2.1 Measuring

Jitterdebugger II

Do two things, and do/dispatch that wellJitterdebugger + Stress→ Archive



2.1 Measuring

Jitterdebugger II

Do two things, and do/dispatch that wellJitterdebugger + Stress→ Send to Host→ Jittersamples→ StatisticalSoftware



2.1 Measuring

Jitterdebugger II

Data HandlingI Reproducible & systematic approachI Regression & comparisonI CertificationI Decouple measurement from statistical

methods

Time ResolutionI Advanced statistical analysisI Machine Learning! AI!I Improve worst-case latency analysis


2.1 Measuring

Jitterdebugger III: Main Advantages

Measuring ProperlyI Reproducibility – can others reproduce/interpret results?I Sufficient Duration – when is certainty achieved?I Traceability – do we understand what’s going on?

First rule of data analysisI 80% of effort: cleaning up dataI 20% of effort: analysis


2.2 Archiving

Archiving I

Reproducibility & Cleanun: Tidy Data1. Each variable forms a column2. Each observation forms a row3. Each type of observational unit forms a table

I Separate file (CSV)I Separate Entity (HDF5)


2.2 Archiving

Archiving II

Jitersamples FormatsI CSV

I You may have heard that beforeI UniversalI Building structures: FS level

I HDF5I Hierarchical Data FormatI 1987: AEHOOI Comprehensive support in analysis software

(R, Octave, Python, Mathematica, Julia, . . . )I Embedded structures in single file

Image Source: www.desy.de/web/mosaic/hdf-browsing.html


2.2 Archiving

Archiving III

www.desy.de/web/mosaic/hdf-browsing.html

Jitersamples FormatsI CSV

I You may have heard that beforeI UniversalI Building structures: FS level

I HDF5I Hierarchical Data FormatI 1987: AEHOOI Comprehensive support in analysis software

(R, Octave, Python, Mathematica, Julia, . . . )I Embedded structures in single file

Organisation: Best PractiseI Identical filename for each

measurementI Active parameters: directoriesI Derived parameters: file(s)I Reproducibility: Keep microcode

binaries, non-upstream patches etc. aspart of measurement results

Image Source: www.desy.de/web/mosaic/hdf-browsing.html


2.2 Archiving

Archiving III

www.desy.de/web/mosaic/hdf-browsing.html

Recorded VariablesI Kernel version + parameters (activated fixes etc.)I Microcode version(s)I Online CPUsI Usual suspects: Included by Jitterdebugger by default


2.2 Archiving

Archiving IV





3. Analysis Examples

Outline

Three ways of understanding data1. Descriptive analysis2. Exploratory analysis3. Confirmatory analysis

Order mattersI Simple analyses before formal testI Proper visual understanding often more important than explicit calculations



Visualising and Understanding Data

1. Fine-grained detail analysis2. Improving/quantifying trustworthiness of WCET estimates



Analysis Examples

New 2 New 3

Base New 1

0 50 100 150 0 50 100 150

50000

100000

150000

50000

100000

150000

Latency

Cou

nt


3.1 Comparing Distributions

Comparing Distributions I

Why?I Track behavioural changes after

system changesI Load vs. idle behaviourI Evaluate alternative choices

How?I Comparing summaries 7

I Visual/explorative inspection 3

I Formal methods/tests 37



Comparing Distributions II

(Empirical) Cumulative Distribution FunctionI Sums up fraction of values that fall into [0, x] at position xI Parameter free!I Interpretation requires trained eye



Comparing Distributions II

New 2 New 3

Base New 1

50 75 100 125 50 75 100 125

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Latency

Cou

nt

ggplot(dat, aes(x=latency)) + stat_ecdf() + facet_wrap(~type)



Comparing Distributions III

Point of viewWhich one is better?

Formal TestsI t-test: Check with identical mean values (Gaussian distribution/large sample size)I Wilcoxon signed-rank test: Test for identical distributions

Visual TestsI Quantile-Quantile plot 37

I Facetted(!) histograms 37

I Empirical cumulative distribution functions (ecdf) 3



Comparing Distributions IV

ECDF & Probabilities for Worst-Case Latencies

New 2 New 3

Base New 1

60 80 100 120 60 80 100 120

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Latency

ED

CF

I pe: Probability of exceeding WCET wI w = F−1(1− pe)



Comparing Distributions V

9 10

5 6 7 8

1 2 3 4

0 20 40 60 0 20 40 60

0 20 40 60 0 20 40 60

2e−044e−046e−048e−04

0.000250.000500.00075

5e−04

1e−03

5e−04

1e−03

1e−042e−043e−044e−04

0.000250.000500.000750.00100

0.002

0.0040.006

0.001

0.0020.003

1e−042e−043e−044e−045e−04

0.00050.00100.00150.00200.0025

Latency[us]

Den

sity

[log

, a.u

.]

ggplot(dat, aes(x=Latency)) + geom_density() + facet_wrap(~range) + scale_y_sqrt()


3.2 Time-Resolved Analysis

Time-Resolved Analysis

9 10

5 6 7 8

1 2 3 4

0 20 40 60 0 20 40 60

0 20 40 60 0 20 40 60

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Latency[us]

y

ggplot(dat, aes(x=Latency)) + stat_ecdf() + facet_wrap(~range)




1 2 3 4 5 6 7 8 9 100

12

3

0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60

0.000.250.500.751.00

0.000.250.500.751.00

0.000.250.500.751.00

0.000.250.500.751.00

Latency[us]

y

ggplot(dat, aes(x=Latency)) + geom_density() + facet_wrap(CPU~range)




1 2 3 4 5 6 7 8 9 100

12

3

0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60

1e+052e+053e+05

1e+052e+053e+05

1e+052e+053e+05

1e+052e+053e+05

Latency[us]

coun

t

ggplot(dat, aes(x=Latency)) + geom_histogram(bins=100) + facet_wrap(CPU~range) + scale_y_sqrt()




01

23

0 20 40 60

2.5

5.0

7.5

10.0

2.5

5.0

7.5

10.0

2.5

5.0

7.5

10.0

2.5

5.0

7.5

10.0

Maximal Latency[us]

Tim

e R

ange

ggplot(dat.max, aes(x=Latency)) + geom_bar(stat=’identity’) + facet_wrap(CPU~.)




9 10

7 8

5 6

3 4

1 2

0 20 40 60 0 20 40 60

0123

0123

0123

0123

0123

0123

0123

0123

0123

0123

Maximal Latency[us]

Cor

e

ggplot(dat.max, aes(x=Latency)) +geom_bar(stat=’identity’) + facet_wrap(range~.)




Accuracy: ProblemsI Correct tail classification: Huge number of samplesI Values larger than sample cannot appearI How reliable is a measurement?



Comparing Distributions VI

StatementsI “The highest threshold we’ve seen is XYZ”

I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ”

while TimBird was standing in front of the system

I “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEFpercent”


3.3 Estimating Upper Bounds/WCET

WCET Estimation I

StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ”

while TimBird was standing in front of the system




WCET Estimation I

StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ” while Tim

Bird was standing in front of the system




WCET Estimation I

StatementsI “The highest threshold we’ve seen is XYZ”I “After XYZ hours of measuring, the highest threshold we’ve seen is XYZ” while Tim

Bird was standing in front of the systemI “At a confidence level of ABC, the probability to exceed threshold XYZ is at most DEF

percent”



WCET Estimation I

Traditional: Two-Step Approach1. Measure2. Determine Maximum

New: Three-Step Approach1. Measure2. Determine model from measurement3. Infer WCET and uncertainty/credibility from model



WCET Estimation II

Model PropertiesI Generality

I Applicability to multiple real-world situationsI Realism

I Accurately represent real-world phenomenaI Precision

I Minimise errors compared to real-world results



WCET Estimation III

Model PropertiesI Generality 3

I Applicability to multiple real-world situationsI Realism 7

I Accurately represent real-world phenomenaI Precision 3

I Minimise errors compared to real-world results



WCET Estimation III

Fundamental: Random VariablesI Random Variable XI Measurements/observations x1, x2, . . . , xN sampled from X

Random Variable 6= Random Properties

1 2 3 25

1 2 3 4 5 2.5 5.0 7.5 10.0 4 8 12 50 60 70 80 900

20

40

60

0

50

100

150

0

50

100

150

200

0

50

100

150

200

Sum of Dice

Cou

nt



WCET Estimation IV

Latencies: Maxima matter!I Latency is a random variable XI Measurements x1, x2, . . . , xN sampled from X

I Quantity of interest: max(x1, x2, . . . , xN )



WCET Estimation V

What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})

I Jitterdebugger: Postprocess results to obtain block maxima!



WCET Estimation V

What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})

I Jitterdebugger: Postprocess results to obtain block maxima!

Generalised Extreme Value Distribution

G(z) = exp

(−(1 + ξ

(z − µσ

)) 1ξ

)I Distribution function for (extremely) rare eventsI Determine parameters from measured maxima

I ξ: Distribution shapeI σ: ScaleI µ: Location



WCET Estimation V

What can we say about maxima?I Independent and identically distributed (iid) random variables {X1, X2, . . . , Xn}I Maxima: max({X1, X2, . . . , Xn})I Jitterdebugger: Postprocess results to obtain block maxima!

Generalised Extreme Value Distribution

G(z) = exp

(−(1 + ξ

(z − µσ

)) 1ξ

)I Distribution function for (extremely) rare eventsI Determine parameters from measured maxima

I ξ: Distribution shapeI σ: ScaleI µ: Location



WCET Estimation V

Fit ParametersI gev(dat.measured$max)I ξ = 0.229± 0.027, σ = 5025.82± 147.45, µ = 9340.03± 182.53

But Wait! Model Assumptions + Quality!I Carefully inspect diagnostic plotsI Prediction Quality: Usual ML approach

I Split dataI Model building/testing



WCET Estimation V

ProblemsI IID assumption

I Caches, Branch Prediction: No independence between runsI Continuous vs. discrete distribution

I Only a discrete set of runtimes is possible

Don’t Forget: Models and/vs. RealityI No model is correctI Some models may be useful



WCET Estimation VI

Thanks for your interest!



cyclic tests unleashed: large-scale rt analysis with jitterdebugger · 2019. 12. 21. · cyclic...

Documents