automated detection of performance regressions using statistical process control techniques
TRANSCRIPT
![Page 1: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/1.jpg)
1
Automated Detection of Performance Regressions Using Statistical Process Control
Techniques
Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. HassanQueen’s University, Kingston, Canada
Mohamed Nasser, Parminder FloraResearch in Motion, Waterloo, Canada
![Page 2: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/2.jpg)
2
Performance Regression
![Page 3: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/3.jpg)
3
What is a performance regression?
Version 1 Version 1.1
Baseline Target
![Page 4: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/4.jpg)
4
How to detect performance regression?
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
![Page 5: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/5.jpg)
Challenge in Performance Regression Testing
5
Layer 1 – Agent 1
Layer 1 – Agent 2
Layer 2 – Agent 1
Layer 2 – Agent 2
Layer 2 – Agent 3
Layer 2 – Agent 4
Layer 3 – Agent 1
Layer 4 – Agent 1
56 counters x 8 agents = 448 counters
56 counters x 2 agents = 112 counters
Layer 1 Layer 2
Lots of data
![Page 6: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/6.jpg)
6
Data mining
![Page 7: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/7.jpg)
7
Data mining -> Reduce and Relate
Reduce Relate
![Page 8: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/8.jpg)
8
Proposed approach to use control charts to find performance regression
BaselinePerformance counters
TargetPerformance counters
Determine the LCL, CL, UCL
0 5 10 15 20 25710
720
730
740
750
760
770
780
Performance counter
![Page 9: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/9.jpg)
9
Using control charts to verify load test results
BaselinePerformance counters
TargetPerformance counters
Determine the LCL, CL, UCL
0 5 10 15 20 25710
720
730
740
750
760
770
780
Performance counter
Violation ratio
Reduce
![Page 10: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/10.jpg)
10
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
0 5 10 15 20 25710720730740750760770780
Performance counter
BaselinePerformance counters
0 5 10 15 20 25680
700
720
740
760
780
Performance counter
Low violation ratio
High violation ratio
We can use violation ratio to detect regression Relate
![Page 11: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/11.jpg)
11
Is there performance regression?
![Page 12: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/12.jpg)
12
Obstacles #1: Inputs are unstable
1 2 3 4 5 60
5
10
15
20
25
30
35
40
45
Version 1.0Version 1.1
Time
CPU
%
Is there a performance regression?
![Page 13: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/13.jpg)
13
It is very difficult to maintain stable input across test runs
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
Randomization CacheWarm up
Background tasks
![Page 14: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/14.jpg)
14
Solution #1: Scale the counter according to the input
• Step 1: Determine α and β
• Step 2:CPU% Request/s
![Page 15: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/15.jpg)
15
Solution #1: Example of the effectiveness of scaling
![Page 16: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/16.jpg)
16
Obstacles #2: Multiple inputs
10 20 30 40 50 60 70 80 90 1000
5
10
15
20
25
30
35
Density plot of two test runs
CPU Usage
Dens
ity %
IF … THEN…ELSE…
![Page 17: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/17.jpg)
17
10 20 30 40 50 60 70 80 90 1000
5
10
15
20
25
30
35
Density plot of two test runs
CPU Usage
Dens
ity %
Solution #2: Isolating the counters
Local minima
![Page 18: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/18.jpg)
18
Scale and filter
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
Scale
Scale
Filter
Filter
![Page 19: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/19.jpg)
19
Case study 1
![Page 20: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/20.jpg)
20
Experiment set up
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
Average violation ratio should be low
![Page 21: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/21.jpg)
21
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
Average violation ratio should be high
Experiment set up
![Page 22: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/22.jpg)
22
Bad Query
Limit
Extra p
rint
Extra c
onnection
Missing k
ey index
Missing t
ext index
0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
100.00%
NormalProblem
Series3
Average violation ratio
![Page 23: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/23.jpg)
23
Case study 2
![Page 24: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/24.jpg)
24
Experiment set up
V.S.
Precision is highRecall should be high
![Page 25: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/25.jpg)
25
0
10
20
30
40
50
60
70
80
90
100
PrecisionRecallF
Threshold
%
![Page 26: Automated Detection of Performance Regressions Using Statistical Process Control Techniques](https://reader034.vdocuments.us/reader034/viewer/2022052318/58f0c18b1a28ab90628b4585/html5/thumbnails/26.jpg)
26