automatic load test verification using control charts
TRANSCRIPT
1
Automatic Load Test Verification Using Control Charts
Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. HassanQueen’s University, Kingston, Canada
Mohamed Nasser, Parminder FloraResearch in Motion, Waterloo, Canada
2
Load testing: Ensure the software performance
With 2000 concurrent users, a GET request should be responded within 2 seconds
3
Development
FunctionalTesting
LoadTesting
4
Importance of software performance:More users
Load testing is regression based
5
Apply operational load e.g. 2000 requests/min
Existing version e.g. version 1.2
New release e.g. version 1.3
BaselinePerformance counters
TargetPerformance counters
Compare the counters to detect performance
regressionCode changes
Automatede.g. HP LoadRunner,MS Visual Studio, …
Still a manual process
6
Challenge in load testing: Lots of data
There are thousands of counters with millions of samples
Frontend BackendClient
Client
Client
Frontend
Client
Frontend
Client
Backend
Backend
Load balancer
CPU%User CPU%Privileged CPU%IO data bytes/secIO read bytes/sec
IO write bytes/secPage faults/secPool paged bytesPrivate bytesVirtual bytes
7
Manufacturing process also requires automatic verification
8
Control charts example
0 5 10 15 20 25710
720
730
740
750
760
770
780
Milk Filled (ml)
CL
LCL
UCL
Violation ratio: 0% 20% 20% 60%
9
Using control charts to verify load test results
BaselinePerformance counters
TargetPerformance counters
Determine the UCL, CL, LCL
0 5 10 15 20 25710720730740750760770780
Performance counter
10
Using control charts to verify load test results
BaselinePerformance counters
TargetPerformance counters
Determine the UCL, CL, LCL
0 5 10 15 20 25710720730740750760770780
Performance counter
Violation ratio
11
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
0 5 10 15 20 25710720730740750760770780
Performance counter
BaselinePerformance counters
0 5 10 15 20 25680
700
720
740
760
780
Performance counter
Low violation ratio
High violation ratio
12
We conduct two case studies
• Enterprise system:– Large and complex– Multi-tiers
• Dell DVD store– Small and simple– 3-tiers
13
Type Description
Front end problems
CPU hog a) 50% and b) 75% of CPU on the front end server
Back end problems
CPU hog a) 50% and b) 75% of CPU on one of the back end servers
Busy waiting Busy waiting in the front end server code
Query Extra query to the database back end
NormalPerformance counters
ProblemsPerformance counters
Type Description
Normal Runs of existing version without performance problems
RQ1: Can control charts detect a problematic run?
14Enterprise’s Response Time
Normal Front 50% Front 75% Back 50% Back 75%0
10
20
30
40
50
60
70
80
90
100
Violation ratio
15DVD Store’s Response Time
Normal Front 50% Front 75% Back 50% Back 75% Busy Query0
10
20
30
40
50
60
70
80
90
100
Violation Ratio
16
RQ2: Can control charts identify the problematic component?
Frontend BackendClient
Client
Client
Frontend
Client
Frontend
Client
Backend
Backend
Load balancer
High violation in frontend’s counters
High violation in backend’s counters
17
Backend 50% stress
F B B B B F O B F O0%
10%20%30%40%50%60%70%80%90%
100%
Violation ratioBaseline
Target
18
Filter out unstable countersBaseline
Target
F F F F B B B B B B0%
10%20%30%40%50%60%70%80%90%
100%
Violation ratio
These counters are always out-of-control We filter then out
19
Backend 50% stress
B B B B B O0%
10%20%30%40%50%60%70%80%90%
100%
Violation ratioBaseline
Target
20
Frontend 50% stress
F F F F F F O0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Violation ratioBaseline
Target
21