behavior-driven load testing using contextual knowledge · 2020-04-17 · behavior-driven load...
TRANSCRIPT
Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung
10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)
Mumbai, India – April 11, 2019
Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –
Henning Schulz, Dušan Okanović,
André van Hoorn
Vincenzo Ferme, Cesare Pautasso
@andrevanhoorn
Load Testing – Approach and Challenges
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 3
Workload
specification
SUTLOAD
Test results
Load
driver
System
under Test
Expert
High manual effort for maintaining load tests
Load tests need much time to execute
There are no suitable load tests
BenchFlow – Declarative Load Testing
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 4
Background
SUTLOAD
Workload
specification
(declarative)
Load
driver
System
under Test
BenchFlow
SUT definition
Test results
• Generation of executable artifacts
• SUT deployment
• Load test execution
• Collection of test results
Ferme et al., ICPE 2018
ExpertExample BenchFlow specification
Manual workload specification needed.
ContinuITy – Continuous Load Testing in DevOps
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 5
Background
Monitoring data
Workload
specification
SUTLOAD
Test resultsContextual
information
ContinuITy
Load
driver
System
under Test
• Workload model extraction
• Workload model evolution
• Workload prediction
• Workload model selection
Schulz et al., LTB@ICPE 2018
workload
contextual
knowledge
Black Friday Black Friday
Christmastime
past future
Workload – prediction and context
Limited load test execution automation.
Behavior-driven (Functional) Testing – Workflow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 6
Background
ThenWhenGiven
Examples ©
Initial context
Changes
Outcome
Behavior-driven Load Testing (BDLT) – Example
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 7
Approach
BDLT definition
then
when
Given
BDLT example definition
Given the next Black Friday,
when varying the CPU cores between 1 and 4,
then run the experiment for 1h
andensure the maximum CPU utilization is less than 60%.
thenoutcome
quality gates
run time
metrics
whenchanges
num. users
event
app config
Giveninitial context
event
app config
date (range)
Behavior-driven Load Testing (BDLT) – Workflow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 8
Approach
Monitoring data
Workload
specification
SUTLOAD
Test resultsContextual
information
ContinuITy
Workload
specification
(declarative)
Load
driver
System
under Test
BenchFlow
BDLT definition
Then
When
GivenSUT definition
BDLT Language Definition
EBNF-based BDLT grammar with extension points for events
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 9
Giveninitial context
event
app config
date (range)
Whenchanges
num. users
event
app config Thenoutcome
quality gates
run time
metrics
Mapping from BDLT to ContinuITy and BenchFlow
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 10
Evaluation – Research Questions
How expressive is the BDLT language in regards to load test concerns of industrial use cases?
RQ1
How would BDLT be used in industrial contexts?RQ2
What are the benefits and limitations of using BDLT in comparison to defining load test scripts?
RQ3
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 11
Case Study – System and Method
● System under study
● IoT system from automotive sector
● IoT endpoint migrating to Cloud
● Methodology
○ Workshops
○ We iteratively developed test specifications based on expert feedback
○ Incorporation of production monitoring data (per-device behavior and intensity)
● Four scenarios
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 12
Case Study – Configuration Exploration
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 13
Given the next three monthsandthe number of users set to the maximum
when varying the CPU cores
between 0.5 and 4 in steps of 0.5andvarying the number of instancesbetween 1 and 5
andvarying the RAM among (1GB, 2GB, 4GB)
then run each experiment for 1 hourandensure the average CPU load is less than15%
andensure the message latency is less than2 seconds
Case Study – Recovery Spike
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 14
Given 2018/10/15 9:00
when an outage happened from2018/10/15 7:00 to 2018/10/15 9:00
then run the experiment for 2 hours
and
ensure the final queue length isless than 100
Evaluation – Lessons Learned
How expressive is the BDLT language in regards to load test concerns of industrial use cases?
RQ1
How would BDLT be used in industrial contexts?RQ2
What are the benefits and limitations of using BDLT in comparison to defining load test scripts?
RQ3
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 15
Could express all use cases
Custom extensions needed
Could replace manual tests
Natural language helps understanding
and communication (with non-experts)
“The approach has potential“
Some concerns hard to express
Current (technical) limitation to HTTP
Helps identifying new concerns
Too many (technical) details in the Then clause?
+ Additional (laboratory) case study
+ Supplementary material
• Extending the language
• Improving tool support
• Further evaluation
• Supporting regression queries
• Integration into agile development
methods, e.g., user stories from
Scrum tickets
• Integration into Declarative Performance
Engineering landscape
Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 16
Summary Future Work
Walter et al., ICPE 2016
Avritzer et al.,ECSA 2018
Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung
10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)
Mumbai, India – April 10, 2019
Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –
Henning Schulz, Dušan Okanović,
André van Hoorn
Vincenzo Ferme, Cesare Pautasso
@andrevanhoorn
References
Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz,
André van Hoorn: A Quantitative Approach for the Assessment of Microservice
Architecture Deployment Alternatives by Automated Performance Testing. ECSA
2018: 159-174
Vincenzo Ferme, Cesare Pautasso: A Declarative Approach for Performance Tests
Execution in Continuous Software Development Environments. ICPE 2018: 261-272
Henning Schulz, Tobias Angerstein, André van Hoorn: Towards Automating
Representative Load Testing in Continuous Software Engineering. ICPE Companion
2018: 123-126
Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, Helmut
Krcmar: WESSBAS: extraction of probabilistic workload specifications for load
testing and performance prediction - a model-driven approach for session-based
application systems. Software and System Modeling 17(2): 443-477 (2018)
Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, Samuel Kounev:
Asking "What"?, Automating the "How"?: The Vision of Declarative Performance
Engineering. ICPE 2016: 91-94
Ferme et al., ICPE 2018
Schulz et al., LTB@ICPE 2018
Walter et al., ICPE 2016
Avritzer et al.,ECSA 2018
Vögele et al., SoSyM 2016