behavior-driven load testing using contextual knowledge · 2020-04-17 · behavior-driven load...

Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung

10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)

Mumbai, India – April 11, 2019

Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –

Henning Schulz, Dušan Okanović,

André van Hoorn

Vincenzo Ferme, Cesare Pautasso

@andrevanhoorn

Load Testing – Approach and Challenges

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 3

Workload

specification

SUTLOAD

Test results

Load

driver

System

under Test

Expert

High manual effort for maintaining load tests

Load tests need much time to execute

There are no suitable load tests

BenchFlow – Declarative Load Testing

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 4

Background

SUTLOAD

Workload

specification

(declarative)

Load

driver

System

under Test

BenchFlow

SUT definition

Test results

• Generation of executable artifacts

• SUT deployment

• Load test execution

• Collection of test results

Ferme et al., ICPE 2018

ExpertExample BenchFlow specification

Manual workload specification needed.

ContinuITy – Continuous Load Testing in DevOps

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 5

Background

Monitoring data

Workload

specification

SUTLOAD

Test resultsContextual

information

ContinuITy

Load

driver

System

under Test

• Workload model extraction

• Workload model evolution

• Workload prediction

• Workload model selection

Schulz et al., LTB@ICPE 2018

workload

contextual

knowledge

Black Friday Black Friday

Christmastime

past future

Workload – prediction and context

Limited load test execution automation.

Behavior-driven (Functional) Testing – Workflow

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 6

Background

ThenWhenGiven

Examples ©

Initial context

Changes

Outcome

Behavior-driven Load Testing (BDLT) – Example

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 7

Approach

BDLT definition

then

when

Given

BDLT example definition

Given the next Black Friday,

when varying the CPU cores between 1 and 4,

then run the experiment for 1h

andensure the maximum CPU utilization is less than 60%.

thenoutcome

quality gates

run time

metrics

whenchanges

num. users

event

app config

Giveninitial context

event

app config

date (range)

Behavior-driven Load Testing (BDLT) – Workflow

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 8

Approach

Monitoring data

Workload

specification

SUTLOAD

Test resultsContextual

information

ContinuITy

Workload

specification

(declarative)

Load

driver

System

under Test

BenchFlow

BDLT definition

Then

When

GivenSUT definition

BDLT Language Definition

EBNF-based BDLT grammar with extension points for events

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 9

Giveninitial context

event

app config

date (range)

Whenchanges

num. users

event

app config Thenoutcome

quality gates

run time

metrics

Mapping from BDLT to ContinuITy and BenchFlow

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 10

Evaluation – Research Questions

How expressive is the BDLT language in regards to load test concerns of industrial use cases?

RQ1

How would BDLT be used in industrial contexts?RQ2

What are the benefits and limitations of using BDLT in comparison to defining load test scripts?

RQ3

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 11

Case Study – System and Method

● System under study

● IoT system from automotive sector

● IoT endpoint migrating to Cloud

● Methodology

○ Workshops

○ We iteratively developed test specifications based on expert feedback

○ Incorporation of production monitoring data (per-device behavior and intensity)

● Four scenarios

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 12

Case Study – Configuration Exploration

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 13

Given the next three monthsandthe number of users set to the maximum

when varying the CPU cores

between 0.5 and 4 in steps of 0.5andvarying the number of instancesbetween 1 and 5

andvarying the RAM among (1GB, 2GB, 4GB)

then run each experiment for 1 hourandensure the average CPU load is less than15%

andensure the message latency is less than2 seconds

Case Study – Recovery Spike

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 14

Given 2018/10/15 9:00

when an outage happened from2018/10/15 7:00 to 2018/10/15 9:00

then run the experiment for 2 hours

and

ensure the final queue length isless than 100

Evaluation – Lessons Learned

How expressive is the BDLT language in regards to load test concerns of industrial use cases?

RQ1

How would BDLT be used in industrial contexts?RQ2

What are the benefits and limitations of using BDLT in comparison to defining load test scripts?

RQ3

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 15

Could express all use cases

Custom extensions needed

Could replace manual tests

Natural language helps understanding

and communication (with non-experts)

“The approach has potential“

Some concerns hard to express

Current (technical) limitation to HTTP

Helps identifying new concerns

Too many (technical) details in the Then clause?

+ Additional (laboratory) case study

+ Supplementary material

• Extending the language

• Improving tool support

• Further evaluation

• Supporting regression queries

• Integration into agile development

methods, e.g., user stories from

Scrum tickets

• Integration into Declarative Performance

Engineering landscape

Schulz, Okanovic, van Hoorn, et al. | Behavior-driven Load Testing Using Contextual Knowledge – Approach and Experiences 16

Summary Future Work

Walter et al., ICPE 2016

Avritzer et al.,ECSA 2018

Untersuchung der Prozessgrenzen und Wärmeakkumulationbei der Präzisionsbear-beitung von Metall mit ultrakurz gepulster Laserstrahlung

10th ACM/SPEC Int. Conf. on Performance Engineering (ICPE 2019)

Mumbai, India – April 10, 2019

Behavior-driven Load Testing Using Contextual Knowledge– Approach and Experiences –

Henning Schulz, Dušan Okanović,

André van Hoorn

Vincenzo Ferme, Cesare Pautasso

@andrevanhoorn

References

Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz,

André van Hoorn: A Quantitative Approach for the Assessment of Microservice

Architecture Deployment Alternatives by Automated Performance Testing. ECSA

2018: 159-174

Vincenzo Ferme, Cesare Pautasso: A Declarative Approach for Performance Tests

Execution in Continuous Software Development Environments. ICPE 2018: 261-272

Henning Schulz, Tobias Angerstein, André van Hoorn: Towards Automating

Representative Load Testing in Continuous Software Engineering. ICPE Companion

2018: 123-126

Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, Helmut

Krcmar: WESSBAS: extraction of probabilistic workload specifications for load

testing and performance prediction - a model-driven approach for session-based

application systems. Software and System Modeling 17(2): 443-477 (2018)

Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, Samuel Kounev:

Asking "What"?, Automating the "How"?: The Vision of Declarative Performance

Engineering. ICPE 2016: 91-94

Ferme et al., ICPE 2018

Schulz et al., LTB@ICPE 2018

Walter et al., ICPE 2016

Avritzer et al.,ECSA 2018

Vögele et al., SoSyM 2016