the secret art of agile performance testing

The Secret Art of Agile Performance TestingAlois Reitbauer, dynaTrace Software

The Classical Approach

Waterfalls are pretty

But might get scary

Dream

Comfort Zone

Reality

Iteration

Iteration

Iteration

Test

Test

Dream

Comfort Zone

Reality

Iteration

Iteration

Iteration

Test

Test

Iteration

Iteration

Iteration

NFRs and Maslow

The Focus Problem

Methodology

The Agile Approach

Defining our Strategy

Start Early

Test Continuously

Break in Pieces

Creating FocusRi

sk

Impact

AutomateRegular TestingSkip

Frequency vs. Granularity

Frequency

Gra

nu

lari

ty

JUnit-basedTests (2x day)

Total SystemTests

Long-running Stabiltiy Tests(2 w duration)

Granularity

Comparability

Complexity

Quality

Define KPIs

Throughput

Response Time

Memory Consumption

Other KPIs …

MonitorGain Visibility

ResolveReduce Resolution

Time

PreventProactively Avoid

Problems

Follow a Strategy

Avoid Re-Runs

• What could happen?• Which information do you

want?• What describes your system?• What is different from the last

run?

Don´t Forget the Real World

Lessons learned

Profiling is not enough

Good for finding problems

Result comparison hard

Only valid until next check-in

Too much work

Measurements

Stability of Tests

Use Dedicated Hardware

Comparability

Stability

Efficiency

Trends in Unstable Tests

Create Instability

.. adding some volatility increases the likelyness to discover problems …“

Testing scalability

Small Dump Operations Big Dump Operations

Understand your measurements

Response Time only Response Time and GC

Motivate your team

How to make developers write tests

#1 Heroism#2 Boomerang#3 The other guy#4 Bug me not#5 Feedback#6 Code vs. Wine#7 Newb vs. Noob

Test Case Complexity

FirstStart dynaTrace infrastructureWhen readyStart n WebSphere instances on servers … When readyStart Loadtest against WebSphere serversAfter loadtest startExecute test case

Making complex things easy

@DtdRemoteSud(            host = "lab2",            name = "WAS7.0",            startupPriority = 1,            postStartClosure = WaitForWebSphereSudIsUp.class    )    private SudInterface webSphereSud;

Finding the responsible code

Version Control History Lookup

E-Mail Notification

… better humans

Wrap Up1. Say „Yes“2. Focus3. Define KPIs4. Redefine Done5. Remove Barriers6. Automate7. Monitor8. Resolve9. Prevent

alois.reitbauer@dynatrace.com Mailblog.dynatrace.com Blog

AloisReitbauer Twitter