top ten secret weapons for performance testing in an agile environment
TRANSCRIPT
Top ten secret weapons for performance testing in an agile environment
[email protected](and [email protected])
© ThoughtWorks 2011
SECRET
WEAPON
#
Make Performance Explicit
© ThoughtWorks 2011
1
SECRET
WEAPON
# So that I can make better investment decisionsAs an investorI want to see the value of my portfolio presented on a single web pagemust have “good” performance, less than 0.2s
page load for about 10,000 concurrent users
© ThoughtWorks 2011
© ThoughtWorks 2011
So that investors have a high-quality experience as the business growsAs the Operations ManagerI want the portfolio value page to render within 0.2s when 10,000 users are logged in
SECRET
WEAPON
#
One Team
© ThoughtWorks 2011
2
© ThoughtWorks 2011
Team Dynamics
© ThoughtWorks 2011
Performance Testers Part of Team
© ThoughtWorks 2011
© ThoughtWorks 2011
Performance Testers Part of Team
© ThoughtWorks 2011
Pair on Performance Test Stories
© ThoughtWorks 2011
Rotate Pairs
SECRET
WEAPON
#
Customer Driven
© ThoughtWorks 2011
3
© ThoughtWorks 2011
What was a good source of requirements?
© ThoughtWorks 2011
Existing Pain Points
© ThoughtWorks 2011
An example...
© ThoughtWorks 2011
So that we can budget for future hardware needs as we growAs the data centre managerI want to know how much traffic we can handle now
© ThoughtWorks 2011
Another example
© ThoughtWorks 2011
So that we have confidence in meeting our SLAAs the Operations ManagerI want to ensure that a sustained peak load does not take out our service
© ThoughtWorks 2011
Personas
© ThoughtWorks 2011
Who is the customer?End Users
OperationsPower Users
Marketing
Investors
SECRET
WEAPON
#
Discipline
© ThoughtWorks 2011
4
© ThoughtWorks 2011
Is the hypothesis
valid?
Change the application
code
Observe test results
Formulate an hypothesis
Design an experiment
Run the experiment
What do you see?
Why is it doing that?
How can I prove that’s what’s happening?
Take the time to gather the evidence.
Safe in the knowledge that I’m making it faster
© ThoughtWorks 2011
© ThoughtWorks 2011
Saw tooth pattern (1 minute intervals)
Directory structure of (yyyy/mm/minuteofday)?. Slow down due to # of files in directory?
1 directory should result in even worse performance...
We ran the test…
Is the hypothesis
valid?
Change the application
code
Observe test results
Formulate an hypothesis
Design an experiment
Run the experiment
One Directory
© ThoughtWorks 2011
SECRET
WEAPON
#
Play Performance Early
© ThoughtWorks 2011
5
© ThoughtWorks 2011
End
Start End
Other projects start performance testing
here
Start
Agile projects start performance testing as
early as possible
SECRET
WEAPON
#
Iterate Don’t (Just) Increment
© ThoughtWorks 2011
6
© ThoughtWorks 2011
© ThoughtWorks 2011
We Sashimi
© ThoughtWorks 2011
Sashimi Slice By... Presentation
© ThoughtWorks 2011
So that I can better see trends in performanceAs the Operations ManagerI want a graph of requests per second
© ThoughtWorks 2011
So that I can better see trends in performanceAs the Operations ManagerI want a graph of average latency per second
© ThoughtWorks 2011
So that I can easily scan results at a single glanceAs the Operations ManagerI want a one page showing all results
© ThoughtWorks 2011
Sashimi Slice By... Scenario
© ThoughtWorks 2011
So that we never have a day like “October 10”As the Operations ManagerI want to ensure that a sustained peak load does not take out our service
© ThoughtWorks 2011
So that we never have a day like “November 12”As the Operations ManagerI want to ensure that an escalating load up to xxx requests/second does not take out our service
SECRET
WEAPON
#
Automate, Automate, Automate
© ThoughtWorks 2011
7
© ThoughtWorks 2011
AutomatedCompilation
AutomatedTests
AutomatedPackaging
AutomatedDeployment
Automation => Reproducible and ConsistentAutomation => Faster FeedbackAutomation => Higher Productivity
Why Automation?
© ThoughtWorks 2011
© ThoughtWorks 2011
AutomatedApplication Deployment
AutomatedLoad
Generation
AutomatedTest
Orchestration
AutomatedAnalysis
Automated Scheduling
Automated Result Archiving
SECRET
WEAPON
#
Continuous Performance Testing
© ThoughtWorks 2011
8
© ThoughtWorks 2011
Application
Build Pipelines
© ThoughtWorks 2011
Performance
© ThoughtWorks 2011
SECRET
WEAPON
#
Test Drive Your Performance Test Code
© ThoughtWorks 2011
9
© ThoughtWorks 2011
V Model Testing
http://en.wikipedia.org/wiki/V-Model_(software_development)
Performance Testing
Slower + Longer
Fast
Speed
© ThoughtWorks 2011
We make mistakes
© ThoughtWorks 2011
V Model Testing
http://en.wikipedia.org/wiki/V-Model_(software_development)
Performance Testing
Slower + Longer
Fast
Speed
Unit test performance
code to fail faster
© ThoughtWorks 2011
Fail Fast!
Fast feedback!
Faster learning
Faster results
© ThoughtWorks 2011
Classic Performance Areas to Test
Analysis
Information Collection
Visualisation
Publishing
Presentation
SECRET
WEAPON
#
Get Feedback
© ThoughtWorks 2011
10
© ThoughtWorks 2011
Frequently (Weekly) Showcase
Here is what we learned this week....
© ThoughtWorks 2011
Frequently (Weekly) Showcase
And based on this... We changed our directory structure.
© ThoughtWorks 2011
Frequently (Weekly) Showcase
Should we do something different knowing this new information?
© ThoughtWorks 2011
List of All Secret Weapons
1. Make Performance Explicit2. One Team3. Customer Driven4. Discipline5. Play Performance Early6. Iterate Don't (Just) Increment7. Automate, Automate, Automate 8. Test Drive Your Performance Code9. Continuous Performance Testing10. Get Feedback
© ThoughtWorks 2011
Photo Credits (Creative Commons licence)• Barbed wire picture: http://www.flickr.com/photos/lapideo/446201948/• Eternal clock: http://www.flickr.com/photos/robbie73/3387189144/• Sashimi from http://www.flickr.com/photos/mac-ash/3719114621/
Questions