Download - Performance Testing in the Agile Enterprise
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 1
Performance Testing in the Agile Enterprise
Chief Technologist, PerfTestPlus, Inc. [email protected] www.perftestplus.com
Co-Founder: Workshop On Performance and Reliability
www.performance-workshop.org
by: Scott Barber
Co-Author: Author: Contributing Author:
Books: www.perftestplus.com/pubs About me: about.me/scott.barber
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 2
Have you ever wondered…
teams have no idea where performance “fits” in an Agile
development lifecycle?
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 3
Because performance
“fit” in an Agile development lifecycle…
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 4
Performance is (or should be)
in an Agile development lifecycle!
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 5
Overview
Performance Testing is inherently Agile, but...
Integrating Performance Testing
into Agile Efforts effectively is not inherently easy!
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 6
Agility in Performance Testing Performance Testing is a repeating cycle of:
Complicated by unknowns, estimations, &
approximations
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 7
Agility in Software Development Agile is a repeating cycle of:
Gated by inexact and
variable notions of acceptance
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 8
Agile Perf Testing in Agile Dev Putting them together is…
…a tad complicated
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 9
Conceptual Models “On Demand”
• A.K.A. Center of Excellence • Call ‘em when you (think you) need ‘em • “Pay” only for what you (think you) need
“On Retainer”
• Performance Tester on team, but mostly independent • Always available
“Full Immersion” • Performance is everyone’s job • Coordinated by embedded Performance Test Specialist • Spans entire lifecycle
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 10
On Demand Roles & Interactions:
• Similar to outsourcing except that… • you expect familiarity with the project/product (unlikely) • you expect familiarity with the process (less likely) • you won’t get 100% focus on your project • the assigned resource(s) also report to someone who can
trump the project manager • Minimal interaction with team • In general, they collect info, run tests, & report results.
Strengths: • Fewer performance testers can service more projects • Pooled tools and environments can save time and money • Performance testers benefit from peer collaboration
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 11
Performance Test Cycle; On Demand
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 12
On Demand Challenges:
• Service tends to be “generic” • Skills/tools/environments may not work for *your* project • Availability often restricted to highest priority projects • Performance testing tends to be both late and infrequent • Validation of tuning often has to wait until the next “on
demand cycle” • No continuity release to release • A core team member must serve as performance lead • Core team responsible for performance exploration
∆ from non-Agile efforts: • Difficult to schedule • System under test a faster moving target • Testing can’t keep up with continuous production delivery
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 13
On Retainer Roles & Interactions:
• Performance testing specialist • is embedded into the team full time • may be managed by Project Manager, Product Manager,
or Sr. Engineer • Performance is not (yet) integral to project team culture
Strengths: • Performance testing specialist
• knows project/product/culture/environment details • customizes service to those details • builds relationships that enhance “performance culture”
• Always available • Many more tests conducted than On Demand • Performance exploration and tuning validation “in scope”
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 14
Performance Test Cycle; On Retainer
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 15
On Retainer Challenges:
• Performance tester(s), tool(s), and environment(s) needed for every project
• Performance testers • Are often seen as “outsiders” • Tend to loose peer collaboration • Might not be fully utilized
• Performance assessments still tend to occur at end of sprints
• No additional support during “crunch time” ∆ from non-Agile efforts:
• Planning is necessarily more reactive • Tool and environment sharing virtually impossible • Testing done in smaller segments
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 16
Full Immersion Roles & Interactions:
• Performance specialist coordinates performance related activities including:
• Creation of performance stories • Code profiling and performance unit testing • Performance assessment • Collaboration with production support/capacity planning
• Performance is part of everyone’s job, every day Strengths:
• Performance is, or is becoming, an integral part of the team culture
• No delays, performance testing is done in line with dev • Cross-functional & cross-departmental collaboration • Best chance of no performance nightmares
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 17
Performance Test Cycle; Full Immersion
See role and task details in additional resource slides included at the end of this deck
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 18
Full Immersion Challenges:
• Performance tester(s), tool(s), and environment(s) needed for every project
• Organizational culture shift typically required • Demands an “architect level” performance specialist • Getting team dynamics “right” takes time – often 2-3
projects ∆ from non-Agile efforts:
• Performance testing at many application levels, daily • Root cause analysis dramatically easier • Tuning is continual • Unity of purpose and goals is essential • The only “external” gates are acceptance testing and
executive business decisions
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 19
First Steps Involve management in evolving performance culture Get performance into user stories
• Response times • Resource allocations
Involve developers • Code profiling • Integrate performance measurement into unit tests
Make performance part of story acceptance • Quantitative • Qualitative
Involve performance testers • Throughout the cycle • Gain respect as advisors as well as testers
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 20
Valuable Resources Performance Testing Guidance for Web Applications
• Microsoft patterns&practices 2009 • *especially* Chapters 1-7 • Free from http://perftestingguide.codeplex.com/
How to Reduce the Cost of Testing • CRC Press, 2011 • Chapter 16 (free from PerfTestPlus.com) • Order book on Amazon
Other Articles and Websites • An Explanation of Performance Testing on an Agile
Team (part 1, part 2) • What is performance testing? • Software performance testing: There is no 'I' in 'team'
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 21
Questions
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 22
Contact Info
Scott Barber Chief Technologist
PerfTestPlus, Inc E-mail: [email protected]
Web Site: www.PerfTestPlus.com
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 23
Oversee, Review and/or
Approve
• Executives set the tone • Articulate performance goals and risks to the business • The rest of the team must
• Achieve goals • Mitigate risks • Assess & report both -- continually
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 24
Build Capacity Model for n+2 Design n+2Architect &
Model n+2Supervise,
Troubleshoot, Validate and/
or Update Architecture
Model
Observe, Advise,
Troubleshoot, Validate and/
or UpdateArchitecture
Model
Cross-Validate n+2
Capacity Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
• Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Troubleshoot, tune, update design & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 25
Baseline n+1 in Prod
Build Capacity Model for n+2
Reverse Cross-Validate Release
Readiness Perf Tests in Staging
Validate n+2
Capacity Predictions
in Prod
Cross-Validate n+2
Capacity Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
Provide Current Production Info & Models, Consult When Needed
• Be “hardware architects” • Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Provide production feedback • Troubleshoot, tune, update design & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 26
Add Performance & Profiling Stories
Develop Stories; Profile; Unit
Level Performance
Testing
Design n+2
• Use coding practices that support good performance • Code to achieve goals, targets, budgets & mitigate risk • Assess components continually • Troubleshoot, tune, & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 27
Roles & Tasks (Reference)
Reverse Cross-Validate Release
Readiness Perf Tests in Staging
Project Investigative Performance
Testing
SystemInvestigative Performance
TestingCross-
Validate n+2 Capacity
Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
Observe, Consult and
Assist Where
Appropriate
Update Project Performance
Strategy; Design/Develop
Investigative Performance
Tests
Prod Readiness
Perf Testing (System)
• Manage whole system assessment • Own simulation (usage) model(s) • Contribute to component assessments • Cross-validate across environments