testing in a continuous delivery environment
DESCRIPTION
Talk at Iqnite conference for software quality and testing about how we have do continuous delivery at here.com Together with Siddharth Somasundaram http://www.iqnite-conferences.com/suisse/programme/programme.aspxTRANSCRIPT
![Page 1: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/1.jpg)
Testing in a Continuous Delivery
EnvironmentSiddharth Somasundaram
Stefan Verhoeff
![Page 2: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/2.jpg)
HERE
● Formerly known as Nokia / NAVTEQ Maps
● Powers Bing, Facebook and Yahoo Maps
● 4 out of every 5 cars with in-dash navigation are
powered by HERE
![Page 4: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/4.jpg)
Testing Before Agile
● Testers are only involved in the later stage
● Long regression testing cycle before release
● Developers and Testers have separate work streams
○ Leads to conflicts between them
● Business pressure to release the software
○ Compromise on quality?
![Page 5: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/5.jpg)
Importance of Agile
● Requirements change over time
● Testers are part of the planning process
● The entire team is responsible for Quality
● Faster release cycle
○ From once every 3 months to daily releases
![Page 6: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/6.jpg)
Continuous Delivery
● Release early and often to production
● Automate testing and deployment
● Faster feedback, reduce risks and overhead for releasing
● Hot trend in the industry, especially web companies
![Page 7: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/7.jpg)
But what about Quality Assurance?
● Lots of automation for regression testing
● Build enough trust in the test coverage
● Focus shifts from prevention of bugs to rapid discovery
and resolution
● Exploratory testing can be decoupled from releasing
![Page 8: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/8.jpg)
How do we do it?● Team Culture and Technical practices we found to be useful
![Page 9: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/9.jpg)
Team Culture Practices
● One agile Team, One Product and One Goal
○ Product Owner
○ Scrum Master
○ Developers / Testers
○ Offshore teams
● Communication solves most of the problems
![Page 10: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/10.jpg)
Collaborate using Work Agreements
The team formulates rules and hold each other
accountable. For example:
● Running tests before committing
● No commits on red build
● First priority is to keep the build green
![Page 11: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/11.jpg)
Build Monitor visible in every room
![Page 12: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/12.jpg)
Pair Programming
![Page 13: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/13.jpg)
Share knowledge with Tech Talks
![Page 14: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/14.jpg)
Combine developer and operation teams
![Page 15: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/15.jpg)
Technical Practices
![Page 16: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/16.jpg)
Continuous Integration
![Page 17: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/17.jpg)
Test Driven Development● Code and tests are developed at the same time
![Page 18: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/18.jpg)
Build Pipeline
●
![Page 19: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/19.jpg)
Test Coverage at all levels
© Alister Scott http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/
![Page 20: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/20.jpg)
Eliminate Flaky Tests
● Essential to keep the team
trusting the test results
● Mocking dependencies
![Page 21: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/21.jpg)
Feature Flags
● No branching
○ Prevent integration problems
○ Quality built in
● Turn features on in production
when they are ready
● Quality boundaries for
different release stages
![Page 22: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/22.jpg)
Logging & Monitoring
● What happens after we go live?
● Production monitoring ensures edge cases are covered
● Real-time metrics for: system, application, business
![Page 23: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/23.jpg)
Logging & Monitoring
● Quickly responding to alerts
● Production is the ultimate proof the code works
Bug introduced We fixed it :-)
![Page 24: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/24.jpg)
In Conclusion...
![Page 25: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/25.jpg)
Is it Successful? The Metrics
2010 Now
Release quarterly Release daily
Mostly manual testing 7000+ unit tests, 400+ GUI tests
Preparing release takes weeks
Release package ready in 45 minutes
Deployment takes days Deploy 3 sites in 10 minutes
Management signoff required
Team empowered to make changes
![Page 26: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/26.jpg)
Deployments 2012 vs 2013
![Page 27: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/27.jpg)
Business Benefits
● Reduce risk involved in big changes
● Reduce time to market
● Smaller more frequent feature rollouts
● Quickly learn from feedback on product changes
![Page 28: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/28.jpg)
Business Benefits
● Predictable releases
● Improved team morale
○ Less time spend in stressful firefighting
○ Less bug fixing, more feature delivery
○ Satisfaction of delivering to the customer
![Page 29: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/29.jpg)
Challenges
● Everybody agreeing on the process
● Discipline comes over time only
● Feedback needs to be given quickly
● High investment in test automation and keeping it
running smoothly
● Need to react to incidents quickly
![Page 30: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/30.jpg)
Future Plans
● Deploy every build to production
● Keep on investing in Automation and Monitoring
● Canary deployments to validate new releases before
global rollout
● Leverage quick cycle time for A/B testing driven feature
development
![Page 31: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/31.jpg)
HERE ChecklistTeam Culture
● One Team
● Work Agreements
● Build Monitor in the
room
● Pair Programming
● Tech Talks
● DevOps
Technical
● Continuous Integration
● Test Driven Development
● Build Pipeline
● Test Coverage on all
levels
● Feature Flags
● Logging & Monitoring
![Page 32: Testing in a continuous delivery environment](https://reader034.vdocuments.us/reader034/viewer/2022051209/5485052eb47959d80c8b4d00/html5/thumbnails/32.jpg)
Merci BeaucoupThank You
Danke SchönGrazie