Download - NYC MeetUp 10.9
Automated Tes,ng, Con,nuous Integra,on & Con,nuous Deployment
William Josephson Co-‐Founder, Solano Labs @SolanoLabs Automated Tes,ng NYC
Some Metrics on how Fast we have Sped Up our Customers:
• 18 hrs serial, 2 + hrs on Parallel Team City w/ local HW to 13 mins • 30 min on Jenkins to 6 mins • 5 hrs serial PyPy test suite to 15 mins • 5 hour queuing to 5 min max wait time
Solano Labs – Lets you Test More, Faster
2 Solano Labs © 2013 Confidential
Some Facts & Figures*
High performing organizations:
• Deploy code 30 times more frequently.
– 8000 times faster than their peers
– Deploy multiple times a day, versus an average of once a month.
• Have 50 percent fewer failures.
• Restore service 12 times faster than their peers.
– Fewer failures and faster recovery mean less risk to the business when
changes are deployed.
*Source: Puppet Labs 2013 State of DevOps Report
How are they doing this?
I don’t have all the answers…
But here’s one: Automa,on
Automated Testing Does More Than Find Bugs Its Helps Teams Build a Better Product
Bug are Common, Difficult to Eliminate & Very Costly
Automated Testing: Using software to test software
• Reduces cost of bugs by reducing ,me between introduc,on and discovery
• Frees engineers to make large changes fearlessly, reducing cycle ,me
• Reduces complexity and risk at feature integra,on
Automation: Reduces the Cognitive Feedback Loop
Automa,on shortens the cogni,ve loop for engineers • Automa,on delivers results for a build quickly • Shorter turn-‐around ,mes change how
engineers work • Test-‐Driven Development (TDD) is one logical
extreme
Waiting for tests to run is no fun!
Incorporate Feedback
Run Tests
Write Code
Get Results
What Makes for a Good Automated Test?
• Eliminate repetitive manual tests
• Yield reproducible results
• Cover common and edge/error cases
• Ideally either pure unit or integration test
• Strive for clarity even when code is gnarly
• Test for business requirements
Why Not Automate Everything?
• Writing tests does have a cost
– May not be appropriate for experimental code
– May be fragile – UI/mobile automated testing is evolving
– Difficult or impossible to express design and human factors
• Thorough testing requires commitment and discipline
– Writing tests is a short-term pain for a specific individual
– Test suite as a whole is valuable to the team over time
What do you do with all those automated tests?
Continuous Integration: A Discipline for Testing Every Change
I Strong Isola,on
Detailed Instrumenta,on
Consistent Run,me Environment
Reproducible Results
Components that Make for Effective CI:
Typical CI Setup:
Continuous Deployment: Automatically Release Validated Changes
• Goal: make release a non-event – Popularized by Web 2.0, but much more widely applicable – Final decision to deploy may still be manual
• Valuable to business and engineering sides of the house: – Selectively enabled features – feature flags – Consistent, widespread measurement
• Key Ingredients: – Small changes, continuously integrated – Automated infrastructure (DevOps) – Staging/test environment accurately models production
Further Reading
• Martin Fowler from ThoughtWorks: martinfowler.com
• Book: Continuous Delivery: Reliable Software (Jez Humble)
• IMVU: http://timothyfitz.com/2009/02/10/continuous-deployment-at-
imvu-doing-the-impossible-fifty-times-a-day/
• Etsy Blog: codeascraft.com
• Github: teach.github.com/articles/lesson-continuous-delivery/
• http://info.puppetlabs.com/2013-state-of-devops-report.html
• http://engineering.quora.com/Continuous-Deployment-at-Quora