quality initiatives in the devops tool chain...quality initiatives in the devops tool chain jan...
TRANSCRIPT
Quality Initiatives in the DevOps Tool Chain
Jan Sahlström, Betsson Technologies@testsson_se
A QA’s Role in a DevOps World
Nordic Testing Days 2018Tallinn, Estonia
Photo: Jan Sahlström, 2017
Source: Wikipedia
“A term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment, where building, testing, and releasing software can happen rapidly, frequently, and more reliably.”
DevOps“A term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment, where building, testing, and releasing software can happen rapidly, frequently, and more reliably.”
DevOps is a culture, not (only) a role
• There is no clear definition• Developed by developers and ops, no testers involved• ”Everything can be automated”
Are there any problems with DevOps?
DevOps ToolchainTest
here!
DevOps Toolchain Test here!
DevOps Toolchain
Test here!
Test here!
Test here!
Test here!
Test here! Test
here!
Test here!
Inspired by: @DanAshby04
DevOps Toolchain
1. Understand the activity in each stage
CreateCreate is composed of the building, coding, and configuring of the software development . The process specific activities are:• Design of the software and configuration• Coding including code quality and performance• Software build and build performance• Release candidate
a.k.a: Coding
DevOps Toolchain
1. Understand the activity in each stage
2. Identify quality initiatives
Create | QA initiativesWhat Comment How
TDD Help drive the testing and understanding of the feature Sapient
Unit Test Verify the functionality of a specific section of code Machine
Integration Test Verify the interfaces between components Machine
Component interfaceTest
Check the handling of data passed between various units Machine
Code Quality Configured and measured using tools Machine
Code Review Any form of four eye principle Sapient
Code Coverage Where are the weak spots? Machine
DevOps Toolchain
RRepeat for each section
R
R R
R
R
R
Continuous Delivery Pipeline
Define where the variousinitiatives fit and what shouldhappen if a verification step fails
Some initiatives
Adapting to the increasing speed of deliveries and ways of working
Everyone have a test environment.Some are lucky enough to have another environment for their production code.
Environments
LOCAL TEST QA PRODUCTION
Unit Test Smoke Test System Test Load Test(!)
Integration Test RST UAT
Acceptance Test Regression Test Performance Test
4 Eye Principle System Test
Dev PC
MONITORING
PreProd
MONITORINGMONITORING
PROD
Frequent -Guarded
Highly Controlled Deployment
Controlled Deployment
LOCAL TEST QA PROD
Environments
LOCAL TEST QA PRODUCTION
Unit Test Smoke Test System Test Load Test(!)
Integration Test RST UAT
Acceptance Test Regression Test Performance Test
4 Eye Principle System Test
Dev PC
MONITORING
PreProd
MONITORINGMONITORING
PROD
3rd PartyTest Doubles
3rd PartyLive Environment
3rd PartyTest Environment
LOCAL TEST QA PROD
Test Doubles: Fake, Mock, Stub
Linting
• Linting - Running a program that will analyze code for potential errors• Start small with a just a few rules and expand as you go
Unit Test / Integration Test• Run on LOCAL and TEST• Code Coverage – %?• Code Coverage – Shouldn’t deteriorate• Should be fast, if not try to parallelize
Build Test Set 01
Test Set 02
Test Set 03
Build Test Set 01 Test Set 02 Test Set 03
Test Data – Setup and Teardown
Why?End and start tests in a know state will avoid tests failing due to other breaking tests
Initial State
Setup
Execute
Teardown
Return to Initial State
Branches – When to test
A #1
Master #01 Master #02Master #00
Test Deep
Branches – When to test
A #1
Master #01 Master #02Master #00
B #1
Master #03
Test Deep
Test Deep
Tests done in the branchmight not be valid after B #1
Tests done in Mastermight be too late dueto breaking changes
Branches – When to test
A #1
Master #01 Master #02Master #00
A #1
Master #03
1. Test Deep
2. Test Shallowi.e. Smoke Test
Middle of the road approach
Branches – When to test
A #1
Master #01 Master #02Master #00
A #1
Master #03
Master should always be deployableJez Humble, DevOps Research and Assessment LLC
Master should never contain known errorsJakob Alander, Spotify
On test automation (checks)• Writing automated checks is bug prevention, if it’s done by a
developer. I.e. bugs are found while writing the tests.• Let new tests qualify for the CI monitoring (Non-deterministic tests)
Code Review• Review all code, including tests• Don’t just read the code. Run the code and the applicable tests during
the review• Testers can join in on code reviews even if they cannot code
Canarying / Canary Deploy• Deploy / switch on and verify one instance / section at a time• Start with the “cheapest” one…• Monitor closely
Empty board right-to-left i.e. empty Test column before starting a new feature
The Test columnWho owns it?
ToDo In Progress Test Done
!
Join up with someone in the team and do joint test sessionGather the team in front of a big monitor and do a mob session
Pair/Mob Testing
Image source: Taipei Times Dec 24, 2014. AFP / KCNA via KNS
Make sure that there is a DOD before starting on a new feature
Done DoneOr almost done?
In Progress Test Almost Done... Done
Monitor the builds
• The state of the build should be everyone’s primary interest• Must always be green – Avoid the broken window syndrome• Display on a dedicated monitor• All green != no defects…
Monitor your systems
• Look into different strategies for monitoring (RED, USE, Other)• Use the same monitoring cross the different environments• Share usefull monitoring
Monitoring vs. Testing
Testing Monitoring
Cove
rage
Increased risk of defects in PROD →
Monitoring vs. Testing. Why not? If:
ü Possibility to perform A-B testing, Canary Release or equivalentü Hidden PROD environment (AWS Green & Blue)ü Support & Tech organization available to act quicklyü Extremely loyal users
DevOps Toolchain
1. Understand the activity in each stage
2. Identify quality initiatives
3. Prioritize and implement
TeQuILA
Bringing it all together
Modeling
Level 1•Set up test coverage bundles•Set up a continuous build•C lassify your tests as Sm all, M edium and Large•Identify nondeterm inistic tests•C reate a sm oke test suite
Level 2•N o re leases w ith red tests•R equire a sm oke test suite to pass before a subm it•Increm ental coverage by a ll tests >=50%•Increm ental coverage by sm all tests >=10%•At least one feature tested by an integration test
Level 3•R equire tests for a ll nontriv ia l changes•Increm ental coverage by sm all tests >=50%•N ew significant features are tested by integration testsLevel 4•Autom ate running of sm oke tests before subm itting new code.•Sm oke tests should take less than 30 m inutes to run•N o nondeterm inistic tests•Total test coverage should be at least 40%•Test coverage for sm all tests a lone should be at least 25%•A ll s ignificant features are tested by integration tests
Level 5•Add a test for each nontriv ia l bug fix•Actively use available analysis tools•Total test coverage should be at least 60%•Test coverage for sm all tests a lone should be at least 40%
Google Test CertifiedA 5 level model developed at Google for enhancing quality
Modeling
Google Test CertifiedA 5 level model developed at Google for enhancing quality
Level 1•Set up test coverage bundles•Set up a continuous build•C lassify your tests as Sm all, M edium and Large•Identify nondeterm inistic tests•C reate a sm oke test suite
Level 2•N o re leases w ith red tests•R equire a sm oke test suite to pass before a subm it•Increm ental coverage by a ll tests >=50%•Increm ental coverage by sm all tests >=10%•At least one feature tested by an integration test
Level 3•R equire tests for a ll nontriv ia l changes•Increm ental coverage by sm all tests >=50%•N ew significant features are tested by integration testsLevel 4•Autom ate running of sm oke tests before subm itting new code.•Sm oke tests should take less than 30 m inutes to run•N o nondeterm inistic tests•Total test coverage should be at least 40%•Test coverage for sm all tests a lone should be at least 25%•A ll s ignificant features are tested by integration tests
Level 5•Add a test for each nontriv ia l bug fix•Actively use available analysis tools•Total test coverage should be at least 60%•Test coverage for sm all tests a lone should be at least 40%
Personalize the modelA dded•D efine testable requirem ents for a ll new stories•G uarded com m its•Four eyes princip le•O w n m etrics•...
DependenciesSort out the internal dependencies for the identified initiatives
1 2 3 4
...
Each team need a person within the team to drive the inititiative. This can be any team member.
Driver
Implementation
Get started
Set up workshops and go through each step deciding on how to implement them
Team effort!
None of this will be possible without the participation of the whole team
Working software over new functionallity
TeamArchitect
QADeveloper
SEAL | All roles somewhat overlappingSoftware Engineering ALchemist
This is where the magic happens
DevOps Culture
Malum est consilium, quod mutari non potest
It’s a bad plan if it admits no changes
Publilius Syrus (85-43 BC)
Caveat
@testsson_seFam ilyG uyYourself
Thank you!
Temptare necesse est, navigare non est necesse
Q&A
Inspirations and sources
• World Quality Report 17/18 9th edition
• How Google Tests Software by James A. Whittaker, Jason Arbon, Jeff Carollo
• Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble
• A Practical Guide to Testing in DevOps by Katrina Clokie
• Blog post "Continuous Testing in DevOps…“ by Dan Ashby
• Anna Sundin Almén, AddQ
• Jakob Alander, Spotify
• Henrik Andersson, Betsson Technologies AB
• Sigurdur Birgisson, Avensia
• Richard Bradshaw, Ministry of Test
• Angie Jones, Twitter
Publications Talks & Discussions
Beside all bits and pices picked up on conferences, meetups, tweets and various other places (too many and too scattered to list) that has gone into this presentation there are some that stands out.