continuous testing: a key to devops success
TRANSCRIPT
Continuous Testing: A Key to DevOps Success
Sujay Honnamane
Test Director
Cognizant Technology Solutions
1
© 2015 Cognizant
2
Continuous Delivery is critical for an enterprise
Dev & Test Teams
Operation Teams Gap
Evolving Customer
Expectations Faster Time to Market
Need for higher quality
Changing Business Requirements
Business Agility, performance, Availability and Stability
Leads to Continuous Delivery Challenges…
Lack of collaboration
Infrequent, manual, error prone
deployments Long QA feedback Loop
Mobile Cloud Social Big Data Connected
Systems
Emerging Technologies
DevOps Deploy
Test Continuous Delivery with DevOps
Let’s face it: • Applications that companies deal with
actually contain very complicated business logic implementations than what Facebooks, Twitters and Netflix/es are
• Years of legacy – these applications are out there for more than decade!
• There are patterns adopted by Development, Operations and QA and organization’s expectations revolve around it!
• Agile revolutionized development to the extent – Operations remained in silo and QA remained bottleneck
Image source : https://paimeiitguy.wordpress.com/2008/07/28/a-beacon-of-hope-for-the-keep-it-simple-stupid-way-of-working/
Reality Check – Simplicity is highly overrated by Unicorns!
3
• Traditional Ops – putting out fires, deal with recurring failures & urgent fixes • Test Environment overhead– Each environment is unique by zillion different ways • Several thousands of interdependent regression cases running through UI layer for
several days • Longer test cycles & manual handoffs – much slower feedback • Loosely defined test data management strategy • Downstream/Upstream dependencies • Silos within silos inhibiting cross functional leverage
Image source: http://legacyportfolio.co.uk/blog/2012/06/08/what-are-the-challenges-facing-corporate-occupiers-today/
Challenges for horses today
4
Spot the opportunities & Correct them – One at a time
5
OPS
What in the world?
Dev QA Product owner
• Build failure rate > 60% • OPS : DEV problem , DEV: OPS problem • Avg. time for build deployment > 2 days
Our teams..Chaos – First DevOps Meeting
6
There is a better way! And QA can help!!
Oh Really?
Who cares? Deployment is painful task, these folks won’t ever get it!
QA Architect
QA’s first attempt to break the silos
OPS
Dev
Product owner
7
Someone is making sense!!
No kidding! We work hard to make everyone’s life easy
Product owner
Hmm! We are listening!
A start..constructive discussion – QA gets Dev in alignment
OPS
Dev
8
• QA has greater insights to both Development and Operations side of the worlds
• Deployment delays hurts QA more than any one
• Continuous Integration is first step in right direction – QA and Dev collaboration can make it happen
Can QA be the DevOps enabler? Yes, we can !
QA
9
DevQA
With group of motivated individuals from both QA and Dev
Goal Statement: “Predictable builds, deployments and quality assurance procedure that delivers the working software from developer’s IDE to the hands of business users with click of button.”
And, DevQA journey began!
Image source: http://encontrosdesign.info/how-team-building-exercises-can-help-your-business/
10
WE, the people: Collaborative – Working together to build innovative solutions Open minded - Experiment,fail,learn and Adapt Tool agnostic – solution driven than driven by tool Fearless – Makes problem visible One Team spirit – Combined ownership of problems, no blame Love coding – ruby, python, ant whatever comes our way
DevQA team’s attitude
Image source : http://voices.whitman.syr.edu/student-life/undergraduate/teamwork-makes-the-dream-work
11
SVN Server
Automation Team
Jenkins Server
Slave Agents
Dev / QA Environments
Stakeholders
Continuous Integration – Dev / QA Environments
Updated Automation
Script
Triggers Build Verification Tests Health Checks Smoke Tests Intelligent regression
Re
sult
s
Code Deployments in Dev / QA environments
Accessing Environments
Build verification Results to stakeholders through email
Dev code: New User Stories /
Defect Fixes – Code Commit
Polling the Updated Code
Updated Automation
Script
Dev Team
Automated Build Creation
Automated Security Scan
Unit Test
Smoke Results
Critical Scenario Results
Regression Results
Static Code analysis
Build Quality In
Code Coverage analysis
Instant feedback on
quality
Virtualized services
Production data subset
12
© 2014 Cognizant
Continuous Integration - Critical Success Factors
Inte
gra
te,
Bu
ild
,D
ep
loy
an
d V
ali
da
te
1 2 3 Continuous Testing
4 Service Virtualization 5 Environments on Demand
“Shift Left” QA and Dev synchronization
13
13
Existing full regression running across three different tools took us 50hrs to complete, we needed to build something intelligent that validates relevant stuff for in shortest possible time! Rest of the regression runs over the weekend!
It addresses most common challenges QA organization face today: Legacy automation built on commercial tools over the period of time runs
slow Often, automation is built over altogether in different tools to cover for UI,
APIs, Mobile coverage etc. Business influence over regression definition often push testing team to
build long running tests
Intelligent Regression Model – The key success factor
14
Intelligent regression model – Key considerations
Image source: http://www.signaturelandscaping.ca/
• Dynamic regression scope for each build based on type of code change - Automatically generated release notes feeds in to regression job
• Test cases are traceable to the code files • Intelligent framework capable of turning test scenario
ON/OFF based on application change • Risk based approach with close to zero risk • Centralized repository of code-to- test case traceability
15
DevOps Transformation – QA Levers
Automated Test design through intelligent model based framework Faster feedback through
• Well defined heartbeat tests & deployment verification test • Massive parallelization of tests powered by efficient load
balancing of tests between nodes • Shift from UI based tests to Non-UI based tests • Intelligent test suite
Reduced testing dependency
• Service virtualization • Adoption to more isolated tests over End to end tests • Test Data Management
16
Conclusion: This is Dev Ops !
17
THANK YOU
18