the evolution of test automation for devops
TRANSCRIPT
Clint SprauveMalcolm Isaacs
The Evolution of Test Automation for Dev/Ops
Quality driving value through DevOps
The “why”--How DevOps presents a great opportunity for quality professionals to expand their organizational impact The “what”--Specific new processes and skills quality professionals bring to DevOps and Shift Left models
The “how”-- How quality professionals and teams are adopting DevOps practices and capabilities
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3
Quality matters to DevOps
Why?
Evolution of Functional Testing
Agile and DevOps challenge testing processes
Current challenges…
Open Source and Continuous Integration Tools
User Experience changesthe definition of quality
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
93%of IT leaders use Agile
for software development projects
$60B
annual savings by businesses using open source
Global population estimated to be connected by 2019*
autonomous things estimated to be attached to the Internet by 2020*
57%
30B
Database activity
Security validation
Inventory query
Database activity
3rd party ship
service
Logistical services
Acc’t info
3rd party database
3rd party reporting
Security validation
3rd party payment services
Account authorization
Shopping app(web or mobile)
Purchase
receivedAdd to cart
Shipping detail
Delivery tracking
Customer login Checkou
t
A Retailer Real-Life ScenarioA Modern, Complex Architecture
Geographies of users?
Flows?
User types?(mobile, web, other)
Services availability and cost?
Single vs. Multiple?
Type of network connections?
Environment availability?
Is it secure?
Does it function?
Functional
Development
USE CASE:
Performance
DevOps
Security
Mobile
QUIT SCENARIO
DevOps is a transformationIt’s a sustained effort to achieve speed with Quality
• With DevOps there is no one true authority
• It’s not a product, spec, job title
• There is no one “right way”
• But one thing is certain- Quality matters
The role and importance of Quality in today’s AD teams is moving from finding
defects to protecting the corporate image World Quality Report– 2015-16
People
Process Technology
The good news– Quality is centralDispel the myths that Quality goes away with DevOps
Dev Ops
Quality
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Quality in the age of DevOps
What
Quality Within DevOpsDefining The What
1. Starts with Culture Change2. Quality Integrated Throughout3. Test Constantly4. Relentlessly Automate5. Create Visibility
Starts with Culture ChangeQuality within DevOps
1. Starts with Culture Change. Teams unite around making the org as a whole faster, more competitive.
“Coming together is a beginning; keeping together is progress; working together is success” Henry Ford
Quality Integrated ThroughoutQuality within DevOps
2. Quality Integrated Throughout. Move everything to happen earlier. Testing happens earlier, get comfortable with IDEs, leverage development assets, but add your testing acumen.
Test ConstantlyQuality within DevOps
3. Test Constantly. If you’ve followed point 2 above, you’re testing earlier. You’re also going to test much more often. And consider the Quality Trifecta—ensure Functionality, Performance, Security.
Relentlessly AutomateQuality within DevOps
4. Relentlessly Automate. Automated testing shortens time to market and lowers risk of human error in testing, while increasing the cycles in same amount of time, and increasing coverage. Automate testing, provisioning and virtualize. Then tie it to build life and automated execution integration.
Create VisibilityQuality within DevOps
5. Create Visibility. Seamless communication between people, tools and processes. Removing constraints that slow you down and connecting tools through a framework.
FrameworkQuality within DevOps
AutomateOrchestrate TransformApplication
DeliveryManagement
IT OperationsManagement
Application testing for desktop, web
and mobile
Infrastructure, domain and compliance automation
From silos to integrated lifecycle management
Processes to improve agility
Customer experience by simulating real world
scenarios
Customer experience by delivering products and
services
DevOpsShared Dev & Ops
methodology across Dev/Test/Stage/Production
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
In Practice
How
How we have delivered at HPE SoftwareQuality within DevOps
Starts with Culture ChangeQuality within DevOps
• Why we thought to change = Productivity
• Development environment based on shared infrastructure • Unstable• Can be easily damaged• Common for everyone• Operational cost is very high• Unexpected behavior
• Work from anywhere• Latency is killing your connections
Quality Integrated ThroughoutQuality within DevOps
Plan Define Develop Build Test Deploy Release Operate
AGM AGM AGM
Sonar
OO
SM/SAW
Puppet
UCMDB
BSMFortify
Jenkins
Wiki
OpenGrok
GIT/Stash
IntelliJ Maven
ALM OO
Puppet
SM/SAW
Nexus
LeanFT
LR SiS
CSA
Nexus Nexus
Maven Site
Vagrant
Sahara
3RD P
artie
sPr
oces
sH P
CSA
Docker
JUnit
JBehave
Jasmine
Selenium
SharePoint
NV and SV
Organization Example - Evolve the Team StructureChange based on product complexity and team skill set
Dev/Test Organization Additional Responsibilities QA & DEV – one team Support Tickets
Automation from day 1 Forums & social networsk
Manual Testing is Essential Build CI/CD infrastructure
No more deliveries to QA Train internal teams on dev/ops practices
Structure and responsibilities will vary from team to team, project to project
Do not implement a “cookie cutter” approach to quality in Dev/Ops
Test ConstantlyQuality within DevOps
Tests run on each push13,000 automated functional tests run daily
Service Anywhere 3.3 Service Anywhere 3.4336 application tests 624 application tests
40 platform tests 70 platform tests
Manual coverage 100% sanity coverage
Can run only on CI deployment Runs on real deployment too
Relentlessly AutomateQuality within DevOps
• End2End automated flow for continuous testing
• Jenkins acts as the orchestrator
• Flexible & customizable to accommodate specific customer workflows
DEV
QA
STAGE
PROD
Build
Deploy
Test
Jenkins
HP CodarALM
Test case1
Test case2Test case
3....
Test case n
Example – Dev/Ops Automation Test Coverage DistributionTEST TYPE Client Backend Protocols/Regions
Coverage 60% 80% 100%
# of Active Tests
133 306 All Protocols, All regions
Create VisibilityQuality within DevOps
• Accurately Predict if a Code Commit Will Break a Build
• Accurately Predict and Plan for Defect Introduction
Key Lessons LearnedCase Study Results
• DevOps is possible. Huge potential for speed and quality
• The culture changes when you incorporate DevOps
• Discipline – DevOps requires focus and dedication
• Automation and tools are important, but not everything
• Communication, collaboration and trust
More info
– Trials and free editions of tools you can use– Trials
–NV Trials: http://www8.hp.com/us/en/software-solutions/network-virtualization/try-now.html–LeanFT: http://saas.hp.com/signup/try/leanFT
– Community Editions–LR 12.50 with NV: http://saas.hp.com/signup/try/LoadRunner
– Freemium(s)–NV Freemium: http://www.hp.com/go/NVFreemium
Thank you
27