continuous testing vs. test automation: three key differences · continuous testing analysts align...
TRANSCRIPT
W9 Test Automation Wednesday, May 2nd, 2018 1:45 PM
Continuous Testing vs. Test Automation: Three Key Differences
Presented by:
Wayne Ariola
Tricentis
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-- -268- - -8770 ·· 904- --278-- -0524 - [email protected] - http://www.stareast.techwell.com/
Wayne Ariola Tricentis Known for his efforts to promote continuous testing, Wayne Ariola has contributed to the design of many innovative technologies and received several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, software quality governance, and application security, Wayne is a frequent contributor to industry publications and author of Continuous Testing for IT Leaders. He has been a contributor to software testing for fifteen years and in the software industry for more than twenty.
4/25/18
1
© 2016 by © 2018 by
Wayne Ariola
Continuous Testing v. Test Automation
© 2016 by 2
4/25/18
2
© 2016 by
Testing at the Speed of DevOps
What has Changed? 1
2
3
Agenda
The three differences with Continuous Testing
© 2016 by 4
How would you rate the success of test automation…
a success story
so so
a failure
?
4/25/18
3
© 2016 by
Rate the Success of Test Automation…
0
10
20
30
40
50
60
Q3-‐2016 Q1-‐2017 Q3-‐2017 Q1-‐2018
Success So So Failure
© 2016 by
Test automation has not kept its promises.
4/25/18
4
© 2016 by
80% Manual Testing
20% Automated Testing
UI
API
Today
The reliance on manual testing is the
top technical challenge in app development.
World Quality Report, 2017
Test automation requires developers.
© 2016 by
We didn’t do anything wrong, but somehow, we lost.
‹‹ Stephen Elop, CEO Nokia, 2013 ››
4/25/18
5
© 2016 by
What has Changed?
§ People § Process § Technology § Tools
© 2016 by
Offshore Labor
38 kUSD/yr
60%
Traditional Onshore Labor
100 kUSD/yr
Save cost through
labor arbitrage
Satisfied?
People
4/25/18
6
© 2016 by
Process Test as a time-boxed event
Fixed Fixed
Plan Develop Build Test Release Plan Develop
Build Test
Release
Build Test
Build Test
Build Test
Build Test
Build Test
Build Test
Build Test
Test is Continuous
© 2016 by
Technology and Disruption
• Security • Safety CriDcal • Financial • FDA • AutomoDve • Aerospace
Compliance
• DevOps • Agile • Lean • Bi-‐Modal
IniDaDves
• Microservices • Containers • Cloud • IoT
Technology
• ProducDon Outage • ApplicaDon Slow • User Experience • ConnecDvity
Failure
• SaaS • Mobile • APIs • IoT
Switching Cost
4/25/18
7
© 2016 by
Top Tier Vendors
IBM Punts Development to HCL http://www.theregister.co.uk/2016/10/03/ibm_hcl_software_outsourcing_deal/
HPE Strategy Clear As Mud http://it-tna.com/2016/09/28/hpe-strategy-clear-as-mud/
Tricen@s Delivering Con@nuous Tes@ng at the Speed of DevOps
© 2016 by
Continuous Testing
Analysts Align
“Continuous Testing is required for extreme automation”
voke
“Continuous Delivery Needs Continuous Testing”
Forrester
“DevOps initiatives will cause enterprises to implement continuous testing”
Gartner
”Continuous Testing has obvious application in agile environments.”
Bloor
4/25/18
8
© 2016 by
The Perfect Storm – Digital Disruption
Mitigate Business Risk
ConDnuous TesDng • Model Based Test AutomaDon • Risk Based TesDng • Exploratory TesDng • Mobile TesDng
• Security • Safety CriDcal • FDA • AutomoDve • Aerospace
Compliance • DevOps • Agile • Lean • Bi-‐Modal
IniDaDves • Microservices • Containers • Cloud • IoT
Technology • ProducDon Outage • ApplicaDon Slow • User Experience • ConnecDvity
Failure • SaaS • Mobile • APIs • IoT
Switching Cost
• BI/DW TesDng • API TesDng • Test Data Management • Service VirtualizaDon
© 2016 by © 2018 by
Wayne Ariola
3 Key Differences Continuous Testing v. Test Automation
4/25/18
9
© 2016 by
Continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.
What is Continuous Testing
© 2016 by
Continuous Testing – What’s Different?
What’s Different?
RISK A feedback system for beXer trade-‐off decisions between release scope, Dme and quality
BREADTH The boundaries of an applicaDon are eroding. Advanced automaDon is required for end-‐2-‐end tesDng
TIME Its speed because Dme-‐to-‐market maXers not because development can release faster
4/25/18
10
© 2016 by
Survey on Risk 2017 Survey on the Future Role of the Tester hXps://www.surveymonkey.com/r/SWB7F3Q
© 2016 by
Does the release candidate have an acceptable level of risk?
4/25/18
11
© 2016 by
How to Achieve Quality @ Speed
§ Business risks drive quality activities
§ Teams collaborate on risk definition
§ Risk definition drives test case design
§ Execution prioritized per risk definition
§ Acceptance criteria visible and measured
§ Results of quality activities are visible and
translated for all levels of business
© 2016 by Ever-Growing Challenge
Technical Complexity & Rate of Change
Years Months
Months Weeks
Weeks Days
Days Hours
Hours Minutes
Delivery Cycle Time
Perceived Disruption
Today
Introduce Continuous Testing to meet demand
4/25/18
12
© 2016 by 23
18% (official) overall test automation
degree
8% (official) enterprise
test automation degree
Really ? X ?
© 2016 by
Continuous Testing requires Automation
24
80% Manual Testing
20% Automated Testing
UI
API
Today Future
Exploratory Tests
Automated UI Tests
API Tests
Test Driven Service Virtualization
+85%
- key enabler for high automation rates -
4/25/18
13
© 2016 by © 2018 by
Wayne Ariola
Key Testing Practices for Continuous Testing
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation
Active Test Data Management Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
4/25/18
14
© 2016 by
Introduce Session Based Exploratory Testing
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
All testing starts with a Risk Based Prioritization
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
4/25/18
15
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Optimize your risk coverage: Test Case Design
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
4/25/18
16
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation
Prevent false+: Active Test Data Management Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation
Active Test Data Management Provide stable infrastructure: Test Driven Service Virtualization
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
4/25/18
17
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation
Active Test Data Management Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization Test Case Design
UI- and API based test automation
Active Test Data Management Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
4/25/18
18
© 2016 by
Key Points of Continuous Testing hXps://www.tricenDs.com/what-‐is-‐conDnuous-‐tesDng/
• Primary goal is assessing business risk coverage
• Establishes a safety net that helps the team protect the user
• Always on, stable test environment
• Seamlessly integrated into the so`ware delivery pipeline and DevOps toolchain
• Delivers acDonable feedback appropriate for each stage of the delivery pipeline
© 2016 by
Q&A § Continuous Testing vs Test Automation Paper
https://www.tricentis.com/resource-assets/continuous-testing-vs-test-automation/
§ What is Continuous Testing (w/ 14 Key Points) https://www.tricentis.com/what-is-continuous-testing/
§ Survey – The Role of "Tester" in the New Digital Age https://www.surveymonkey.com/r/SWB7F3Q