continuous testing in practice - amazon web services · 40% manual testers. the continuous testing...
TRANSCRIPT
Continuous Testing In Practice
Feb 2019
Speaker
2
Guy Arieli
CTO
Agenda
3
What Is Continuous Testing?01 10 min
Demo03 20 min
02 Case Study 10 min
Q&A04 10 min
What is Continuous Testing?
Executing automated tests as part of the Continuous Integration pipeline for immediate feedback on the risks of a release candidate
4 Developer A
master branch
CI Server
Build
Test
Evaluate
Feature branch
Co
mm
it
Pu
ll
req
ue
st
Developer B
Feature branch
Co
mm
it
Pu
ll
req
ue
st
Agenda
5
What Is Continuous Testing?01 10 min
Demo03 20 min
02 Case Study 10 min
Q&A04 10 min
6
Leading North American transportation company
Case study background
• 3 apps
• R&D organization:
~ 200 developers in the US
~100 QA - offshore in India with SI
• Use Appium + Selenium
• Process started 2 years ago
Background
• Test automation does not identify many bugs
• Automation stability issues
• QA Environment issues – availability, low use case coverage
• Disconnect with R&D on process
• Feedback loop is 1-2 weeks
• Automation has low business impact
Challenges
Slow R&D process, low quality, negative business impact
➔Speed up R&D progress
➔Improve quality
➔Shorten time to market
Objectives
7
Organizational structure and communication flow
8
Siloed structure DevOps structure
OpsDev Engineer TesterOpsDev Engineer
Tester
OpsDev Engineer
Tester
OpsDev Engineer
Tester
9
Case study: North American financial institute
Drive for more automation requires workforce transformation
Performance & Volume
5%
Environment and Data10%
25%
, 60%
Automation Testers & Software engineers in test
Exploratory testers
Performance & Volume10%
Environment and Data20%
Automation Testers30%
Manual testers40%
The Continuous testing dev-test process
Epic Story thinning Acceptance
Criteria (user story scenarios)
Acceptance Criteria
(feature files)
Code incl. White box Test
(step definitions)Build & Deploy
Automated testing
Acceptance testing
Showcase / Demo Sign-off
Dev Engineer
Dev & Test EngineerDevOps EngineerDev & Test Engineer
Functional Spec
Product Owner
Analyst Product Owner
Analyst Owner / analysts
Tester
TesterProduct Owner
CI/CD cycle
Dev Engineer
Tester
Development Engineer
Fast feedback with continuous testing
Commit step definitions + feature files
Trigger test execution
Load app to cloud
Tester + Test
Engineer
Pull request to the main branch
Consolidate test results
Continuous Integration
reporttest
build Pull request Approved?
no yes
Refuse pull requestDebug / update test suite
1Pull app source2
Build app3
Pull test source4
Build test project5
6 7 8
9
10
Git Hook
Merge to master branch
Distributed Digital Assurance Lab
Single unified UI externalizing different geographical locations
12
60 devices & Browser
40 devices & Browsers
Agenda
13
What Is Continuous Testing?01 10 min
Demo03 20 min
02 Case Study 10 min
Q&A04 10 min
14
• Jenkins
• Git
• Digital Assurance Lab
• Pre-prepared tests
Tools used
15
Demo
16
Process outcome
• Regression feedback within minutes
• New feature feedback within 24 hours
• Environment issue handling down from days to hours
Process
• Story points produced by R&D *3
• Quality improvement
• improved app store rating
• Higher customer satisfaction
• Market feedback
Organizational impact
17
Fast feedback vs. Coverage mode
Best Practices for test creating and execution
• Usually used by developers
• Purpose is quick validation
• Execute immediately on any available device
• Automatically Triggered by pull requests
• Usually used by automation team
• Purpose is ensuring app works across devices
• Scheduled nightly executions
• Run tests on all specified devices
Fast Feedback Coverage Mode
Agenda
18
What Is Continuous Testing?01 10 min
Demo03 20 min
02 Case Study 10 min
Q&A04 10 min
• Short / small tests
• Limit the test selection by time (5 minutes). Grow in device number if needed
• Select tests that ‘touch’ the entire system
• Remove naive tests
• Ignore device matrix and execute in fast feedback mode
Continuous Testing Best Practices
Continuous testing ≠ test automation
19
