innovations in test automation: it’s not all about regression
DESCRIPTION
Although classic test automation, which usually focuses on regression testing, has its its place in testing, there is much more you can do to improve testing productivity and its value to the project and your organization. Through experience-based examples, video clips, and demonstrations, John Fodeh shares one company’s innovation journey to improve its test automation practice. John illustrates how they learned to apply automated “test monkeys” that explore the software in new ways each time a test is executed. Then, he describes how the test team uses weighted probability tables to increase each test’s “intelligence” factor. Find out how they implemented model-based testing to improve automation effectiveness and how this practice led to the even more valuable behavior-driven testing approach they employ today. With these and other alternative approaches you, too, can get more mileage from your automation efforts. Join John to get inspired and start your own journey of innovation with new ideas that enhance your test automation strategy.TRANSCRIPT
T15 Test Automation
5/2/2013 1:30:00 PM
Innovations in Test Automation: It's Not All about Regression
Presented by:
John Fodeh
Cognizant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
John Fodeh
Managing Cognizant's testing services in the Nordics, John Fodeh has more than fifteen years of experience in software testing with extensive expertise in quality management, process improvement, testing tools, and strategy. John has helped organizations throughout Europe deal with issues in the areas of banking, insurance, retail, life sciences, and government. He has written a number of articles and contributed to books, including Experiences of Test Automation. John is an active member of special interest groups in software testing, chairman of the Danish Software Testing Board, and speaker at international testing conferences including STAREAST, STARWEST, and EuroSTAR. Contact John at [email protected].
4/16/2013
1
John Fodeh
Cognizant Technology Solutions
Innovations in Test
Automation
It’s Not All about Regression
| ©2012, Cognizant
Outline
• Innovation and testing
• The promise of automation
• Going beyond regression testing
• Automated Test Monkeys
• Model-Based and Behaviour-Driven approaches
• The human factor in automation
2
4/16/2013
2
| ©2012, Cognizant
� The word “innovation” derives
from the Latin word innovatus
� The noun form of innovare is "to
renew or change," stemming
from in—"into" + novus—"new“
� It is the process to renew an
existing idea
3
Innovation?
3
| ©2012, Cognizant
The Promise of Test Automation
• Replacing repetitive and tedious manual tasks
• Ensuring the consistency and repeatability of tests
• Performing tests that are difficult to run manually
• Accelerating test execution
4* Image source: Rosemont Engineering
4/16/2013
3
| ©2012, Cognizant
“Insanity is doing the same thing over and over again and expecting different results”
5
| ©2012, Cognizant
Traditional Test Automation
6
Typically
Automation of regression tests
Purpose
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is performed
when the software or its environment is changed.
[ISTQB Glossary, v2.1]
4/16/2013
4
| ©2012, Cognizant
Characteristics of Traditional Test Automation
• Static
• Simple
• Synchronized
• Vulnerable
7
| ©2012, Cognizant 8
Testing Beyond the GUI
GUI
Business Logic
Middleware
Data
4/16/2013
5
| ©2012, Cognizant
Monkey Testing
9
Monkey Testing refers to the
process of randomly exercising a
software program by means of an
automated test tool
| ©2012, Cognizant
Test Monkey
10
#script file: Test4.ascCP_KEY FOCUS_UP;TRACKBALL 0, 64;KEY L;KEY 5, LONG;KEY 8;
#script file: Test4.ascCP_KEY FOCUS_UP;TRACKBALL 0, 64;KEY L;KEY 5, LONG;KEY 8;CP_KEY FOCUS_UP;
Random Test Tool
#script file: Test4.ascKEY L;KEY 5, LONG;CP_KEY FOCUS_UP;#KEY BOX_SIZE_UPKEY 8;TRACKBALL 0, 64;KEY SHIFT, R;
Action List
Log file
Application Under Test
#script file: Test4.ascKEY L;KEY 5, LONG;CP_KEY FOCUS_UP;
#KEY BOX_SIZE_UPKEY 8;TRACKBALL 0, 64;KEY SHIFT, R;
Utilities
4/16/2013
6
| ©2012, Cognizant
Basic Features
A test monkey should:
• Select randomly from input range
• Enter input to AUT through the user interface
• Detect “life signs” of AUT
• Have robust logging facility
• (Re-) start and initialize AUT
• Have limited application knowledge but general environment awareness
11
| ©2012, Cognizant
Metrics – Application Reliability
0
20000
40000
60000
80000
100000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21Mean number of random operations between
Failures
Build no.
12
Entry
Criterion
Release
Criterion
4/16/2013
7
| ©2012, Cognizant
Added Value
• Early testing
• Cost effective
• Negative testing
• Long and complex test runs
• Reliability indicator
13
| ©2012, Cognizant
Limitations
• Misses obvious defects
• Does not emulate real use situations
• Long tests runs can be difficult to debug
• For a reliable statistical basis, long and multiple test runs are needed
14
Monkey Testing is a supplement
to your manual and automated
testing - not a replacement!
4/16/2013
8
| ©2012, Cognizant
Advanced Test Monkeys
• Wide application knowledge
• Application modeled in state table
• Illegal input to test error handling and recovery
• Emulate real and complex use scenarios
• Effective in finding defects
• Higher development and maintenance cost
15
| ©2012, Cognizant
State Transition Testing
16
Random Test Tool
State table
Log file
Application Under Test
Utilities
Start
State
Action End
State
ScriptStart state Action Probability
Input
range End state
Scanning Adjust frequency 45 3 20 Scanning
Adjust brightness 20 1 100 Scanning
Abort 5 Displaying (abort)
Edit preferences 10 Editing preferences
Increment depth 20 Scanning
4/16/2013
9
| ©2012, Cognizant
Intelligent Randomness
• Probability Tables
• Focus on selected areas
• Weight on specific use scenarios
• Continuous expansion and improvement
• Possibility to exclude certain parts
17
Select Text
Enter Text
Delete
Copy
Cut
Task n
…
Action
25%
5%
10%
5%
5%
* source: Whitmill, Kelly
| ©2012, Cognizant
Model-Based Automation
18
Test Tool
Log file
Application Under Test
Utilities
Start
State
Action End
State
ScriptState tableModel
4/16/2013
10
| ©2012, Cognizant
Behavior-Driven approach
• Based on Behavioral specifications
• User Stories defined using ubiquitous language
• Principles of Test Driven Development
• Collaborative approach for business analysts, Software development & testing
• Shared tools and process
19
| ©2012, Cognizant
People Issues
• Test automation depends on the competence, creativity and motivation of your team
• Dedicated resources are required
• Management support is essential
209
4/16/2013
11
| ©2012, Cognizant
Summary
• No “one size fits all” solution
• Maturity is about analyzing the current practice and finding ways to do evolve
• Are you ready for your innovation journey?
21