testcover.com copyright © 2005 testcover.com, llc. all rights reserved. the bug stops here...
TRANSCRIPT
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
The Bug Stops HereThe Bug Stops Here
Improving Test Designwith a
Novel Test Case Generator
Technical PresentationSalon 6
Wednesday May 18, 1:45 – 2:45
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comHow to Spin a WebHow to Spin a Web
of Effective Test Casesof Effective Test Cases• Test case generator makes “all
pairs” testing simpler, quicker, more practical
• New technology improves test case coverage and fault discovery
• Service continually integrates research advances to reduce number of test cases
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comTesting Breakthrough Testing Breakthrough
NeededNeeded• Increasing dependency on software,
networks and distributed applications
• Increasing expectation and need for quality
• Facts of life:• More features and complexity• Shorter development cycles• More reuse of components
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
This PresentationThis Presentation• Describes approaches to software test
case generation• Test configurations• System-level functional tests – integration
tests, functional verification tests, regression tests etc.
• Does not address• Unit testing (e.g. code coverage)• Stability and load testing
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
This PresentationThis Presentation• Explains how the Testcover.com Covering
Test Case Generator Service improves test case coverage and fault discovery
• Shows how to use the service with examples• Tells how to receive 1 month service free• Is available on the web at Testcover.com
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
• Founded in 2003 to aid testers in the selection of efficient sets of test cases
• Covering Test Case Generator Service• On the web at Testcover.com• Technology based on:
• advances in covering array construction• practical software engineering
experience
Testcover.com Dedicated to Excellence in Test
Design
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
• Test three new applications on a networked computer• Test factors and their values:
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• A simple approach:
• One test configuration for each factor value
• Select a most common or default value for each factor
• Vary each factor over all its values while the other factors are set to their default values
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Default Values
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 + 4 + 3 + 2 + 4 = 16Duplicate test cases: - 4
________= 12
• A simple approach:• How many test configurations? 12
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• A simple approach:
• Small number of test configurations• All individual factor values are covered• Many pairs of factor values are not:
• Application 2 with Cable Connection• Firefox Browser with ME Operating System• Application 3 with Medium Resolution• More…
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• A better approach:
• Include all pairs of factor values• Test all pair-wise interactions
• How many pairs of factor values? 101• How many test configurations? 16
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comConfiguration ExampleConfiguration Example
How many pairs of factor values?
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39
= 4 x 3
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39
= 4 x 3 + 4 x 2
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39
= 4 x 3 + 4 x 2 + 4 x 4 = 36
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36
= 3 x 2
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36
= 3 x 2 + 3 x 4 = 18
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Application BrowserDisplay
ResolutionOperating System Connection
Application 1Internet Explorer High XP Dial-up
Application 2 Netscape Medium ME DSL
Application 3 Mozilla Low Cable
Firefox LAN
= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36
= 3 x 2 + 3 x 4 = 18
= 2 x 4 = 8+_______
101
Configuration ExampleConfiguration Example How many pairs of factor values?
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• 16 configurations cover all 101 pairsTest Combo
Case ID Countdown
3 Values 4 Values 3 Values 2 Values 4 Values 101
1 App1 Netscape High ME LAN 91
2 App3 Firefox Medium ME Dial-up 81
3 App2 Mozilla Low ME Cable 71
4 App2 Firefox High XP DSL 61
5 App1 IE Low XP LAN 52
6 App3 Netscape Medium XP Cable 43
7 App1 Mozilla Medium ME DSL 36
8 App3 IE High ME Cable 30
9 App2 Mozilla High XP Dial-up 24
10 App3 Netscape Low ME DSL 19
11 App2 IE Medium ME DSL 15
12 App3 Mozilla Medium XP LAN 11
13 App1 Firefox Low ME Cable 7
14 App2 Netscape Low ME Dial-up 4
15 App1 IE High XP Dial-up 2
16 App2 Firefox Low ME LAN 0
Display Resolution Operating System ConnectionApplication Browser
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Combo CountdownCombo Countdown• Coverage metric• Number of untested pairs after each
test case• Stops at 0 or the number of
untested pairs when there are constraints
• Illustrates diminishing returns• Supports risk assessment
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Test Test DefinitionsDefinitions
• Test factor – any variable whose values are to be controlled during testing
• Test factor value – specific value taken by one of the test factors
• Test case – set of test factor values with one allowed value for each of the test factors
• Combination – association of some number of factor values (pairs, triples, etc.)
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Array DefinitionsArray Definitions• Size – number of test cases,
number of rows in array• Degree – number of test factors,
number of columns in array• Order – number of values for a test
factor• Strength – size of combinations
(2: pairs, 3: triples, etc.)
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Mixed arraysMixed arrays• Mixed array has factors with different
orders• Example: 52 43 32 21
• 2 factors with 5 values• 3 factors with 4 values• 2 factors with 3 values• 1 factor with 2 values
• Degree = 2 + 3 + 2 + 1 = 8• There is a 52 43 32 21 array with strength 2 and
size 25
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Early Early ApproachesApproaches
• Orthogonal arrays used in agricultural and medical experiments
• Dr. Genichi Taguchi used orthogonal arrays to select test parameters
• Statistical coverage where exhaustive testing is not possible
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Real-world ConstraintsReal-world Constraints• What if the applications in the
configuration example are ported to Linux and Macintosh OSX?
• Internet Explorer is supported on Windows only
• There are Browser-Operating System pairs that must be excluded
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• The wrong answer:
Test Combo
Case ID Countdown
3 Values 4 Values 3 Values 4 Values 4 Values 129
1 App1 IE High XP Dial-up 119
2 App2 IE Medium ME DSL 109
3 App3 IE Low Linux Cable 99
4 App1 Netscape Medium Linux LAN 89
5 App2 Netscape High MacOSX Cable 79
6 App3 Mozilla High ME LAN 69
7 App1 Mozilla Low MacOSX DSL 59
8 App2 Firefox Low XP LAN 49
9 App3 Firefox Medium MacOSX Dial-up 39
10 App3 Netscape Low XP DSL 32
11 App2 Mozilla Low Linux Dial-up 25
12 App1 Firefox Low ME Cable 18
13 App3 Mozilla Medium XP Cable 12
14 App3 Firefox High Linux DSL 6
15 App3 IE Low MacOSX LAN 3
16 App3 Netscape Low ME Dial-up 0
Application Browser Display Resolution Operating System Connection
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comConstrained Array Test Constrained Array Test
SystemSystem• Motivated by testing Local Area Network
product configurations• Developed in 1990 to handle constraints• Employed greedy search algorithm• Used internally at AT&T as best practice• Described in STAR proceedings May 1994
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Current ChallengesCurrent Challenges
1. Find small array quickly2. Adapt to system constraints simply3. Support multiple system states4. Allow easy use
Testcover.com addresses these challenges
Let’s see how…
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Sample ArraysSample Arrays
Size Array Size Array Size Array
6 210 23 48 61 710
8 230 28 424 91 730
11 35 33 58 78 811
15 320 45 530 120 830
19 330 46 69 81 910
16 45 76 627 153 930
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Sample Mixed ArraysSample Mixed Arrays
Size Degree
Array Size Degree
Array
15 30 36 224 12 10 41 33 26
22 25 45 220 20 17 51 44 212
30 25 45 320 25 8 52 43 33
27 23 53 320 30 30 61 55 224
39 23 63 320 36 15 62 51 312
49 23 73 420 42 15 71 62 312
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Challenge 1: Fewer Test Challenge 1: Fewer Test CasesCases
• Small arrays result from multiple techniques
• Testcover.com continually integrates research advances to reduce the numbers of test cases
• New technology provides quicker results
The results given here are subject to improvement without notice!
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comChallenge 2: Handle Challenge 2: Handle
ConstraintsConstraints• Direct Product Block notation
enables simple descriptions of complex constraints and dependencies
• Let’s see how DPB notation works with the Configuration Example
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• No constraints
– one block
Configuration Example
Application
Browser
Display Resolution
Operating System
Connection
#
App1 App2 App3
IE Netscape Mozilla Firefox
High Medium Low
XP ME
Dial-up DSL Cable LAN
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comDirect Product Block Direct Product Block
NotationNotation• Block represents all possible test
cases for selected values of the test factors
• Multiple blocks represent all allowed test cases – as a set of products
• Simpler than enumerating all disallowed combinations
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Configuration ExampleConfiguration Example• Browser/OS
constraints – two blocks
Configuration Example - With Constraints
Application
Browser
Display Resolution
Operating System
Connection
#
App1 App2 App3
IE Netscape Mozilla Firefox
High Medium Low
XP ME
Dial-up DSL Cable LAN
+
App1 App2 App3
Netscape Mozilla Firefox
High Medium Low
Linux MacOSX
Dial-up DSL Cable LAN
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comChallenge 3: Multiple Challenge 3: Multiple
StatesStates• Calendar example illustrates:
• Direct Product Block notation• Constraints – different month lengths• Multiple states using different
partitions of test cases –• Normal (good dates)• Error handling (too long months)
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCalendar Calendar ExampleExample
Request Part 1
Calendar Example
Month
Day
Year
#ok All good dates
jan feb mar apr may jun jul aug sep oct nov dec
1 10
2003 2004 2005
+ long month last day
jan mar may jul aug oct dec
31
2003 2004 2005
+ short month last day
apr jun sep nov
30
2003 2004 2005
+ feb last day
feb
28
2003 2005
+ leap day
feb
29
2004
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCalendar Calendar ExampleExample
Request Part 2
#err Too long month
+ too long long month
jan mar may jul aug oct dec
32
2003
+ too long short month
apr jun sep nov
31
2004
+ feb too long, regular year
feb
29
2005
+ feb too long, leap year
feb
30
2004
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCalendar Calendar ExampleExample
Results Part 1
Test Combo
Case ID Countdown
12 Values 6 Values 3 Values 126
ok1 jan 1 2003 123
ok2 jan 10 2004 120
ok3 feb 10 2003 117
ok4 feb 1 2005 114
ok5 mar 1 2004 111
ok6 mar 10 2005 108
ok7 may 31 2005 105
ok8 aug 31 2004 102
ok9 oct 31 2003 99
ok10 nov 30 2004 96
ok11 apr 30 2005 93
ok12 jun 30 2003 90
ok13 feb 29 2004 87
ok14 apr 1 2003 85
ok15 apr 10 2004 83
ok16 may 10 2003 81
ok17 jun 1 2004 79
ok18 jun 10 2005 77
ok19 jul 1 2003 75
ok20 jul 10 2004 73
YearMonth Day
#1. All good dates
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCalendar Calendar ExampleExample
Results Part 2
Test Combo
Case ID Countdown
12 Values 6 Values 3 Values 126
ok21 aug 10 2003 71
ok22 aug 1 2005 69
ok23 sep 10 2003 67
ok24 sep 1 2004 65
ok25 oct 10 2004 63
ok26 nov 10 2003 61
ok27 nov 1 2005 59
ok28 dec 10 2003 57
ok29 dec 1 2004 55
ok30 jul 31 2005 53
ok31 dec 31 2005 51
ok32 jan 31 2005 49
ok33 mar 31 2003 47
ok34 sep 30 2005 45
ok35 feb 28 2003 43
ok36 may 10 2004 42
ok37 may 1 2005 41
ok38 oct 1 2003 40
ok39 oct 10 2005 39
ok40 feb 28 2005 38
YearMonth Day
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCalendar Calendar ExampleExample
Results Part 3
Test Combo
Case ID Countdown
12 Values 4 Values 3 Values 96
err1 jan 32 2003 93
err2 apr 31 2004 90
err3 feb 29 2005 87
err4 feb 30 2004 84
err5 mar 32 2003 82
err6 may 32 2003 80
err7 jul 32 2003 78
err8 aug 32 2003 76
err9 oct 32 2003 74
err10 dec 32 2003 72
err11 jun 31 2004 70
err12 sep 31 2004 68
err13 nov 31 2004 66
Month Day Year
#2. Too long month
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Challenge 4: Ease of UseChallenge 4: Ease of Use• Direct Product Block notation
• Simple to learn• Flexible and efficient
• Internet-based service• Easy access without special installation• Independent of development
environment• Enhancements without upgrade work
• Practical for everyday use
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Challenge 4: Ease of UseChallenge 4: Ease of Use• Advanced Web Search example
illustrates:• Practical for real HTML form• Direct Product Block notation• Constraints – language button use• Small number of test cases
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comAdvanced Web Search Advanced Web Search
ExampleExample• See HTML form
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comAdvanced Advanced
Web Search Web Search ExampleExample
• Request part 1
Y! Advanced Search Form
All words
All page part
Exact phrase
Exact page part
Any words
Any page part
Excluded words
Excluded page part
Updated
Site/domain
Format
Filter
Country
Language
Arabic
Chinese
English
Japanese
Russian
Thai
Results/page
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comAdvanced Advanced
Web Search Web Search ExampleExample
• Request part 2
#+any language
all list 1,all list 2,all list 3,all list 4any,title,url
exact phrase 1,exact phrase 2,exact phrase 3,exact phrase 4any,title,urlany list 1,any list 2,any list 3,any list 4
any,title,urlexcluded list 1,excluded list 2,excluded list 3,excluded list 4
any,title,urlanytime,past 3 months,past 6 months,past year
any,.com,.edu,.gov,.org,domain 1,domain 2all,.html,.pdf,.xls,.ppt,.doc,.xml,.txton,off
any,Argentina,Brazil,Denmark,France,Germany,United Statesany
uncheckedunchecked
uncheckeduncheckedunchecked
unchecked10,15,20,30,40,100
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comAdvanced Advanced
Web Search Web Search ExampleExample
• Request part 3
+specified language(s)
all list 1,all list 2,all list 3,all list 4
any,title,url
exact phrase 1,exact phrase 2,exact phrase 3,exact phrase 4
any,title,url
any list 1,any list 2,any list 3,any list 4
any,title,url
excluded list 1,excluded list 2,excluded list 3,excluded list 4
any,title,url
anytime,past 3 months,past 6 months,past year
any,.com,.edu,.gov,.org,domain 1,domain 2
all,.html,.pdf,.xls,.ppt,.doc,.xml,.txt
on,off
any,Argentina,Brazil,Denmark,France,Germany,United States
specified
checked,unchecked
checked,unchecked
checked,unchecked
checked,unchecked
checked,unchecked
checked,unchecked
10,15,20,30,40,100
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comAdvanced Web Search Advanced Web Search
ExampleExample• 81 72 61 45 34 28 Array• Degree 21• 2715 pairs, 6 not covered
(constraints)• Size 63 (compared to 8 x 7 = 56)
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comHow to Spin a WebHow to Spin a Web
of Effective Test Casesof Effective Test Cases• New technology improves test
case coverage and fault discovery• Service continually integrates
research advances to reduce number of test cases
• Exclusive Direct Product Block notation simplifies use
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Exhibit at Booth 8Exhibit at Booth 8• See these and other examples run• Receive your brochure and gift• Register for 1 free month of
service
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comSTAREAST 2005 First Month STAREAST 2005 First Month
FreeFree4 Month Subscription for the Price of 3
• For STAREAST 2005 Attendees Only• Register at Booth 8 May 18 – May 19• Try Testcover.com Service Free for 1
Month• Your Choice:
• Pay invoice for 4-Month Subscription• Cancel On-line
• 1 Month Free Service either way – $100 Value
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Back-up SlidesBack-up Slides
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Test Case ObjectivesTest Case Objectives• Run enough of the “right” test cases
to exercise system interactions completely
• Avoid wasting time and money on unnecessary test cases
• Insure system behavior is thoroughly verified
• Allow for complex and changing system requirements
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comOrthogonal Arrays & Covering Orthogonal Arrays & Covering
ArraysArrays• Every pair of columns in an orthogonal
array contains all pairs of values an equal number of times
• Every pair of columns in a covering array contains all pairs of values at least once
• Orthogonal arrays are covering arrays• Both types can be mixed (different
orders)
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comOrthogonal Arrays & Covering Orthogonal Arrays & Covering
ArraysArrays• Orthogonal arrays are balanced –
suggest which factor values give best measured results• Tuning complex systems• Medical & agricultural trials
• Covering arrays are not required to be balanced – can be smaller for pass/fail type tests
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comCovering Arrays vs. Random Covering Arrays vs. Random
ValuesValues• Goal: Validate operation quickly –
integration, function, regression – Arrays provide coverage with few test cases
• Goal: Validate continuous operation – stability, load, stress – Random values can provide many different test cases
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
• Strength t arrays are defined similarly – Every set of t columns in an array contains all t-tuples of values (3-tuples, 4-tuples, etc.)
• Size (number of test cases) increases with strength – Minimum size is product of the t largest orders
• Example: 52 43 32 21 arrays
Strength: 2 3 4
Minimum Size: 25 100
400
Higher Strength ArraysHigher Strength Arrays
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
Higher Strength ArraysHigher Strength Arrays• Higher strength arrays can cover more
combinations of factor values, with a cost of more test cases
• Generally finding higher strength arrays requires more processing
See Sherwood, Martirosyan, and Colbourn, Sections 4 & 5.
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.com
RDF/XML ResultsRDF/XML Results• The Resource Description Framework (RDF) is a general-
purpose language endorsed by the World Wide Web Consortium (W3C) for representing information on the web.
• RDF is based on the idea of identifying things using Uniform Resource Identifiers (URIs) and describing resources in terms of simple properties.
• RDF represents simple statements about resources as a graph of nodes and arcs representing the resources, and their properties and values.
• RDF also provides an XML-based syntax (called RDF/XML) for recording and exchanging these graphs.
• In addition to HTML results, the Testcover.com test case generator provides RDF/XML results for use by other applications.
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comSources & Further Sources & Further
ReadingReadingCompanies Seek to Hold Software Makers Liable for Flaws, David Bank, The Wall Street Journal, page B1, February 24, 2005.
To Darned Big to Test, Keith Stobie, ACM Queue, vol. 3, no. 1, pages 30-37, February 2005.
Products of Mixed Covering Arrays of Strength Two, C. J. Colbourn, S. S. Martirosyan, G. L. Mullen, D. Shasha, G. B. Sherwood, and J. L. Yucas, preprint, 2004.
Covering Arrays of Higher Strength From Permutation Vectors, G. B. Sherwood, S. S. Martirosyan, and C. J. Colbourn, preprint, 2004.
On the Construction of Orthogonal Arrays and Covering Arrays Using Permutation Groups, George Sherwood, http://home.att.net/~gsherwood/cover.htm March 2002 - June 2004.
Copyright © 2005 Testcover.com, LLC. All rights reserved.
Testcover.comSources & Further Sources & Further
ReadingReadingEffective Testing of Factor Combinations, George Sherwood,
Third International Conference on Software Testing, Analysis & Review, May 8-12, 1994, Washington, DC.
Orthogonal Arrays Theory and Applications, A. S. Hedayat, N. J. A. Sloane, and John Stufken, 1999 Springer-Verlag.
Quality Engineering Using Robust Design, M. S. Phadke, 1989, Prentice Hall.
M. J. Suárez-Cabal and J. Tuya. Using an SQL Coverage Measurement for Testing Database Applications. ACM Software Engineering Notes. 29(6):253-262, 2004.