testing in the small (aka unit testing, class testing) 1209
TRANSCRIPT
![Page 1: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/1.jpg)
Testing in the Small (aka Unit Testing, Class Testing)
1209
![Page 2: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/2.jpg)
Unit Testing
• Focus on the smallest units of code: – Functions– Methods– Subroutines
• Frequently done (at least partially) during code development by code developers
• Often the target of testing frameworks such as JUnit
![Page 3: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/3.jpg)
Each component is
• Tested in isolation from the rest of the system• Tested in a controlled environment
– Uses appropriately chosen input data
– Uses component-level design description as guide
![Page 4: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/4.jpg)
During Unit Tests
• Data transformations across the module are tested• Data structures are tested to ensure data integrity
![Page 5: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/5.jpg)
Unit Test Procedures
Driver
Module to be tested
Stub Stub
Results Testcases
![Page 6: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/6.jpg)
Two fundamental approaches:
• Black box– Based on specification– Inner structure of test object is not considered
• White box– Based on specification– Inner structure of test object is the basis of test case
selection
• Effectiveness of black box is similar to white box, but the mistakes found are different. (Hetzel 1976, Myers 1978)
![Page 7: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/7.jpg)
Black Box Unit Testing
1209
![Page 8: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/8.jpg)
What is black-box testing?
• Unit (code, module) seen as a black box
• No access to the internal or logical structure
• Determine if given input produces expected output.
Input Output
![Page 9: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/9.jpg)
Black Box Testing
• Test set is derived from requirements • Goal is to cover the input space• Lots of approaches to describing input space:
– Equivalence Classes– Boundary Value Analysis– Decision Tables– State Transitions– Use Cases– . . .
![Page 10: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/10.jpg)
Advantage and Disadvantage
• Advantage: it does not require access to the internal logic of a component
• Disadvantage: in most real-world applications, impossible to test all possible inputs
• Need to define an efficient strategy to limit number of test cases
![Page 11: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/11.jpg)
General Black Box Process
• Analyze requirements
• Select valid and invalid inputs
• Determine expected outputs
• Construct tests
• Run tests
• Compare actual outputs to expected outputs
![Page 12: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/12.jpg)
Equivalence classes: Basic Strategy
• Partition the input into equivalence classes– This is the tricky part– It’s an equivalence class if:
• Every test using one element of the class tests the same thing that any other element of the class tests
• If an error is found with one element, it should be found with any element
• If an error is not found with some element, it is not found by any element
• Test a subset from each class
![Page 13: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/13.jpg)
Basic strategy:
Example: fact(n): if n<0, or n>=200 errorif 0<=n<=20, exact valueif 20<n<200, approximate value within .1%
What classes can you see?
G
![Page 14: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/14.jpg)
Basic strategy:
Example: fact(n): if n<0, or n>=200 errorif 0<=n<=20, exact valueif 20<n<200, approximate value within .1%
Obvious classes are n<0, 0<=n<=20, 20<n<200, and 200<=n.
Might be some subclasses here, also, but this is a good start.
![Page 15: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/15.jpg)
Simple Example:
• Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior.
• The price depends on the type of traveler. • The seat reservation does not depend on the type
of traveler.• How many test cases can you identify for the
reservation component and the billing component?
G
![Page 16: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/16.jpg)
Finding equivalence classes:
• Identify restrictions for inputs and outputs in the specification
![Page 17: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/17.jpg)
Finding equivalence classes:
• Identify restrictions for inputs and outputs in the specification
• If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
![Page 18: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/18.jpg)
Finding equivalence classes:
• Identify restrictions for inputs and outputs in the specification
• If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
• If a number of values is required, create one valid and two invalid classes (one invalid, two invalid)
![Page 19: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/19.jpg)
Finding equivalence classes:
• Identify restrictions for inputs and outputs in the specification
• If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
• If a number of values is required, create one valid and two invalid classes
• If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set
![Page 20: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/20.jpg)
Finding equivalence classes:
• Identify restrictions for inputs and outputs in the specification
• If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
• If a number of values is required, create one valid and two invalid classes
• If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set
• If there is a condition, create two classes, one satisfying and one not satisfying the condition
![Page 21: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/21.jpg)
Boundary Values:
• Programs that fail at interior elements of a class usually fail at the boundaries too.
• Test the boundaries. – if it should work for 1-99, test 0, 1, 99, 100.– if it works for A-Z, try @, A, Z, [, a, and z
• The hard part is identifying boundaries
![Page 22: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/22.jpg)
Hints
• If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11– It may be possible to test the boundaries of outputs,
also.• For ordered sets, check the first and last elements• For complex data structures, the empty list, full
lists, the zero array, and the null pointer should be tested
• Extremely large data sets should be tested• Check for off-by-one errors
![Page 23: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/23.jpg)
More Hints
• Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed)– Numeric limits (e.g., test 255 and 256 for 8-bit
values)– Implementation limits (e.g., max array size)
![Page 24: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/24.jpg)
Boundary Value: in class
• Determine the boundary values for US Postal Service ZIP codes
• Determine the boundary values for a 15- character last name entry.
G
![Page 25: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/25.jpg)
Decision Tables
• Construct a table (to help organize the testing)
• Identify each rule or condition in the system that depends on some input
• For each input to one of these rules, list the combinations of inputs and the expected results
![Page 26: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/26.jpg)
Decision Table Example
Test Case C1
Student
C2
Senior
Result
Discount?
1 T T T
2 T F T
3 F T T
4 F F F
Theater ticket prices are discounted for senior citizens and students.
![Page 27: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/27.jpg)
Pairwise Testing
![Page 28: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/28.jpg)
Problem 1From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech
House Publishers, 2004.
• A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
• It must work using RealPlayer, MediaPlayer, or no plugin.• It needs to run under Windows ME, NT, 2000, XP, and
Vista, 7.0, and 8.0• It needs to accept pages from IIS, Apache and WebLogic
running on Windows NT, 2000, and Linux servers
• How many different configurations are there?
G
![Page 29: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/29.jpg)
Problem 1
• A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
• It must work using RealPlayer, MediaPlayer, or no plugin.• It needs to run under Windows ME, NT, 2000, XP, Vista,
7.0, and 8.0• It needs to accept pages from IIS, Apache and WebLogic
running on Windows NT, 2000, and Linux servers
• How many different configurations are there?• 9 browsers x 3 plugins x 7 OS x 3 web servers x 3 server
OSs = 1701 combinations
![Page 30: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/30.jpg)
Problem 2
• A bank is ready to test a data processing system• Customer types
– Gold (i.e., normal)– Platinum (i.e., important)– Business– Non profits
• Account types– Checking– Savings– Mortgages – Consumer loans– Commercial loans
• States (with different rules): CA, NV, UT, ID, AZ, NM
• How many different configurations are there?
From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2004.
![Page 31: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/31.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test
![Page 32: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/32.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business
![Page 33: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/33.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases
![Page 34: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/34.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases• Choose a few that the programmers already ran
![Page 35: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/35.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases• Choose a few that the programmers already ran• Choose the tests that are easy to create
![Page 36: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/36.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases• Choose a few that the programmers already ran• Choose the tests that are easy to create• List all combinations and choose first few
![Page 37: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/37.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases• Choose a few that the programmers already ran• Choose the tests that are easy to create• List all combinations and choose first few• List all combinations and randomly choose some
![Page 38: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/38.jpg)
When given a large number of combinations: (options improve …)
• Give up and don’t test• Test all combinations … miss targets, delay
product launch, and go out of business• Choose one or two cases• Choose a few that the programmers already ran• Choose the tests that are easy to create• List all combinations and choose first few• List all combinations and randomly choose some• “Magically” choose a small subset with high
probability of revealing defects
![Page 39: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/39.jpg)
Orthogonal Arrays
• A 2-D array with the property– All pairwise combinations occur in every pair of
columns
• Example: consider 3 variables (columns) with {A,B}, {1,2}, and (α,β)
1 2 3
1 1 A α
2 1 B β
3 2 A β
4 2 B α
![Page 40: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/40.jpg)
Orthogonal Array Example
1 2 3
1 1 A α
2 1 B β
3 2 A β
4 2 B α
Look at each pair of columns (1 and 2),( 1 and 3), and (2 and 3)
Does each of the 4 pairings appear in each?
(Yes, of course!)
![Page 41: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/41.jpg)
Pairwise Testing Algorithm
1. Identify variables
2. Determine choices for each variable
3. Locate an orthogonal array
4. Map test cases to the orthogonal array
5. Construct tests
![Page 42: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/42.jpg)
Example using Bank
1. Identify variables1. Customers (4)2. Account types (5)3. States (6)
2. Determine choices for each variable1. Customer types (Gold, Platinum, Business, Non Profit)2. Account types (Checking, Savings, Mortgages, Consumer loans,
Commercial loans3. States (CA, NV, UT, ID, AZ, NM)
3. Locate an orthogonal array 4. Map test cases to the orthogonal array5. Construct tests
![Page 43: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/43.jpg)
Locate an orthogonal array (look up on web)
1 1 1
1 2 2
1 3 3
1 4 4
2 1 2
2 2 1
2 3 4
2 4 3
3 1 3
3 2 4
3 3 1
3 4 2
4 1 4
4 2 3
4 3 2
4 4 1
5 5 1
5 1 2
5 2 3
5 3 4
6 5 2
6 1 1
6 2 4
6 3 3
1 5 3
2 5 4
3 5 1
4 5 2
5 4 1
6 4 2
![Page 44: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/44.jpg)
Map Test Cases (30)CA Check Gold
CA Save Plat
CA Mort Busi
CA Cons NonP
NV Check Plat
NV Save Gold
NV Mort NonP
NV Cons Busi
UT Check Busi
UT Save NonP
UT Mort Gold
UT Cons Plat
ID Check NonP
ID Save Busi
ID Mort Plat
ID Cons Gold
AZ Comm Gold
AZ Check Plat
AZ Save Busi
AZ Mort NonP
NM Comm Plat
NM Check Gold
NM Save NonP
NM Mort Busi
CA Comm Busi
NV Comm NonP
UT Comm Gold
ID Comm Plat
AZ Cons Gold
NM Cons Plat
![Page 45: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/45.jpg)
Pairwise Summary• When the combination is large, test all pairs,
not all combinations
• Studies indicate that most defects are single or double-mode effects – Can be found in pairing– Few defects require more than two modes
• Orthogonal arrays have the pairwise property and can be found or generated
![Page 46: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/46.jpg)
State Transition Testing
![Page 47: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/47.jpg)
State Transition Testing
• Build STD of system or component• Cover the STD
– Visit each state– Trigger each event– Exercises each transition– Exercise each path*
* Might not be possible
![Page 48: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/48.jpg)
Example
Res MadePaid
CancelledNon Pay
Ticketed
CancelledCust
Used
Cust Order/start timer
Payment madeTicket Printed
Time Expires
Ticket DeliveredCancel/refund
Cancel/refund
Cancel
Customer makes reservation and haslimited time to pay. May cancel at anytime.
Groups: How many tests to visit each state?
![Page 49: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/49.jpg)
Example: Each State
Res MadePaid
CancelledNon Pay
Ticketed
CancelledCust
Used
Cust Order/start timer
Payment madeTicket Printed
Time Expires
Ticket DeliveredCancel/refund
Cancel/refund
Cancel
3 tests needed: From start to final From start to cancelled non-pay From start to Res Made to Cancelled Cust
![Page 50: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/50.jpg)
Example: Each Event
Res MadePaid
CancelledNon Pay
Ticketed
CancelledCust
Used
Cust Order/start timer
Payment madeTicket Printed
Time Expires
Ticket DeliveredCancel/refund
Cancel/refund
Cancel
3 tests needed: From start to final From start to cancelled non-pay From start to Res Made to Cancelled Cust
![Page 51: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/51.jpg)
Example: Each Transition
Res MadePaid
CancelledNon Pay
Ticketed
CancelledCust
Used
Cust Order/start timer
Payment madeTicket Printed
Time Expires
Ticket DeliveredCancel/refund
Cancel/refund
Cancel
5 tests needed
![Page 52: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/52.jpg)
Use Case Testing
• Use the use cases to specify test cases
• Use case specifies both normal, alternate, and exceptional operations
• Use cases may not have sufficient detail– Beizer estimates that 30-40% of effort in testing
transactions is generating the test data– Budget for this!
![Page 53: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/53.jpg)
White-Box Testing
Pfleeger, S. Software Engineering Theory and Practice 2nd Edition. Prentice Hall, 2001.Ghezzi, C. et al., Fundamentals of Software Engineering. Prentice Hall, 2002.Pressman, R., Software Engineering A Practitioner’s Approach, Mc Graw Hill, 2005.Hutchinson, Marnie, Software Testing Fundamentals, Wiley, 2003.
0310
![Page 54: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/54.jpg)
White Box Testing
• Also known as: – Glass box testing– Structural testing
• Test set is derived from structure of code • Code structure represented as a Control Flow Graph• Goal is to cover the CFG
![Page 55: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/55.jpg)
Control Flow Graphs• Programs are made of three kinds of statements:
– Sequence (i.e., series of statements)
– Condition (i.e., if statement)
– Iteration (i.e., while, for, repeat statements)
![Page 56: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/56.jpg)
Control Flow Graphs• Programs are made of three kinds of statements:
– Sequence (i.e., series of statements)
– Condition (i.e., if statement)
– Iteration (i.e., while, for, repeat statements)
• CFG: visual representation of flow of control.– Node represents a sequence of statements with single entry
and single exit
– Edge represents transfer of control from one node to another
![Page 57: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/57.jpg)
Control Flow Graph (CFG)
n1
Join
n3
n1n1
Join
Sequence If-then-else If-then Iterative
![Page 58: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/58.jpg)
Control Flow Graph (CFG)
n1
Join
n3
n1n1
Join
Sequence If-then-else If-then Iterative
When drawing CFG, ensure that there is one exit: include the join node if needed
![Page 59: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/59.jpg)
Example 1: CFG
1. read (result);2. read (x,k)3. while result < 0 then {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result
Draw CFG
![Page 60: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/60.jpg)
Example 1: CFG
1. read (result);2. read (x,k)3. while result < 0 then {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result
3
5
6
7
9
8
2
1
4
![Page 61: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/61.jpg)
Example 1: CFG
3
4,5
6
Join
9
1. read (result);2. read (x,k)3. while result < 0 then {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result
7,8
1,2
3
5
6
7
9
8
2
1
4
![Page 62: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/62.jpg)
In Class: Write CFGsExample 2
1. if (a < b) then 2. while(a < n)3. a a + 1;4. else 5. while(b < n)6. b b + 1;
Example 3
1. read (a,b); 2. if (a 0 && b 0) then {3. c a + b;4. if c > 10 then 5. c max6. else c min }7. else print ‘Done’
Example 4
1. read (a,b); 2. if (a 0 || b 0) then {3. c a + b4. while( c < 100)5. c a + b; }6. c a * b
![Page 63: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/63.jpg)
Write a CFG
Example 2
1. if (a < b) then 2. while (a < n)3. a a + 1;4. else 5. while (b < n)6. b b + 1;
1
2
3
Join
5
6
![Page 64: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/64.jpg)
Answers
Example 3
1. read (a,b); 2. if (a 0 && b 0) then {3. c a + b;4. if c > 10 then 5. c max6. else c min }7. else print ‘Done’
1,2
Join
3,4
5 6
Join
7
![Page 65: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/65.jpg)
Answers
Example 4
1. read (a,b); 2. if (a 0 || b 0) then {3. c a + b4. while( c < 100)5. c a + b; }6. c a * b
1, 2
3
Join
4
5
6
![Page 66: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/66.jpg)
Coverage
• Statement
• Branch
• Condition
• Path
• Def-use
• Others
![Page 67: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/67.jpg)
Statement Coverage
• Every statement gets executed at least once
• Every node in the CFG gets visited at least once
![Page 68: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/68.jpg)
Examples: number of paths needed
for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
![Page 69: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/69.jpg)
Examples: number of paths needed
for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
4 1
![Page 70: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/70.jpg)
Branch coverage
• Every decision is made true and false• Every edge in a CFG of the program gets
traversed at least once• Also known as
– Decision coverage– All edge coverage– Basis path coverage
• Branch is finer than statement• C1 is finer than C2 if
T1C1 T2C2 T2 T1
![Page 71: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/71.jpg)
Examples: number of paths needed
for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
![Page 72: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/72.jpg)
Examples: number of paths needed
for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
5 2
![Page 73: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/73.jpg)
Condition coverage
• Every complex condition is made true and false by every possible combination– E.G., (x and y)
• x = true, y = true• x=false, y=true• x = true, y= false• x =false, y = false
• There are lots of different types of condition coverage: Condition, multiple condition, condition/decision, modified condition/decision (MCDC), …I’m only covering the simplest.
• Condition coverage is not finer than branch coverage – There are pathological cases where you can achieve Condition and not
Branch– Under most circumstances, achieving Condition achieves Branch
![Page 74: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/74.jpg)
Condition Coverage: CFGs• One way to determine the number of paths is to break the compound
conditional into atomic conditionals• Suppose you were writing the CFG for the assembly language
implementation of the control construct If (A AND B) then
CEndif
(short circuit eval) (no short circuit eval) LD A LD A ; in general, lots of BZ :endif LAND B ; code for A and
B LD B BZ :endif BZ :endif JSR C JSR C :endif nop:endif nop
![Page 75: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/75.jpg)
AND Condition
1. read (a,b); 2. if (a == 0 && b == 0) then {3. c a + b }4. else c a * b
paths:1, 2A,2B,3, J1, 2A, 2B, 4, J1, 2A, 4, J
2A
2B
Join
3
4
1
![Page 76: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/76.jpg)
OR Condition
1. read (a,b); 2. if (a == 0 || b == 0) then }3. c a + b;4. while( c < 100)5. c a + b;}
Paths:1, 2A, 3, 4, 5, 4 … J1, 2A, 3, 4, J1, 2A, 2B, 3, 4, 5, 4, … J1,2A, 2B, J
2A
3
Join
4
2B
5
1
![Page 77: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/77.jpg)
Path
• A path is a sequence of statements
• A path is sequence of branches
• A path is a sequence of edges
![Page 78: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/78.jpg)
Examples: number of paths needed
for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
![Page 79: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/79.jpg)
Examples: number of paths needed
for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
5 16
![Page 80: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/80.jpg)
Path Coverage-1• Every distinct path through code
is executed at least once
• Example1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’
• Test Paths:1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,1, 2, 3, 6, J1, 7, 9, J2
1,2,3
4,5
Join1
6
7
8
Join2
9
![Page 81: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/81.jpg)
Counting Paths
• It is not feasible to calculate the total number of paths
![Page 82: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/82.jpg)
Linearly independent paths
• It is feasible to calculate the number of linearly independent paths
• The number of linearly independent paths is the number of end-to-end paths required to touch every path segment at least once
• A linearly independent path introduces at least one new set of process statements or a new condition
![Page 83: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/83.jpg)
Cyclomatic Complexity• Software metric for the logical complexity of a
program.• Defines the number of independent paths in the
basis set of a program• Provides the upper bound for the number of tests
that must be conducted to ensure that all statements been have executed at least once
• For Edges (E) and Nodes (N) V(G) = E – N + 2
![Page 84: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/84.jpg)
Examples: Complexity of CFGs
1
2
3
1 3,4
5 6
Join
3,4
5
Join
N=3E=2E-N+2= 2-3+2= 1=V(G)
N=1E=0E-N+2= 0-1+2= 1=V(G)
N=4E=4E-N+2= 4-4+2= 2=V(G)
N=3E=3E-N+2= 3-3+2= 2=V(G)
![Page 85: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/85.jpg)
Example1
2,3
6
10
8
4,5
9
7
11
V(G) = 11 – 9 + 2 = 4
Independent paths:1-111-2-3-4-5-10-1-111-2-3-6-8-9-10-1…-111-2-3-6-7-9-10-1…-11
![Page 86: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/86.jpg)
In Class: Compute the cyclomatic complexity
3
4,5
6
Join
9
7,8
1,2
1
2
3
Join
5
6
1,2
Join
3,4
5 6
Join
7
1, 2
3
Join
4
5
6
![Page 87: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/87.jpg)
Example 1: CFG
3
4,5
6
Join
9
7,8
1,2N=7E=8E-N+2= 8-7+2= 3=V(G)
![Page 88: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/88.jpg)
Example 21
2
3
Join
5
6
N=6E=8E-N+2= 8-6+2= 4=V(G)
![Page 89: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/89.jpg)
Answers
1,2
Join
3,4
5 6
Join
7
N=7E=8E-N+2= 8-7+2= 3=V(G)
![Page 90: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/90.jpg)
Answers
1, 2
3
Join
4
5
6
N=6E=7E-N+2= 7-6+2= 3=V(G)
![Page 91: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/91.jpg)
Independent Path Coverage• Basis set does not yield minimal
test set• Example
1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’
• Cyclomatic complexity: 3• Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,
1,2,3
4,5
Join1
6
7
8
Join2
9
![Page 92: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/92.jpg)
Def-Use Coverage• Def-use coverage: every path
from every definition of every variable to every use of that definition is exercised in some test.
• Example1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’
1,2,3
4,5
Join
6
7
8
Join
9
Def: x, zUse: x
Def: y, xUse: x, z
Use: none
Use: y
Use: yUse: none
Test Path: 1, 2, 3, 4, 5, 7, 8, J
![Page 93: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/93.jpg)
Strength of Coverage
Statement
Branch
Def-Use
Path
Arrows point from weaker to stronger coverage.
Stronger coverage requires more test cases.
Condition
![Page 94: Testing in the Small (aka Unit Testing, Class Testing) 1209](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bffc1a28abf838cc1a7c/html5/thumbnails/94.jpg)
What paths don’t tell you
• Timing errors
• Unanticipated error conditions
• User interface inconsistency (or anything else)
• Configuration errors
• Capacity errors