![Page 1: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/1.jpg)
Software Testing
and Reliability
Southern Methodist University
CSE 7314
![Page 2: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/2.jpg)
“A working program remains an elusive thing of beauty”
Robert Dunn
![Page 3: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/3.jpg)
Syllabus
• Instructor; Rob Oshana
• Office hours: By appointment
• Phone; (281) 274-3211
• Fax; (214) 768-3085
• E-mail; [email protected]• Web site;
www.engr.smu.edu/cse/roshana/cse7314
![Page 4: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/4.jpg)
Syllabus
• Required Text Book: Systematic Software Testing, by Rick Craig and Stefan Jaskiel, Artech House, ISBN 1-58053-508-9
• Practical Guide to Testing Object-Oriented Software by David A. Sykes, John D. McGregor Addison-Wesley Pub Co; ISBN: 0201325640
![Page 5: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/5.jpg)
Syllabus
• Student Evaluation; The course grade will be computed as follows:–Midterm Exam 30%– Final Exam 30%– Homework 15%– Project 25%
![Page 6: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/6.jpg)
Course Outcomes
• Upon successful completion of this course the student will be able to:
• 1. Determine the test techniques applicable to a given program
• 2. Construct a test suite using the techniques discussed in class
![Page 7: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/7.jpg)
Course Outcomes
• 3. Determine various test and quality metrics of a program
• 4. Create and manage an effective software testing team
![Page 8: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/8.jpg)
Course is / is not
• Is a roadmap approach for test professionals
• Is not an implementation course
• Is not a software testing tools course
![Page 9: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/9.jpg)
Outline
Trip Topics Readings
1 Overview of the testing process
Risk analysis
Master test planning
Detailed test planning
Craig, Chapters 1-4
2 Analysis and design
Test implementation
Test execution
Craig, Chapters 5-7
3 Test organization
The software tester
The test manager
Improving the process
Craig, chapters 8-11
![Page 10: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/10.jpg)
Outline
Trip Topics Readings
4 Statistical testing techniques
Testing OO systems
Notes
Sykes, Chapters 1-6
5 Testing OO systems Sykes, Chapters 7-10
6 Testing RT systems
Testing safety critical systems
Testing web based systems
Notes
![Page 11: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/11.jpg)
Outline
• Selected readings will be sent to the students on a periodic basis
• Homework; assignments/schedule will be posted on the web site shortly
• Project will be discussed next trip
![Page 12: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/12.jpg)
Testing style
Competence Test question cues
Knowledge List, describe
Comprehension Summarize, discuss, describe
Evaluation Explain, compare
Analysis Analyze, explain, compare
![Page 13: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/13.jpg)
“Errors are more common, more pervasive, and more troublesome in software than with other technologies”
David Parnas
![Page 14: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/14.jpg)
Homework 1a
• Please send me a couple paragraphs describing your background and experience. Describe to me what you want to get out of the course
![Page 15: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/15.jpg)
Homework 1a
• Please read the paper entitled “Improving Software Testability”
• www.stlabs.com/newsletters/testnet/docs/testability.htm
![Page 16: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/16.jpg)
CSE 7314
Software Testing and Reliability
![Page 17: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/17.jpg)
What is testing?
• How does testing software compare with testing students?
![Page 18: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/18.jpg)
What is testing?
• “Software testing is the process of comparing the invisible to the ambiguous as to avoid the unthinkable.” James Bach, Borland corp.
![Page 19: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/19.jpg)
What is testing?
• Software testing is the process of predicting the behavior of a product and comparing that prediction to the actual results." R. Vanderwall
![Page 20: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/20.jpg)
Purpose of testing
• Build confidence in the product
• Judge the quality of the product
• Find bugs
![Page 21: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/21.jpg)
Finding bugs can be difficult
x
x
x
x
x
x
x
x
x
x
x
Mine field
A path through themine field (use case) A path through the
mine field (use case)
![Page 22: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/22.jpg)
Why is testing important?
• Therac25: Cost 6 lives
• Ariane 5 Rocket: Cost $500M
• Denver Airport: Cost $360M
• Mars missions, orbital explorer & polar lander: Cost $300M
![Page 23: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/23.jpg)
Why is testing so hard?
![Page 24: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/24.jpg)
Reasons for customer reported bugs
• User executed untested code• Order in which statements were
executed in actual use different from that during testing
• User applied a combination of untested input values
• User’s operating environment was never tested
![Page 25: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/25.jpg)
Interfaces to your software
• Human interfaces
• Software interfaces (APIs)
• File system interfaces
• Communication interfaces–Physical devices (device drivers)
–controllers
![Page 26: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/26.jpg)
Selecting test scenarios
• Execution path criteria (control)–Statement coverage–Branching coverage
• Data flow – Initialize each data structure–Use each data structure
• Operational profile • Statistical sampling….
![Page 27: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/27.jpg)
What is a bug?
• Error: mistake made in translation or interpretation ( many taxonomies exist to describe errors)
• Fault: manifestation of the error in implementation (very nebulous)
• Failure: observable deviation in behavior of the system
![Page 28: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/28.jpg)
Example
• Requirement: “print the speed, defined as distance divided by time”
• Code: s = d/t; print s
![Page 29: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/29.jpg)
Example
• Error; I forgot to account for t = 0
• Fault: omission of code to catch t=0
• Failure: exception is thrown
![Page 30: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/30.jpg)
Severity taxonomy
• Mild - trivial
• Annoying - minor
• Serious - major
• Catastrophic - Critical
• Infectious - run for the hills
What is your taxonomy ?
IEEE 1044-1993
![Page 31: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/31.jpg)
Life cycle
Requirements
Design
Code
Testing
error
error
error
Errors can be introduced ateach of these stages
Resolve
Isolate
Classify
error
error
error
error
Testing and repair process can bejust as error prone as the developmentProcess (more so ??)
![Page 32: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/32.jpg)
Ok, so lets just design our systems with “testability” in
mind…..
![Page 33: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/33.jpg)
Testability
• How easily a computer program can be tested (Bach)
• We can relate this to “design for testability” techniques applied in hardware systems
![Page 34: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/34.jpg)
JTAG
A standard Integrated Circuit
CoreIC
Logic
Test access portcontroller
Test mode Select (TMS)
Test clock (TCK)
Test data out (TDO)
Test data in (TDI)
BoundaryScan cells
BoundaryScan path
I/O pads
Data in
Data out
TDI TDOcell
![Page 35: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/35.jpg)
Operability
• “The better it works, the more efficiently it can be tested”–System has few bugs (bugs add
analysis and reporting overhead)–No bugs block execution of tests–Product evolves in functional
stages (simultaneous development and testing)
![Page 36: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/36.jpg)
Observability
• “What you see is what you get”– Distinct output is generated for each
input– System states and variables are visible
and queriable during execution– Past system states are ….. (transaction
logs)– All factors affecting output are visible
![Page 37: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/37.jpg)
Observability
– Incorrect output is easily identified
– Internal errors are automatically detected through self-testing mechanisms
– Internal errors are automatically reported
–Source code is accessible
![Page 38: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/38.jpg)
Visibility Spectrum
DSPvisibility
GPPvisibility
Factoryvisibility
End customervisibility
![Page 39: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/39.jpg)
Controllability
• “The better we can control the software, the more the testing can be automated and optimized”–All possible outputs can be
generated through some combination of input–All code is executable through some
combination of input
![Page 40: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/40.jpg)
Controllability
–SW and HW states and variables can be controlled directly by the test engineer
– Input and output formats are consistent and structured
![Page 41: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/41.jpg)
Decomposability
• “By controlling the scope of testing, we can more quickly isolate problems and perform smarter testing”–The software system is built from
independent modules–Software modules can be tested
independently
![Page 42: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/42.jpg)
Simplicity
• “The less there is to test, the more quickly we can test it”–Functional simplicity (feature set is
minimum necessary to meet requirements)–Structural simplicity (architecture
is modularized)–Code simplicity (coding standards)
![Page 43: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/43.jpg)
Stability
• “The fewer the changes, the fewer the disruptions to testing”–Changes to the software are
infrequent, controlled, and do not invalidate existing tests
–Software recovers well from failures
![Page 44: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/44.jpg)
Understandability
• “The more information we have, the smarter we will test”– Design is well understood– Dependencies between external,
internal, and shared components are well understood
– Technical documentation is accessible, well organized, specific and detailed, and accurate
![Page 45: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/45.jpg)
“Bugs lurk in corners and congregate at boundaries”
Boris Beizer
![Page 46: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/46.jpg)
Types of errors
• What is a Testing error?–Claiming behavior is erroneous
when it is in fact correct
– ‘fixing’ this type of error actually breaks the product
![Page 47: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/47.jpg)
Errors in classification
• What is a Classification error ?–Classifying the error into the
wrong category
• Why is this bad ?–This puts you on the wrong path
for a solution
![Page 48: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/48.jpg)
Example Bug Report
• “Screen locks up for 10 seconds after ‘submit’ button is pressed”
• Classification 1: Usability Error • Solution may be to catch user events
and present an hour-glass icon• Classification 2: Performance error• solution may be a modification to a sort
algorithm (or visa-versa)
![Page 49: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/49.jpg)
Isolation error
• Incorrectly isolating the erroneous modules• Example: consider a client server
architecture. An improperly formed client request results in an improperly formed server response
• The isolation determined (incorrectly) that the server was at fault and was changed
• Resulted in regression failure for other clients
![Page 50: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/50.jpg)
Resolve errors
• Modifications to remediate the failure are themselves erroneous
• Example: Fixing one fault may introduce another
![Page 51: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/51.jpg)
What is the ideal test case?
• Run one test whose output is "Modify line n of module i."
• Run one test whose output is "Input Vector v produces the wrong output"
• Run one test whose output is "The program has a bug" (Useless, we know this)
![Page 52: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/52.jpg)
More realistic test case
• One input vector and expected output vector– A collection of these make of a Test
Suite
• Typical (naïve) Test Case– Type or select a few inputs and observe
output– Inputs not selected systematically– Outputs not predicted in advance
![Page 53: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/53.jpg)
Test case definition
• A test case consists of;– an input vector– a set of environmental conditions– an expected output.
• A test suite is a set of test cases chosen to meet some criteria (e.g. Regression)
• A test set is any set of test cases
![Page 54: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/54.jpg)
Requirements as theory model
• Suppose we consider a specification to be a theory describing a program
• How do we test theories?• By examining the theory and using it
to make predictions• First Principle of testing. The
expected results of a test should be known before the test is run
![Page 55: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/55.jpg)
Requirements as theory
• "Accumulating evidence to support a theory is not the appropriate way to test it. What you should do is try to falsify it, to challenge it with your best efforts at proving it false."–Karl Popper
![Page 56: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/56.jpg)
Requirements as theory
• Implications for us doing testing• Testing should not be used to
build confidence, to easy• Testing should attempt to find
deviations from the theory, that is, bugs
• Any other purpose sets up the wrong goal
![Page 57: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/57.jpg)
Requirements as theory
• "Program testing can be used to show the presence of bugs, but never show their absence!" O.-J. Dahl, E. W. Dijkstra, and C.A.R. Hoare, Structured Programming, New York: Academic, 1972.
• "Absence of proof (of bugs) is not proof of absence."; Logic 101
![Page 58: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/58.jpg)
A few words about computability
• From the theory of computability, we know:
• It is undecidable whether a given program will halt on a given input. (Halting problem)
• It is undecidable whether two programs will always output the same answer for a given input. (Equivalence)
![Page 59: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/59.jpg)
Implications for testing
• There is no general solution for the automated oracle problem–no automatic testing strategy can be
devised that will work in all cases
• There is no general way to find the input that causes a specific line of code to be executed
• Coverage is undecidable
![Page 60: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/60.jpg)
All of the following are undecidable
• Will a given statement ever be exercised by any input?
• What input will exercise a given statement?• Will a given input exercise some specified
statement?• Will a given path ever be exercised by any
input?• What input will exercise a given path?• Will a given input exercise some specified
path?
![Page 61: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/61.jpg)
Computability
• Note that even though it is in general undecidable, there is a large class of programs for which these issues can be decided– a large testing tools industry has
emerged because of this
• When examining a tool, make sure that the class of programs for which it works is well understood
![Page 62: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/62.jpg)
Reference book
![Page 63: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/63.jpg)
A few words on combinatorics
• Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |
![Page 64: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/64.jpg)
Example
• Assume a program has a single input, Customer ID (CID)
• May be any value in the domain {00000-99999}
• What is | I | ?
• | I | = 100000
![Page 65: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/65.jpg)
Example
• Now assume we add a second input, the Order ID (OID)
• This may be any value in the domain {00000-99999} as well
• Now what is | I |?
• 100000*100000 = 10,000,000,000
![Page 66: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/66.jpg)
Example
• Finally, add a credit card number to the input
• This is a 12 digit number• | I | has now reached 10**22
• If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!
![Page 67: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/67.jpg)
Example
• Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing• This attempt is doomed to fail
due to the combinatorial explosion
![Page 68: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/68.jpg)
Functional and structural approaches to testing
![Page 69: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/69.jpg)
Engineering the testing process
• Any engineered product (and most other things) can be tested in one of two ways–Knowing the specified function that
a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function
![Page 70: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/70.jpg)
Engineering the testing process
–Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)
![Page 71: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/71.jpg)
Structural testing
• Uses knowledge of the internal workings
• Also known as Clear box/glass box• Code based• Can be useful for finding interesting
inputs• Misses an entire class of faults,
missing code
![Page 72: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/72.jpg)
Behavioral
• Uses knowledge of the specific function that is to be performed
• Based solely on the specification without regard for the internals
• Also known as Black box• More user oriented• Misses an entire class of faults, extra
code (surprises) except by accident
![Page 73: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/73.jpg)
Passing criteria
• How do we know when
• 1. a single test has passed
• 2. when we are done testing
![Page 74: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/74.jpg)
Passing criteria
• A single test passes when its output is correct–This requires a specific definition
of correct and ties into the automated oracle problem
![Page 75: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/75.jpg)
When are we done?
• Conway Criteria:• No syntactic errors (it compiles)• No compile errors or immediate
execution failures• There exists Some set of data for
which the program gives the correct output• A typical set of data produces the
correct output
![Page 76: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/76.jpg)
When are we done?
• Difficult sets of data produce the correct output.
• All possible data sets in the problem specification produce the correct output
• All possible data sets and likely erroneous input succeeds.
• All inputs produce the correct output
![Page 77: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/77.jpg)
Nature of software defects
• Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed
![Page 78: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/78.jpg)
Nature of software defects
• We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis
• Typographical errors are random
More of a case for WHITE box testing……
![Page 79: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/79.jpg)
Summary
• Zeroth Principle of Testing; The purpose of testing is to find bugs
• Corollary to the zeroth principle; "The program is wrong"
![Page 80: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/80.jpg)
Summary
• First Principle of Testing; The results of a test must be known before the test is run
• Second Principle of Testing: Testing is difficult
![Page 81: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/81.jpg)
Summary
• Exhaustive testing is doomed by the combinatorial explosion
• Any other technique is undecidable
• Third Principle of Testing: No single technique will suffice for any non-trivial testing effort
![Page 82: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/82.jpg)
Homework 1b
• Discuss a software failure from your experience or knowledge and attempt to explain the role of testing (or lack thereof) in that failure
![Page 83: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/83.jpg)
Another reference
• Testing techniques newsletter
• www.testworks.com/News/TTN-Online
![Page 84: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/84.jpg)
More on coverage
![Page 85: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/85.jpg)
Logical coverage
• What does statement coverage imply ?
• Each statement must be executed at least once
![Page 86: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/86.jpg)
Example
• 1 IF (( X > 1) AND ( Y == 1))
• 2 Z = Z / X
• 3 END
• 4 IF ((X == 2) OR ( Z >1))
• 5 Z = Z + 1
• 6 END
What is required for statement coverage ?
![Page 87: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/87.jpg)
Example
• To achieve statement coverage, one can choose x = 2, y = 1
• Many possible errors are missed:
• 1 IF ((X > 1) OR (Y ==1))• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y >=1))
![Page 88: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/88.jpg)
Blindness
• Statement coverage is blind to several classes of error when choosing values to achieve coverage
![Page 89: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/89.jpg)
Assignment blindness
• Due to assignment of a particular value to a variable, the error does not propagate
![Page 90: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/90.jpg)
Equality blindness
• Due to an Equality check of a particular variable, the error does not propagate
![Page 91: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/91.jpg)
Self blindness
• Conditional itself covers up the error
![Page 92: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/92.jpg)
Assignment Blindness example
• 1 IF (<conditional>)• 2 X = 1• 3 END• 4 IF (X + Y > 1) //Should have
been (Y > 0)• 5 ....• 6 END
![Page 93: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/93.jpg)
Assignment Blindness example
• In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen
![Page 94: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/94.jpg)
Assignment Blindness example
• If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in
statement 1 always evaluated to true
![Page 95: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/95.jpg)
Equality Blindness Example
• 1 IF (A == 2)
• 2 .....
• 3 IF (B > 1) // Should have been (A + B > 3)
![Page 96: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/96.jpg)
Equality Blindness Example
• In this example, the value of 2 is chosen for A to force execution of the body
• The error in statement 3 is missed
![Page 97: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/97.jpg)
Self Blindness
• 1 IF (X < 1) // Should have been (2X < 1)
• In this example, the value of 0 is chosen for X
![Page 98: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/98.jpg)
Observation
• Statement coverage is the weakest form of coverage
• Many classes of errors can escape this testing
• Typical projects have less than 80% statement coverage!!
![Page 99: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/99.jpg)
Branch coverage
• Each branch of a decision must be executed at least once
• This is stronger that statement coverage, but it is still weak
• Note: A decision is a logical combination of one or more conditions
![Page 100: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/100.jpg)
Branch coverage example
• 1 IF (( X > 1) AND ( Y == 1))
• 2 Z = Z / X
• 3 END
• 4 IF ((X == 2) OR ( Z >1))
• 5 Z = Z + 1
• 6 END
![Page 101: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/101.jpg)
Branch coverage example
• To achieve branch coverage, one option is to choose–x = 2, y = 1
–x = 0, y = 2
![Page 102: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/102.jpg)
Branch coverage example
• Many possible errors are still missed:
• 1 IF ((X > 1) OR ( Y ==1)
• 1 IF ((X >=1) AND (Y ==1))
• 1 IF ((X > 1) AND (Y <=1))
![Page 103: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/103.jpg)
Blindness in branch coverage
• Branch coverage is blind to several classes of error when choosing values to achieve coverage–Compound decisions (more than
one condition) are weakly tested–Boundaries of conditions (within a
decision) are not explicitly tested
![Page 104: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/104.jpg)
Condition coverage
• Each condition within a decision must assume all possible values
• This is 'potentially' stronger than branch coverage, but not always
• It may, in fact, be weaker
![Page 105: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/105.jpg)
Example of condition coverage
• 1 IF (( X > 1) AND ( Y == 1))
• 2 Z = Z / X
• 3 END
• 4 IF ((X == 2) OR ( Z >1))
• 5 Z = Z + 1
• 6 END
![Page 106: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/106.jpg)
Example of condition coverage
• To achieve condition coverage for statement 1, one must choose values such that:
• X > 1 ~(Y == 1)
• ~(X > 1) Y == 1
![Page 107: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/107.jpg)
Example of condition coverage
• Both of these vectors miss the execution of statement 2
• A better choice may be:
• ~(X > 1) ~(Y == 1)
• X > 1 Y == 1
![Page 108: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/108.jpg)
Multi-condition coverage
• Every combination of possible condition values within a decision must be chosen
• This is strictly stronger than branch or condition coverage–still has weaknesses!
![Page 109: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/109.jpg)
Multi-condition coverage example
• 1 IF (( X > 1) AND ( Y == 1))
• 2 Z = Z / X
• 3 END
• 4 IF ((X == 2) OR ( Z >1))
• 5 Z = Z + 1
• 6 END
![Page 110: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/110.jpg)
Multi-condition coverage example
• To achieve multi-condition coverage for statement 1, one must choose values such that:
• 1. X > 1 Y == 1• 2. X > 1 ~(Y == 1)• 3. ~(X > 1) Y == 1• 4. ~(X > 1) ~(Y == 1)
![Page 111: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/111.jpg)
Multi-condition coverage example
• To achieve multi-condition coverage for statement 2, one must choose values such that:
• 5. X == 2 Z > 1• 6. X == 2 ~(Z > 1)• 7. ~(X == 2) Z > 1• 8. ~(X == 2) ~(Z > 1)
![Page 112: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/112.jpg)
Coverage
X Y Z Covers
1 0 1 1,5
2 1 4 4,8
1 1 3 2,6
2 0 1 3,7
![Page 113: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/113.jpg)
Multi-condition coverage example
• There are 9 paths through this code and this test set only covers 4 of those
![Page 114: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/114.jpg)
Multi-condition coverage example
• Consider this error
• Line 2 is incorrectly written as Z = Z –X
• Only the second vector will execute this line but for the values chosen, Z-X = Z/X
![Page 115: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/115.jpg)
Multi-condition coverage example
• Also consider that no vector will execute both line 2 and line 5
• In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5
![Page 116: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/116.jpg)
Multi-condition coverage example
• For instance:
• 2. Z = Z/X; A = 0
• ....
• 5. Z = Z + 1; B = 1/A;
![Page 117: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/117.jpg)
Path coverage
• Every possible path must be chosen
• Finding vectors to accomplish this is, in general, undecidable, and usually difficult
![Page 118: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/118.jpg)
Path coverage
• This still misses boundary conditions
• If statement 1 was mis-typed as X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error
![Page 119: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/119.jpg)
Backup
![Page 120: Software Testing and Reliability Southern Methodist University CSE 7314](https://reader035.vdocuments.us/reader035/viewer/2022081504/56649ea85503460f94babe7c/html5/thumbnails/120.jpg)
Homework • Suppose we have a routine r1 that sends its output to
routine r2, as in a unix pipe. Discuss the implications of R1 != D2, where R1 is the range of routine r1 and D2 is the domain of routine r2
• Suppose a program accepts as input, a 16 bit positive integer, i, and a single 8 bit character, c. The program has several faults. the first fault is the absence of 0 checking in a denominator. The parameter i is used as this denominator. There is another fault in that if the parameter c has a value between a and z, i.e. it is a lower case character, values for the parameter i less than 100 will fail. What is the probability that an input vector chosen at random will discover a fault?