software testing part ii march, 2001. fault-based testing methodology (white-box) 2 mutation testing
TRANSCRIPT
Software TestingPart II
March, 2001
Fault-based Testing Methodology(white-box)
2
Mutation Testing
3
Klaidos įvedimas
Automatinis programos testavimas Klaidos įvedimas Automatinio testavimo pakartojimas
4
Strong mutation testing Called “fault-based” because it’s directed
at the possible faults that can occur in a program
Provides a measure of test-data adequacy The technique begins by generating test
data by some means; it does not really matter how it is generated (can be randomly generated)
5
Strong mutation testing (cont)
Approach: generate variations of the original program, called “mutants”
The original test data is run through both the original program and the mutant
If there is a difference in the output, then the mutant is dead
6
Strong mutation testing (cont) If the mutant is not killed, then the test data is
not adequate to reveal the fault and distinguish the mutant from the original
If this is the case, then the test data must be augmented to reveal the fault and kill the live mutant
The obvious problem: too many simple faults that can be introduced, O(n^2) for an n line program
7
List of possible faults, Budd (1983)
constant replacement scalar replacement scalar variable for
constant replacement constant for variable array reference for
constant replacement array reference for
scalar variable
constant for array reference replacement
scalar variable for array reference replace
array reference for array reference replace
constant replacement data statement
alteration
8
List of possible faults, Budd (1983)
comparable array name replacement
arithmetic operator replacement
relational operator replacement
logical connector replacement
absolute value insert unary operator insertion statement analysis statement deletion return statement replace goto label replace do statement end
replace
9
How effective is mutation testing?
“Test data that distinguishes all programs differing from a correct one by only simple errors is so sensitive that it also finds complex errors”, DeMillo (1978)
mutation testing makes the “competent programmer” assumption: “programmers write programs that are nearly correct”!
10
Example program:
string searching program: the program prompts the user for a positive integer in the range 1 to 20 and then for a string of characters of that length. The program then prompts for a character and returns the position in the string at which the character was first found or a message indicating that the character was not present in the string. The user has the option to search for more characters.
11
main() {(1 ) char a[20], ch, response = ‘y’;(2) int x, i;(3) bool found;(4) cout << “Input an integer between 1 and 20: “;(5) cin >> x;(6) while ( x < 1 || x > 20) {(7) cout << “Input an integer between 1 and 20: “;(8) cin >> x; }(9) cout << “input “ << x << “ characters: “;(10) for (int i = 0; i < x; ++i) cin >> a[i];(11) cout << endl;
12
(12) while ( response == ‘y’ || response == ‘Y’) {(13) cout << “input character to search: “;(14) cin >> ch;(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;(19) if (found) cout << ch << “ at: “ << i << endl;(20) else cout << ch << “ not in string” << endl;(21) cout << “search for another character? [y/n]”;(22) cin >> response; }
13
mutation testing string example
Assume we have branch tested the program
consider lines 15 to 18
(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;
14
Test data for branch testing
x input string search char response expectedoutput
34 wrong input3 abc c pos = 3
yk not in string
n
15
RIP Since the output of the mutant program
differs from the output of the original program, mutant #1 is dead!
It is important to make only one small change so that faults don’t cancel each other
16
mutation testing string example
Assume we have branch tested the program,
create another mutant:(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;
17
Mutant #2 lives!
The output of the mutant is the same as the original program ==> mutant #2 is alive
Analysis of the program reveals that the test data is not adequate to reveal the fault; add a test case to search first position
18
RIP
x input string search char response expectedoutput
34 wrong input3 abc c pos = 3
ya pos = 1
yk not in string
n
19
Super mutants!
Sometimes it’s impossible to kill a mutant because no test data can be found that will reveal the newly introduced fault
an absurd example:
main() { int x = 3; int y = x*3; if (y < 10) cout << “less”;}
20
Suppose the string processing program had contained a fault!
(15) found = false;(16) i = 1;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;
21
Strengths of strong mutation testing
Shows the absence of particular faults; contrary to Dijkstra’s comment! By introducing a fault into the program and then showing that the test data reveals it, we know that the fault cannot be there!
Forces the programmer to scrutinize ( kruopščiai ištirti) and analyze the program under test to think of test data to expose the newly introduced fault, i.e., kill the mutant!
22
Weaknesses of strong mutation testing
Computationally expensive! Huge number of mutants can be
generated There are systems that can help
generate mutant programs and test data; but much work must be done by hand.
23
Weak mutation testing Less demanding that strong mutation take some construct of the program and
create some mutants of this construct consider: if (A && B || C), possible mutants:
if (A || B || C) if (A && B && C) if ((A && B) || C) etc.
24
Weak mutation testing (cont) Howden (1981) identified five candidate constructs:
data access (or variable reference) data storage (or variable assignment) arithmetic expression arithmetic relation boolean expression: same as multiple condition
coverage Howden developed reliable tests for the five
constructs
25
Consider data access or variable reference
We’re considering the r-value here. Aim is to ensure that the correct variable is
being accessed. consider the variables in string processing
program, type-by-type integer variables are referenced on lines 9, 10, 17,
19 and 20; make sure we have adequate test data to ensure that they are getting correct values
bool variable is referenced on line 15
26
Consider arithmetic relations:
they occur on lines 6 and 17; test data that ensures that x has values 0, 1, 2, 1, 20 and 21 would test x and i
27
Strengths of weak mutation
Like strong mutation testing, it promotes a thorough analysis of the program under test
computationally cheaper not necessary to physically generate the
mutants
28
weaknesses of weak mutation
Unlike strong mutation, it is not reliable for the program as a whole; adequacy of the data is local to the construct being mutated
Girgis and Woodward (1986) found that weak mutation was outperformed by data flow testing; they also found that different faults were found by each technique and promoted a combination of testing techniques