software logic mutation testing presented by
TRANSCRIPT
Software Logic Mutation Testing
Presented by
Gary Kaminski
Overview
• Introduction and Motivation
• Logic Faults and Detection
• Case Study Comparison
• Context
• Conclusion
Logic Mutation Testing Process
1) Systematically create program variations (mutants) each containing a single typical logic fault
if (a && b) if (a || b)
2) For each mutant, find an input that distinguishes the original program from the mutant program (killing)
- a && b and a || b evaluate to different truth values
• Focus is on step 1)
Logic Mutation Testing
• Logic mutants are created by applying mutation operators to Boolean predicates
• Boolean predicates may come from:
1) UML Diagrams
2) Security Requirements
3) Looping constructs (while / do-while / for)
4) Preconditions for methods (if - then)
MotivationCurrent mutation systems:
– generate excess logic mutants– do not detect certain classes of logic faults
Goal:
- generate fewer logic mutants
- detect more logic faults
Assumption:
- formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates
Minimal DNF
• Terms separated by OR, literals by AND
ab + a!c vs. a(b + !c)
• Make each term true and other terms false
ab + ac vs. ab + abc
• Impossible to remove a literal or term without changing the predicate
ab vs. abc + ab!c
Excess Logic Mutants - Example
Original: ab + b!c + !bc
1) mutant generated multiple times: ab + _ + !bc
2) equivalent mutants: ac + b!c + !bc
3) mutants guaranteed to be killed by an input that kills some other mutant:
!ab + b!c + !bc and cb + b!c + !bc
4) highly effective mutation operators not used: abc + b!c + !bc
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Expression Negation Fault: !(ab + b!c + !bc)• Expression Stuck At Fault 0: FALSE• Expression Stuck At Fault 1: TRUE• Term Negation Fault: !(ab) + b!c + !bc• Term Omission Fault: ab + _ + !bc• Scalar Variable Replacement TOF: ab + c!c + !bc
• TOF generates same mutant twice: ab + _ + !bc
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Operator Reference Fault+: abb!c + !bc• Operator Reference Fault .: a + b + b!c + !bc• Operator Reference Fault+0: FALSE + !bc• Operator Reference Fault+1: TRUE + !bc• Operator Reference Fault.0: FALSE + b!c + !bc• Operator Reference Fault.1: TRUE + b!c + !bc
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Literal Negation Fault: !ab + b!c + !bc• Literal Stuck At Fault 0: FALSEb + b!c + !bc• Literal Stuck At Fault 1: TRUEb + b!c + !bc• Literal Omission Fault: _b + b!c + !bc• SVR LOF: bb + b!c + !bc
• LOF generates same mutant twice original: abc _bc, a_c, a_c, ab_
Minimal DNF Logic Faults
Original: ab + b!c + !bc
• Literal Reference Fault: ac + b!c + !bc • Literal Reference Fault: a!c + b!c + !bc• Literal Insertion Fault: abc + b!c + !bc• Literal Insertion Fault: ab!c + b!c + !bc
• LRF and LIF generate equivalent mutants
Logic Fault Minimal DNF Detection
Green Arrow – Detection may not holdBlack Arrow – Detection always holds
Red Box – Mutation Operator does not existBlue Box – Mutation Operator partially existsBlack Box – Mutation Operator exists
LOF
ORF.
LRF
LNF
TNF
LIF
TOF
LSTF0
ORF.0
ESTF0
ORF+
ORF+0
ENF
LSTF1
ORF.1
ORF+1
ESTF1
SVRTOF
SVRLOF
Logic Mutation Comparison
Original: ab + b!c + !bc• 61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs
• Equivalent LIF: abc + b!c + !bc• LRF: cb + b!c + !bc & ac + b!c + !bc• LOF: a_ + b!c + !bc
• Equivalent LIF: ab + b!c!a + !bc• LRFs: ab + !a!c + !bc & ab + b!a + !bc
• No Equivalent LIFs for term !bc• LOFs: ab + b!c + _c & ab + b!c + !b_
Mutation Efficiency Measures Mutation Efficiency Score = Faults Detected - Mutants GeneratedMutation Efficiency Ratio = Faults Detected / Mutants Generated• FD assumes an input is found to kill the mutant• FD includes equivalent mutants detected automatically
ab + b!c + !bc
Current Mutation Systems:• MES: 60 FD - 61 MG = -1• MER: 60 FD / 61 MG = .98
LIF-LRF-LOF• MES: 73 FD – 10 MG = 63 • MER: 73 FD / 10 MG = 7.30
Case Study
• Analyzed 19 Boolean predicates in an avionics software system
• Number of unique literals range: 5 to 13
• Compared MES and MER for current mutation systems vs. LIF-LRF-LOF
• Examined how MES and MER vary based on number of unique literals
Case Study Results
• LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819)
• Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082)
• Number of literals matters for MES but not for MER• MES = FD – MG and MER = FD / MG
Current LIF-LRF-LOF
MES 0 932MER 1.00 7.22
Mutation Efficiency Score (FD – MG) vs.
Number of Unique Literals
0
500
1000
1500
2000
2500
5 6 7 8 9 10 11 12 13
LIF-LRF-LOF
CurrentMutation
Mutation Efficiency Ratio (FD/MG) vs.
Number of Unique Literals
0
2
4
6
8
10
12
5 6 7 8 9 10 11 12 13
LIF-LRF-LOF
CurrentMutation
Context: Internal Variable Problem
• What input values kill a mutant?
• Mutation may be buried deep in the code
• Must reach predicate and have mutant predicate and original predicate evaluate to different truth values
• Partial solutions using constraints exist
Context: Minimal DNF in Practice
1) 95% of 20,256 Boolean predicates in avionics software were in minimal DNF*
2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates*
*Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages 14-21.
Context: Industry Audience
• LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals
• Exhaustive coverage for < 5 unique literals
• ab + b!c + bc (8 tests vs. 10 mutants)
• Avionics software*
*Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal, 9(5): 193-200, September 1994.
Related Work1) J.J. Chilenski. An Investigation of Three Forms of the Modified Condition
Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April 2001.
• Non-Minimal DNF based logic coverage criteria
2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, 1999. (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages 606-613.
• Minimal DNF based logic coverage criteria and Double Faults
3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): 411-424, October 1999.
• Seminal work in logic fault hierarchy
Conclusion
• Current logic mutation testing is inefficient• Excess mutants generated and faults are missed • Fault hierarchy approach based on minimal
DNF: detect more faults with less mutants• Applications in avionics software
Other Research and Publications
Logic Coverage Criteria:• Reduce test set size and maintain fault detection • Use feasibility of smaller, less expensive component criteria
Internal variable problem• CoJava and constraint programming
Publications:G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives of Logic Testing for Software. Submitted to Journal of Software Testing, Verification, and Reliability, January 2008. Online.