automatic program repair with evolutionary computation westley weimer computer science dept....
TRANSCRIPT
Automatic Program Repair With Evolutionary ComputationWestley WeimerComputer Science Dept.
University of Virginia
Charlottesville, VA 22904
Stephanie ForrestDept. of Computer Science
University of New Mexico
Albuquerque, NM 87131
Claire Le GouesComputer Science Dept.
University of Virginia
Charlottesville, VA 22904
ThanhVu NguyenDept. of Computer Science
University of New Mexico
Albuquerque, NM 87131
Presented by:
Teodoro RosatiCIS 601, Spring 2014
March 4, 2014
The material in this paper is taken from two original publications, titled “A Genetic Programming Approach to Automated Software Repair” (Genetic and Evolutionary Computation Conference, 2009) and “Automatically Finding Patches Using Genetic Programming” (Proceedings of the 2009 IEEE 31st International Conference on Software Engineering, IEEE Computer Society).
What’s the Problem?Finding bugs is relatively easy…
• Famous Costly Bugs– FDIV Intel Pentium processor (1994): $500 million
– floating point unit had a flawed division table
– Y2K (1999). Cost: $500 billion– 2 digit YR storage (e.g. 95) and “00” = 1900
– Mars Climate Crasher (1998). Cost: $125 million– Imperial Units (lbs of force) Metric Units (Newtons)
• Many techniques and software solutions for detecting and mitigating software errors:– Syntactic Bug Pattern Detection– Decompilation and Data Flow Analysis– Automated Theorem Proving– Model Checking
2
I Found The Bug, Here’s the catch…now I have to fix it
3
One solution…
Manual Program Repair• Up to 90% of total cost of a software project for
maintenance after delivery – Modifying existing code, repairing defects,
maintaining code during its lifecycle
4
– Products often are shipped with known and unknown bugs because lack of development resources
A better solution…
Automatic Program Repair:
5
Traditional Program Analysis MethodsTraditional Program Analysis Methods
Evolutionary Computation (Genetic Programming)Evolutionary Computation (Genetic Programming)
Experimental Design
6
• Genetic Programming • Evolves computer programs tailored to a task• Meaning a program is modified using similar
pathways to genetic evolution (mutation/crossover)
• GP techniques have been applied to unannotated off-the-shelf legacy C programs• Individual variants V with the highest fitness are
selected for continued evolution
7
Evolutionary Computation Genetic Programming
• Inspired by biological natural selection– Endler’s guppy experiment:
• Diverse source population– Guppies variously colored
• Natural selection on population– Habitat variation, coarse vs. fine gravel– Predator presence, sight based
– Evolution of drug resistant bacteria• Source = normal + resistant bacteria
• Antibiotics promote resistant bacteria
8
Evolutionary Computation Genetic Programming• Natural selection and programs
– Population of Program Variants• Variants/Individuals
Genetic Programming Repair Technical Approach1. What is it doing wrong?
– Input a set of negative (-) test cases that characterizes a fault
2. What is it supposed to do?– Input a set of positive (+) test cases that encode
functionality requirements
3. Where should we change it?– Program locations of the (-) test cases
4. How should we change it?– Insert, delete and swap program statements and control
flow. Insertions are preferred
5. When are we finished?– First variant that passes (+) and (-) cases– Minimize differences between variant and original
9
10
Automatic Program Repair Representation• Abstract Syntax Tree (AST)
– C programs• Genes: Statements are basic units
– Conditional “if (x>y) {max = x;}”
• Expressions within a statement– “{max = x;}”
– Selection Actions:
Insert, Delete, Swap of Genes
____________________________
If ( x > y )
{
max = x;
}
If
x y
>
max x
=Gene
AST
Genetic ProgrammingMutation Operators
Mutation Operator– Insert, Delete or Swap a gene
11
Genetic ProgrammingCrossover Operators
Crossover Operator– Between 2 Parent Sub-Trees
Crossback Operator– Between Variant V and Parent
Measuring Fitness
13
• Variants are compiled• Testcase evaluated in virtual machine/sandbox• Fitness measured using formula:
fitness(P) = WPosT { t PosT P passes t } +
WNegT { t NegT P passes t }
Note:WPosT = weight of each successful positive test
WNegT = weight of each successful negative test
For example…
14
• December 31, 2008– A bug was reported in Microsoft Zune
media players• Zune would freeze when the value of the
input days is the last day of a leap year (e.g. 10,593) INFINITE LOOP!
1. What is it doing wrong? Negative Test Cases
• Negative Test Case– input days set to 10,593 (last day of leap year) – program executes lines 1 – 16– then repeats lines 3, 4, 8 and 11 infinitely
2. What’s it supposed to do? Positive Test Cases
• Positive Test Case– input days set to 1,000 (non-leap year)– program executes lines 1–8, 11-18 once as
expected
3. Where should we change it?
• Program Locations visited when executing the negative test cases– lines 3, 4, 8 and 11
4. How should we change it? INSERT
• Insert an entire statement or gene– stmtj is added to stmti
4. How should we change it? DELETE
• Delete an entire statement or gene– stmti is transformed into an
empty block statement
4. How should we change it? SWAP
• Swap an entire statement or gene– Second statement stmtj is
chosen uniformly at random from anywhere in the program to replace stmti
5. When we are finished? Minimize Differences
21
• Variant V program passes all the test cases– Minimization Step to discard unnecessary changes– Average repair time = 42 seconds
Performance of the Zunebug Repair…
22
• Evolution of the Zunebug repair 1 GP trial – The darker curve plots the average fitness of the population
– The lighter curve plots the fitness of the individual V primary repair
Performance of the Zunebug Repair…
23
• Evolution of the Zunebug repair with 20 positive and 4 negative test cases (equally weighted) – The boxes represented the average over 70 distinct trials
– The error bars represent one standard deviation.
Performance of the GP Algorithm…
24
Eleven defects repaired by Genetic Programming
Performance of the GP Algorithm…
25
• GP search time scales with weighted path size.– 18 programs successfully repaired by GP (ave. of 100 runs)
– x-axis log10 of the weighted path length
– y-axis log10 of the total number of fitness evaluations
Caveats…
26
• Limitations (assumptions)• defect is reproducible
• program behaves deterministically on test cases
• postive test cases encode program requirements
• no overlap in path taken by negative and postive test cases
• existing program can provide repair statements
• Evolution• Not rigorously tested (parameter values, selection strategies,
and operator design)
• Fault Localization • critical to find viable fixes, but poorly understood
• Fitness Function• Oversimplification
• Repair Quality• dependent on a high-quality set of positive test cases
Future work…
27
• Generic set of repair templates for GP as source code for mutations
• Extend with data structure definitions and variable declarations
• Assembly- and bytecode-level repairs• Testing on more sophisticated errors
– Race conditions
• Assessing size and distribution of bugs for targeting
