field failure reproduction using symbolic execution...
TRANSCRIPT
![Page 1: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/1.jpg)
FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND
GENETIC PROGRAMMING
Partially supported by: NSF, IBM, and MSR
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
![Page 2: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/2.jpg)
Partially supported by: NSF, IBM, and MSR
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
DSE SBST
FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND
GENETIC PROGRAMMING
![Page 3: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/3.jpg)
FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND
GENETIC PROGRAMMING
Partially supported by: NSF, IBM, and MSR
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
DSE SBST
![Page 4: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/4.jpg)
FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND
GENETIC PROGRAMMING
Partially supported by: NSF, IBM, and MSR
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Field failures are unavoidable!
DSE SBST
![Page 5: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/5.jpg)
FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND
GENETIC PROGRAMMING
Partially supported by: NSF, IBM, and MSR
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Field failures are unavoidable!
DSE SBST
![Page 6: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/6.jpg)
TYPICAL DEBUGGING PROCESS
Bug Repository
Very hard to(1) reproduce(2) debug
![Page 7: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/7.jpg)
TYPICAL DEBUGGING PROCESS
Bug Repository
Recent survey of Apache, Eclipse, and Mozilla developers:
Information on how to reproduce field failures is the most valuable, and difficult to obtain, piece of information for investigating such failures.[Zimmermann10]
Very hard to(1) reproduce(2) debug
![Page 8: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/8.jpg)
TYPICAL DEBUGGING PROCESS
Bug Repository
Recent survey of Apache, Eclipse, and Mozilla developers:
Information on how to reproduce field failures is the most valuable, and difficult to obtain, piece of information for investigating such failures.[Zimmermann10]
Very hard to(1) reproduce(2) debug
OVERARCHING GOAL: help developers (1) investigate field failures,
(2) understand their causes, and(3) eliminate such causes.
![Page 9: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/9.jpg)
OUR WORK SO FAR
Mimicking field failures [icse 2012, icst 2014]
Recording and replaying executions[icsm 2007, icse 2007]
Input anonymization [icse 2011]
Input minimization [woda 2006, icse 2007]
✘
Explaining field failures [issta 2013, TR]
![Page 10: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/10.jpg)
MIMICKING FIELD FAILURESUser run (R) Mimicked run (R’)
•F’ is analogous to F•R’ is an actual execution
F F’in the field in house
![Page 11: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/11.jpg)
MIMICKING FIELD FAILURESUser run (R) Relevant events
(breadcrumbs)Mimicked run (R’)
![Page 12: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/12.jpg)
OVERALL VISION
Crash report(execution data)Synthesized
ExecutionsField FailureReproduction
sed.c:8958 -> sed.c:8958
sed.c:8993 -> sed.c:9011
sed.c:8785 -> sed.c:8786
sed.c:8786 -> sed.c:8786
sed.c:990 -> sed.c:990
Likely faults Field FailureDebugging
InstrumentationSoftwaredeveloper Application
In house In the field
![Page 13: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/13.jpg)
BUGREDUX/SBFR
Crash report(execution data)Synthesized
ExecutionsField FailureReproduction
DSE SBST
![Page 14: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/14.jpg)
Crash report(execution data)
SynthesizedExecutions
BUGREDUXJoint work with Wei Jin
![Page 15: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/15.jpg)
Crash report(execution data)
BUGREDUX
Test Input
Joint work with Wei Jin
![Page 16: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/16.jpg)
Crash report(execution data)
Oracle
Candidateinput
Input generator
BUGREDUX
• Execution data
• Input generation technique
• Point of failure (POF)• Failure call stack• Call sequence• Complete trace
• Guided symbolic execution
Joint work with Wei Jin
Test Input
![Page 17: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/17.jpg)
Input icfg for P goals (list of code locations)Output If (candidate input)
Main algorithminit; currGoal = first(goals)repeat currState = SelNextState() if (!currState) backtrack or fail if (currState.cl == currGoal) if (currGoal == last(goals)) return solve(currState.pc) else currGoal = next(goals) currState.goal = currGoal symbolicallyExec(currState)
SelNextStateminDis = ∞retState = null
foreach state in statesSet if (state.goal = currGoal) if (state.cl can reach currGoal) d = |shortest path state.cl, currGoal| if d < minDis minDis = d retState = statereturn retState
ALGORITHM (SIMPLIFIED)
statesSet= {<cl, pc, ss, goal>}
![Page 18: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/18.jpg)
Input icfg for P goals (list of code locations)Output If (candidate input)
Main algorithminit; currGoal = first(goals)repeat currState = SelNextState() if (!currState) backtrack or fail if (currState.cl == currGoal) if (currGoal == last(goals)) return solve(currState.pc) else currGoal = next(goals) currState.goal = currGoal symbolicallyExec(currState)
SelNextStateminDis = ∞retState = null
foreach state in statesSet if (state.goal = currGoal) if (state.cl can reach currGoal) d = |shortest path state.cl, currGoal| if d < minDis minDis = d retState = statereturn retState
ALGORITHM (SIMPLIFIED)
statesSet= {<cl, pc, ss, goal>}
Optimizations/Heuristics
Dynamic tainting to reduce the symbolic input space
Program analysis information to prune the search space
Some randomness in the shortest path computation
![Page 19: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/19.jpg)
BUGREDUX EVALUATION – FAILURES CONSIDEREDName Repository Size(KLOC) # Faults
sed SIR 14 2grep SIR 10 1gzip SIR 5 2
ncompress BugBench 2 1polymorph BugBench 1 1
aeon exploit-db 3 1glftpd exploit-db 6 1htget exploit-db 3 1socat exploit-db 35 1tipxd exploit-db 7 1aspell exploit-db 0.5 1exim exploit-db 241 1rsync exploit-db 67 1xmail exploit-db 1 1
![Page 20: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/20.jpg)
BUGREDUX EVALUATION – FAILURES CONSIDEREDName Repository Size(KLOC) # Faults
sed SIR 14 2grep SIR 10 1gzip SIR 5 2
ncompress BugBench 2 1polymorph BugBench 1 1
aeon exploit-db 3 1glftpd exploit-db 6 1htget exploit-db 3 1socat exploit-db 35 1tipxd exploit-db 7 1aspell exploit-db 0.5 1exim exploit-db 241 1rsync exploit-db 67 1xmail exploit-db 1 1
None of these faults can be discovered by
a vanilla KLEE with a timeout of 72 hours
![Page 21: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/21.jpg)
BUGREDUX EVALUATION – RESULTSName POF Call Stack Call Seq. Compl. Tracesed #1sed #2grep
gzip #1gzip #2
ncompresspolymorph
aeonrsyncglftpdhtgetsocattipxdaspellxmailexim
One of three outcomes:✘: fail~: synthesize✔: (synthesize and) mimic
![Page 22: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/22.jpg)
BUGREDUX EVALUATION – RESULTSName POF Call Stack Call Seq. Compl. Tracesed #1 ✘ ✘ ✔ ✘
sed #2 ✘ ✘ ✔ ✘
grep ✘ ~ ✔ ✘
gzip #1 ✔ ✔ ✔ ✘
gzip #2 ~ ~ ✔ ✘
ncompress ✔ ✔ ✔ ✘
polymorph ✔ ✔ ✔ ✘
aeon ✔ ✔ ✔ ✔
rsync ✘ ✘ ✔ ✘
glftpd ✔ ✔ ✔ ✘
htget ~ ~ ✔ ✘
socat ✘ ✘ ✔ ✘
tipxd ✔ ✔ ✔ ✘
aspell ~ ~ ✔ ✘
xmail ✘ ✘ ✔ ✘
exim ✘ ✘ ✔ ✔
Synth.: 9/16Mimic: 6/16
Synth.: 10/16Mimic: 6/16
Synth.: 16/16Mimic: 16/16
Synth.: 2/16Mimic: 2/16
![Page 23: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/23.jpg)
BUGREDUX EVALUATION – RESULTSName POF Call Stack Call Seq. Compl. Tracesed #1 ✘ ✘ ✔ ✘
sed #2 ✘ ✘ ✔ ✘
grep ✘ ~ ✔ ✘
gzip #1 ✔ ✔ ✔ ✘
gzip #2 ~ ~ ✔ ✘
ncompress ✔ ✔ ✔ ✘
polymorph ✔ ✔ ✔ ✘
aeon ✔ ✔ ✔ ✔
rsync ✘ ✘ ✔ ✘
glftpd ✔ ✔ ✔ ✘
htget ~ ~ ✔ ✘
socat ✘ ✘ ✔ ✘
tipxd ✔ ✔ ✔ ✘
aspell ~ ~ ✔ ✘
xmail ✘ ✘ ✔ ✘
exim ✘ ✘ ✔ ✔
Observations:
• Faults can be distant from
the failure points:
=> POFs and call stacks
unlikely to help• More information is not
always better• Symbolic execution can
be a limiting factor
Synth.: 9/16Mimic: 6/16
Synth.: 10/16Mimic: 6/16
Synth.: 16/16Mimic: 16/16
Synth.: 2/16Mimic: 2/16
![Page 24: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/24.jpg)
BUGREDUX EVALUATION – RESULTSName POF Call Stack Call Seq. Compl. Tracesed #1 ✘ ✘ ✔ ✘
sed #2 ✘ ✘ ✔ ✘
grep ✘ ~ ✔ ✘
gzip #1 ✔ ✔ ✔ ✘
gzip #2 ~ ~ ✔ ✘
ncompress ✔ ✔ ✔ ✘
polymorph ✔ ✔ ✔ ✘
aeon ✔ ✔ ✔ ✔
rsync ✘ ✘ ✔ ✘
glftpd ✔ ✔ ✔ ✘
htget ~ ~ ✔ ✘
socat ✘ ✘ ✔ ✘
tipxd ✔ ✔ ✔ ✘
aspell ~ ~ ✔ ✘
xmail ✘ ✘ ✔ ✘
exim ✘ ✘ ✔ ✔
Observations:
• Faults can be distant from
the failure points:
=> POFs and call stacks
unlikely to help• More information is not
always better• Symbolic execution can
be a limiting factor
Symbolic execution can
be ineffective for
• programs with highly
structured inputs
• programs that interact
with external libraries
• large complex programs
in general
Synth.: 9/16Mimic: 6/16
Synth.: 10/16Mimic: 6/16
Synth.: 16/16Mimic: 16/16
Synth.: 2/16Mimic: 2/16
![Page 25: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/25.jpg)
SBFRJoint work with
Kifetew, Jin, Tiella, Tonella
Test InputCrash report(execution data) • Execution data
• Input generation technique
• Call sequence
• Genetic Programming
![Page 26: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/26.jpg)
SBFRJoint work with
Kifetew, Jin, Tiella, Tonella
Test InputCrash report(execution data)
Grammar
<a> ::= <b> |λ
![Page 27: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/27.jpg)
Joint work with
Kifetew, Jin, Tiella, Tonella
Test InputCrash report(execution data)
Grammar
<a> ::= <b> |λ
DerivationTree
GeneticProgramming
SBFR
Sentence derivation from the grammar:Random application of grammar rules• Uniform• 80/20• Stochastic (from a corpus)
![Page 28: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/28.jpg)
Joint work with
Kifetew, Jin, Tiella, Tonella
Test InputCrash report(execution data)
Grammar
<a> ::= <b> |λ
DerivationTree
GeneticProgramming
SBFR
Sentence derivation from the grammar:Random application of grammar rules• Uniform• 80/20• Stochastic (from a corpus)
Evolution: Fitness function: Distance b/w execution traces (candidate–actual failure)
![Page 29: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/29.jpg)
Joint work with
Kifetew, Jin, Tiella, Tonella
Test InputCrash report(execution data)
Grammar
<a> ::= <b> |λ
DerivationTree
GeneticProgramming
✔ ︎
Stopping criterion:• Success• Ic reaches the point of failure• The program fails “in the same way”• Search budget exhausted
SBFR
![Page 30: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/30.jpg)
SBFR EVALUATION – FAILURES CONSIDERED
Name Language Size(KLOC) # Productions # Faults
calc Java 2 38 2
bc C 12 80 1
MSDL Java 13 140 5
PicoC C 11 194 1
Lua C 17 106 2
![Page 31: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/31.jpg)
SBFR EVALUATION – FAILURES CONSIDERED
Name Language Size(KLOC) # Productions # Faults
calc Java 2 38 2
bc C 12 80 1
MSDL Java 13 140 5
PicoC C 11 194 1
Lua C 17 106 2
BugRedux was unable to reproduce any of
these failures with a timeout of 72 hours
![Page 32: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/32.jpg)
SBFR EVALUATION – RESULTSName FRP (SBFR) FRP (Random)
calc bug 1 0.0calc bug 2 0.0
bc 0.0MSDL bug 1 0.0MSDL bug 2 0.0MSDL bug 3 1.0MSDL bug 4 0.0MSDL bug 5 0.0
PicoC 0.1Lua bug 1 0.0Lua bug 2 0.0
•Parameters:•Population: 500•Budget: 10,000 unique
fitness evaluations•Performed 10 runs•Measured failure
reproduction probability•Used both 80/20 and
stochastic derivations
![Page 33: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/33.jpg)
SBFR EVALUATION – RESULTSName FRP (SBFR) FRP (Random)
calc bug 1 0.6 0.0calc bug 2 0.8 0.0
bc 1.0 0.0MSDL bug 1 1.0 0.0MSDL bug 2 1.0 0.0MSDL bug 3 1.0 1.0MSDL bug 4 1.0 0.0MSDL bug 5 1.0 0.0
PicoC 0.8 0.1Lua bug 1 0.0 0.0Lua bug 2 0.5 0.0
![Page 34: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/34.jpg)
SBFR EVALUATION – RESULTSName FRP (SBFR) FRP (Random)
calc bug 1 0.6 0.0calc bug 2 0.8 0.0
bc 1.0 0.0MSDL bug 1 1.0 0.0MSDL bug 2 1.0 0.0MSDL bug 3 1.0 1.0MSDL bug 4 1.0 0.0MSDL bug 5 1.0 0.0
PicoC 0.8 0.1Lua bug 1 0.0 0.0Lua bug 2 0.5 0.0
![Page 35: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/35.jpg)
SBFR EVALUATION – RESULTSName FRP (SBFR) FRP (Random)
calc bug 1 0.6 0.0calc bug 2 0.8 0.0
bc 1.0 0.0MSDL bug 1 1.0 0.0MSDL bug 2 1.0 0.0MSDL bug 3 1.0 1.0MSDL bug 4 1.0 0.0MSDL bug 5 1.0 0.0
PicoC 0.8 0.1Lua bug 1 0.0 0.0Lua bug 2 0.5 0.0
Example: failure in bc
segmentation fault triggered by an instruction
sequence that allocates at least 32 arrays and
declares a number of variables higher than
the number of allocated arrays
![Page 36: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/36.jpg)
SBFR EVALUATION – RESULTSName FRP (SBFR) FRP (Random)
calc bug 1 0.6 0.0calc bug 2 0.8 0.0
bc 1.0 0.0MSDL bug 1 1.0 0.0MSDL bug 2 1.0 0.0MSDL bug 3 1.0 1.0MSDL bug 4 1.0 0.0MSDL bug 5 1.0 0.0
PicoC 0.8 0.1Lua bug 1 0.0 0.0Lua bug 2 0.5 0.0
Example: failure in bc
segmentation fault triggered by an instruction
sequence that allocates at least 32 arrays and
declares a number of variables higher than
the number of allocated arrays
Observations:
• Search-based approaches can be effective in
cases that symbolic execution cannot handle
• Stochastic grammars are effective
• SBST more scalable, but less directed
=> SBST and DSE are complementary,
rather than alternative techniques
![Page 37: FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION …crest.cs.ucl.ac.uk/cow/30/slides/COW30_Orso.pdf · FIELD FAILURE REPRODUCTION USING SYMBOLIC EXECUTION AND GENETIC PROGRAMMING](https://reader034.vdocuments.us/reader034/viewer/2022050418/5f8dabaf07879e750b4b68d8/html5/thumbnails/37.jpg)
• Relevant execution data identification• Which types?• Which specific ones?
• Failure explanation• Reproduction is not enough• Can DSE and SBST help?
• Use of different input generation techniques• Grammar-based symbolic execution • Backward symbolic analysis?• Other SBST approaches?• SBST targeted at different kinds of programs?• Combination of techniques
FUTURE WORK / FOOD FOR THOUGHTS