![Page 1: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/1.jpg)
Specification-Based Error Localization
Brian DemskyMartin Rinard
Laboratory for Computer ScienceMassachusetts Institute of Technology
![Page 2: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/2.jpg)
Problem
Error Introduced
Execution with
Broken Data
Structure
Crash or Unexpected
Result
• Have to trace symptom back to cause• Corruption may not cause visible
error in test suite
![Page 3: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/3.jpg)
Problem
Error Introduced
Execution with
Broken Data
Structure
Crash or Unexpected
Result
• Solution: discover bugs when • they corrupt data• not when effect becomes visible
• Perform frequent consistency checks• Bug localized between
• first unsuccessful check and• last successful check
![Page 4: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/4.jpg)
Architecture
next
value
value
o1
o1
o1
o1
values
nodes
Concrete Data Structure Abstract Model
Model DefinitionRules
Consistency Constraints
![Page 5: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/5.jpg)
Architecture RationaleWhy use the abstract model?
• Model construction separates objects into sets• Reachability properties• Field values
• Different constraints for objects in different sets• Appropriate division of complexity
• Data structure representation complexity encapsulated in model construction rules
• Consistency property complexity encapsulated in (clean, uniform) model constraint language
![Page 6: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/6.jpg)
Simplified Freeciv Example
tile grid[EDGE][EDGE];structure tile {
int terrain;city *city;
} structure city {
int population;}
PO MM
OO MP
PO MM
PP MP
Terrain Grid
City Structures
O = OceanP = PlainM = Mountain
![Page 7: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/7.jpg)
Sets and Relations in Model
• Sets of objectsset TILE of tilegrid;set CITY of city;
• Relations between objects – values of object fields, referencing relationships between objectsrelation CITYMAP : TILE -> CITY;relation TERRAIN : TILE -> integer;
![Page 8: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/8.jpg)
Model TranslationBits translated to sets and relations in abstract
model using statements of the form:
Quantifiers, Condition Inclusion Constraint
for x in 0..EDGE*EDGE, true grid[x] in TILE
for t in TILE, true t,t.terrain in TERRAIN
for t in TILE, !t.city = NULL t,t.city in CITYMAPfor t in TILE, !t.city=NULL t.city in CITY
![Page 9: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/9.jpg)
Model in Examplegrid[0]
grid[1]
grid[2]
grid[3]
terrain:
1 2 3 4
city: NULL NULLpopulation:
10,000
grid[0]
grid[1]
grid[2]
grid[3]
Tiles
1234
city
Cities
![Page 10: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/10.jpg)
Consistency PropertiesQuantifiers, Body
• Body is first-order property of basic propositions• Inequality constraints on numeric fields • Cardinality constraints on sizes of sets• Referencing relationships for each object• Set and relation inclusion constraints
• Example:for t in TILE, MIN <= t.TERRAIN and t.TERRAIN<=MAXfor c in CITY, size(CITYMAP.c)=1for c in CITY, !(CITYMAP.c).TERRAIN=OCEAN
![Page 11: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/11.jpg)
Consistency ViolationsEvaluate consistency properties, find
violationsfor c in CITY, size(CITYMAP.c)=1
grid[0]
grid[1]
grid[2]
grid[3]
Tiles
1234
city
Cities
![Page 12: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/12.jpg)
Slide about checks
• TODO
![Page 13: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/13.jpg)
Optimized Implementation• Compilation (4.7x speedup)• Fixed point elimination (210x speedup)
• Evaluate model definition rules using simple traversal• Relation construction elimination (500x speedup)
• Evaluate uses of relations directly on data structures• Set construction elimination (3900x speedup)
• Evaluate constraints while traversing data structures• Bottom line
• Interpreted version X times slower than uninstrumented
• Optimized version Y times slower than uninstrumented
![Page 14: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/14.jpg)
Freeciv Case Study
• Multiplayer Client/Server based online game
• Available at www.freeciv.org• Case study looked at the server• Server contains 73,000 lines of code• Added 750 instrumented sites• 20,000 consistency checks performed in
our sample execution
![Page 15: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/15.jpg)
Case Study
• Created three buggy version of Freeciv• Two groups of three developers
• One used conventional tools• One used specification-based
consistency checking• Each participant was asked to spend at
least one hour on each version• Both populations given a pre-
instrumented version of Freeciv
![Page 16: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/16.jpg)
Consistency Properties• Map exists
size(MAP)=1• Grid of tiles exists
size(GRID)=1• Tiles have valid terrain values
for t in TILE, MIN <= t.TERRAIN and t.TERRAIN<=MAX
• Cities are not in the oceanfor c in CITY, !(CITYMAP.c).TERRAIN=OCEAN
• Each city has exactly one reference from the gridfor c in CITY, size(CITYMAP.c)=1
![Page 17: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/17.jpg)
Bugs Introduced
• Actual errors in buggy versions• First error creates invalid terrain values
(violates valid terrain property)• Second causes two tiles to refer to the
same city (violates single reference property)
• Third causes a city to be placed on ocean (violates cities not in ocean property)
![Page 18: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/18.jpg)
Results
• User study shows benefit from approach• With tool
• All developers found and fixed all bugs• Mean of 11 minutes required
• Without tool• Three developers found total of one
bug (out of nine developer/bug combinations)
• Others spent hour debugging (unsuccessfully)
![Page 19: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/19.jpg)
Repair for Deployed Systems
• Consistency specifications for repair• Input: inconsistent data structure• Output: consistent data structure• Technique enables programs to recover
from data structure corruption• And continue to execute successfully
• OOPSLA ’03 paper describes this technique
Full reference here
![Page 20: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/20.jpg)
Related Work
• Specification languages such as UML or Alloy
• Specification-based testing • Korat (Boyapati et. al. ISSTA 2002)• Testera (Marinov and Khurshid)• Eiffel (Meyer)
• Invariant inference and checking• Daikon (Ernst et. al.)• DIDUCE (Hangal and Lam)• Carrot (Pytlik et. Al.)
![Page 21: Specification-Based Error Localization Brian Demsky Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d5e5503460f94a3dbb3/html5/thumbnails/21.jpg)
Conclusion
• Consistency checking to localize data structure corruption bugs
• Good experimental results• With checker, bugs fixed in minutes• Without checker, bugs not fixed in an
hour• Optimizations for good performance• Data structure repair