austin: a tool for search based software testing and its evaluation on deployed automotive systems...
TRANSCRIPT
![Page 1: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/1.jpg)
AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTINGAnd its Evaluation on Deployed Automotive Systems
Kiran Lakhotia, Mark Harman and Hamilton Gross in
![Page 2: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/2.jpg)
Publication Trend in SBSE
![Page 3: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/3.jpg)
Spread of Activities in SBSE
![Page 4: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/4.jpg)
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil
McMinn)
eTOC – for JAVA (Paolo
Tonella)
Others?
![Page 5: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/5.jpg)
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil McMinn)
eTOC – for JAVA
(Paolo Tonella)
Others?
AUSTIN
![Page 6: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/6.jpg)
Outline
AUSTIN •Introduction to Austin
ETF •Overview of ETF
EmpiricalStudy
•Hypotheses, Test Subjects•Results and Analysis
Conclusion •ETF or AUSTIN?
![Page 7: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/7.jpg)
AUgmented Search based TestINg
CIL
AST, CFG
instrument
gcc -o sut.exe
runAUSTIN
gcc -P -Ec sourcefiles
preprocessed
c sourcefiles
instrumented
c source
![Page 8: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/8.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
![Page 9: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/9.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 87, key2 = 33, p = 0, q = 0
![Page 10: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/10.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 88, key2 = 33, p = 0, q = 0
![Page 11: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/11.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 86, key2 = 33, p = 0, q = 0
![Page 12: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/12.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 84, key2 = 33, p = 0, q = 0
![Page 13: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/13.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0
![Page 14: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/14.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0
Fitness Function:approach_level + norm(branch_distance)
Search Method:Alternating Variable Method
![Page 15: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/15.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
![Page 16: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/16.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
![Page 17: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/17.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
![Page 18: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/18.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
![Page 19: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/19.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
p 0
![Page 20: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/20.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
![Page 21: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/21.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
Symbolic Path Condition:key1 == key2 && p ≠ 0 && p ≠ q
Simplified Path Condition:p ≠ 0 && p ≠ q
Solve: p ≠ 0 && p == qp 0
q
p,q
0
![Page 22: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/22.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0x...
![Page 23: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/23.jpg)
Evolutionary Testing Framework (ETF)
rZSlJ3hb
ETF was developed as part of the EvoTest project Integrated into the Eclipse IDE Supports white & black box testing Uses Evolutionary Algorithms (through the
GUIDE, EO evolutionary library interface) to generate test data
ETF
Selection
Recombinatio
n
Mutation
ReinsertionGUIDE + EO
Send Fitness Values
Send Individuals
![Page 24: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/24.jpg)
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
![Page 25: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/25.jpg)
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
![Page 26: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/26.jpg)
Fitness Function and Algorithm Both ETF and AUSTIN use the standard
fitness function for branch coverage in Search Based Testing apporach_level + norm(branch_distance)
normalization function: 1 - 1.001-branch_distance
ETF uses a Genetic Algorithm AUSTIN uses the Alternating Variable
Method
![Page 27: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/27.jpg)
Pointers in ETF and AUSTIN
AUSTIN Construct an equivalence of memory locations based on =
and ≠ operators in symbolic path condition Initialize (pointer) input variables based on the graph
Assign new memory location via malloc or use existing memory location
On-demand pointer handling ETF
Generate an index and auxiliary variable for each pointer input
Assign all input variables of the same type (including globals) to a pool Optimize indices and values of auxiliary variables along with the
inputs. Indices are used to select variables from pools and assign their address to the pointer input
![Page 28: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/28.jpg)
Pointers in ETF
pool of input parameters and auxiliary variables
pCell
globDbl
val_p, val_q,globX
struct cell{ int value; struct cell* next;};int globX;double globDbl;struct cell* pCell;
void testme(int* p, int* q){ ...}
p = &globX;q = &val_p;
![Page 29: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/29.jpg)
Hypothesis
Null Hypotheses Alternate Hypotheses
AUSTIN is as effective as the ETF in achieving branch coverage.
AUSTIN is more effective than the ETF in achieving branch coverage.
AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.
AUSTIN is more efficient than the ETF in achieving branch coverage of a function.
![Page 30: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/30.jpg)
Empirical Study
3 case studies (comprising 8 functions) Embedded Software Modules from
Automotive IndustryCase Study
Description LOC - Tested
Total Branches
Nr. Inputs
B Adaptive Headlight Control Software
1,236 598 132
C Door-Lock Control Software
383 315 71
D Electric Windows Control Software
339 248 31
![Page 31: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/31.jpg)
Coverage ETF vs AUSTIN
![Page 32: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/32.jpg)
![Page 33: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/33.jpg)
Efficiency ETF vs AUSTIN
![Page 34: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/34.jpg)
![Page 35: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/35.jpg)
Coverage Random vs AUSTIN
![Page 36: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/36.jpg)
Efficiency Random vs AUSTIN
![Page 37: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/37.jpg)
ETF or AUSTIN?
Coverage is about the same AUSTIN is overall a little bit more
efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)
AUSTIN does not construct random memory graphs
ETF is not able generate inputs such as
struct cell{ int mem;};void testme(int* p, struct cell* c) { if( p == &c->mem ) //target}
![Page 38: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/38.jpg)
Summary
![Page 39: AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTING And its Evaluation on Deployed Automotive Systems Kiran Lakhotia, Mark Harman and Hamilton Gross in](https://reader030.vdocuments.us/reader030/viewer/2022032806/56649f075503460f94c1cfd1/html5/thumbnails/39.jpg)
Summary
http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm