search based software testing
Post on 18-Dec-2021
1 Views
Preview:
TRANSCRIPT
Search based Software Testing
Mark Harman!Joint work with Yue Jia and Yuanyuan Zhang
Achievements, Open Problems and Challenges for
University College London
There is a paper accompany this keynote
Yuanyuan !Zhang
Yue Jia!Technical work!and considerable help with slides
… and he’s here in Graz too
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
20 mins walk
National Gallery
Nelson’s Column
ErosRoyalCourts
of Justice
St. Paul’s
Tate ModernGlobe
Theatre
Covent Garden Market
Westminster Abbey
House of Parliament
London Eye
British Museum
Madame Tussaud’sSherlock Holmes
Museum
Marble Arch
National History Museum
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based
Recorded and archived
http://crest.cs.ucl.ac.uk/cow/
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based!
Recorded and archived
http://crest.cs.ucl.ac.uk/cow/
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based
Recorded and archived
http://crest.cs.ucl.ac.uk/cow/
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
COWs
http://crest.cs.ucl.ac.uk/cow/
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
COWs
http://crest.cs.ucl.ac.uk/cow/
#Total Registrations 1347 #Unique Attendees 623 #Unique Institutions 232 #Countries 42 #Talks 372 !
(Last updated on January 31, 2015) !
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
What is SBST
Search Based Optimization
Software Testing
S B S T
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
What is SBSTIn SBST we apply search techniques to search large search spaces, guided by a fitness function that captures natural counterparts as test objectives.
Genetic AlgorithmsHill ClimbingSimulated Annealing Random
Tabu Search
Estimation of Distribution Algorithms
Particle Swarm OptimizationAnt Colonies
LPGenetic Programming
Greedy
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
What is SBSEIn SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures natural counterparts as test objectives.
Genetic AlgorithmsHill ClimbingSimulated Annealing Random
Tabu Search
Estimation of Distribution Algorithms
Particle Swarm OptimizationAnt Colonies
LPGenetic Programming
Greedy
Search Based Software Engineering
History of SBST
20011842
“In almost every computation a great variety of arrangements for the succession of the processes is possible, and various considerations must influence the selection amongst them for the purposes of a Calculating Engine. One essential object is to choose that arrangement which shall tend to reduce to a minimum the time necessary for completing the calculation.”
Extract from ‘Note D’.
“In almost every computation a great variety of arrangements for the succession of the processes is possible, and various considerations must influence the selection amongst them for the purposes of a Calculating Engine. One essential object is to choose that arrangement which shall tend to reduce to a minimum the time necessary for completing the calculation.”
Extract from ‘Note D’.
1879
Checking a large routine by Dr. A. Turing
In this shot paper, Turing suggested the use of manually constructed assertions and we can find the origins of both software testing and software verification.
Checking a large routine by Dr. A. Turing
In this shot paper, Turing suggested the use of manually constructed assertions and we can find the origins of both software testing and software verification.
Checking a large routine by Dr. A. Turing
In this shot paper, Turing suggested the use of manually constructed assertions and we can find the origins of both software testing and software verification.
Sauder formulates the test generation problem as one of finding test inputs from a search space, though the search algorithm is random search, making this likely to be the first paper on Random Test Data Generation.
Sauder formulates the test generation problem as one of finding test inputs from a search space, though the search algorithm is random search, making this likely to be the first paper on Random Test Data Generation.
The seminal PhD thesis by James King
James King used automated symbolic execution to capture path conditions, solved using linear programming
The seminal PhD thesis by James King
James King used automated symbolic execution to capture path conditions, solved using linear programming
“We therefore considered various alternatives that would not be subject to this limitation. The most promising of these alternatives appears to be a conjugate gradient algorithm (‘hill climbing’ program) that seeks to minimise a potential function constructed from the inequalities.”
“We therefore considered various alternatives that would not be subject to this limitation. The most promising of these alternatives appears to be a conjugate gradient algorithm (‘hill climbing’ program) that seeks to minimise a potential function constructed from the inequalities.”
At about the same time, Miller and Spooner were also experimenting with optimisation-based approaches for generating test data (which they refer to as ‘test selection’ in the sense that they ‘select’ from the input space, which, in the more recent literature we would refer to as ‘test data generation’).
At about the same time, Miller and Spooner were also experimenting with optimisation-based approaches for generating test data (which they refer to as ‘test selection’ in the sense that they ‘select’ from the input space, which, in the more recent literature we would refer to as ‘test data generation’).
It appears that SBST research lay dormant for at approximately a decade until the work of Korel, which introduced a practical test data generation approach, the Alternating Variable Method (AVM), based on hill climbing.
It appears that SBST research lay dormant for at approximately a decade until the work of Korel, which introduced a practical test data generation approach, the Alternating Variable Method (AVM), based on hill climbing.
Windows 3.1
The first use of genetic algorithms for software engineering problems is usually attributed also to the field of SBST, with the work of Xanthakis et al., who introduced a genetic algorithm to develop whole test suites.
The first use of genetic algorithms for software engineering problems is usually attributed also to the field of SBST, with the work of Xanthakis et al., who introduced a genetic algorithm to develop whole test suites.
The first suggestion of search as a universal approach to Software Engineering
SBSE
The first suggestion of search as a universal approach to Software Engineering
SBSE
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
Analysis of Trends in SBST
The data is taken from the SBSE
Careful human-based update
100% precision and recall
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
Polynomial yearly rise in the number of papers Search Based Software Testing
y"="0.0013x4"*"0.061x3"+"1.0008x2"*"5.8636x"+"10.443"
0"
100"
200"
300"
400"
500"
600"
700"
800"
1975"
1977"
1979"
1981"
1983"
1985"
1987"
1989"
1991"
1993"
1995"
1997"
1999"
2001"
2003"
2005"
2007"
2009"
2011"
2013"
Accumulated
*Num
ber*o
f*SBS
T*Pu
blica5
ons*
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%#1975#1978#1981#1984#1987#1990#1993#1996#1999#2002#2005#2008#2011#2014#
SBST$ Other$SBSE$Publica2ons$
The changing ratio SBSE to SBST
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST papers at ICST
Reformulating Branch Coverage as a Many-Objective Optimization Problem!Annibale Panichella, Fitsum Meshesha Kifetew and Paolo Tonella!!Behind an Application Firewall, Are We Safe from SQL Injection Attacks?!Dennis Appelt, Cu D. Nguyen, Lionel Briand!!Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection!Debajyoti Mondal, Hadi Hemmati, and Stephane Durocher!!Guided Test Generation for Finding Worst-Case Stack Usage in Embedded Systems!Tingting Yu and Myra B. Cohen!!Re-using Generators of Complex Test Data!Simon Poulding and Robert Feldt!!U-Test: Evolving, Modelling and Testing Realistic Uncertain Behaviours of Cyber-Physical Systems!Shaukat Ali and Tao Yue!!History-Based Test Case Prioritization for Black Box Testing using Ant Colony Optimization!Tadahiro Noguchi, Hironori Washizaki, Yoshiaki Fukazawa, Atsutoshi Sato and Kenichiro Ota!!Combining Minimization and Generation for Combinatorial Testing!Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick.
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
S B S T
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Structural
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Structuralfind tests to!cover !branches,!statements &!dataflow, etc.
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Integration
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Integration
find !best component!ordering
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Temporal
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Temporal
find worst case!execution time
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
CIT
find 2-way, 3-way!n-way!interaction tests
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
SPLs
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Augment
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Augment
find new tests!from old tests
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Regression
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Regression
find good!subsets and !orders of tests
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Functional
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Mutation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
State!based
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Model !based
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Black box
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Failure !Analysis
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Security
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Web/!Services
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanHistory
Agents
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Industrial Applications and Tools
Joachim Wegener and Oliver Bühler. GECCO 2004
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Industrial Applications and Tools
Wasif Afzal, Richard Torkar, Robert Feldt and Greger Wikstrand. SSBSE 2010
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Industrial Applications and Tools
Nikolai Tillmann, Jonathan de Halleux and Tao Xie. ASE 2014
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST Public Tools
AUSTIN applied to real-world embedded automotive industry: Daimler, B&M Systemtechnik. Recommended for testing C.
Kiran Lakhotia,Mark Harman,and Hamilton Gross. I&ST 2013
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST Public Tools
EvoSuite automatically generates test cases for Java code. An excellent and high recommended tool.
Gordon Fraser and Andrea Arcuri. ESEC/FSE 2011
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
More details in the keynote paper in your ICST proceedings
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation techniques (MoSBaT).
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation techniques (MoSBaT).
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation techniques (MoSBaT).
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanAnalysis
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation techniques (MoSBaT).Annibale Panichella, Fitsum Meshesha Kifetew and Paolo Tonella!Reformulating Branch Coverage as a Many-Objective Optimization ProblemNext Session:
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
SBST for Non-Functional Properties
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
SBST for Non-Functional Properties
0"
2"
4"
6"
8"
10"
12"
1996$ 1998$ 2000$ 2002$ 2004$ 2006$ 2008$ 2010$ 2012$ 2014$
Num
ber'o
f'SBS
T'Pu
blica2
on'fo
r'Non
4Func2n
al'Prope
r2es'
Wasif Afzal, Richard Torkar, and Robert Feldt. I&ST 2009Mark Harman, Yue Jia and Yuanyuan Zhang. ICST 2015. Extends Afzal et al 2009.
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Execu&on)&me)15)43%)
QoS)2)6%)
Security)7)
20%)
Usability)7)
20%)
Safety)4)
11%)
The)Categories)of)NonEFunc&onal)Proper&es)from)1996)to)2007)
Execu&on)&me)21)25%)
QoS)4)5%)
Security)13)15%)Usability)
10)12%)
Safety)8)9%)
Availability)1)1%)
Efficiency)8)
10%)
Energy)consump&on)
1)1%)
Flexibility))3)4%)
Robustness)10)12%)
Scalability)5)6%)
The)Categories)of)NonLFunc&onal)Proper&es)from)1996)to)2014)
Changing distribution of workTiny amount on SBET
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)
A smartphone could consume more energy per year than a medium-sized refrigerator
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)
A smartphone could consume more energy per year than a medium-sized refrigerator
IT energy consumption rose 3% in 3 years
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)Measure energy consumption as a fitness function
Efficiency: will need to consider many different test cases
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
Hawthorne Effect: Instrumentation may affect energy consumed
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
Search based Energy Test (SBET)Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
Hawthorne Effect: Instrumentation may affect energy consumed
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBET
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
Search Based Test Strategy Identification (SBTSI)
Move from finding specific inputs to finding strategies for finding inputs
A co-evolutionary SBSTIfor Mutation testing
Unknown CIT Problems
General CIT Solution
A hyperheuristic SBSTI for CIT
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Solutions
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
Unknown CIT Problems
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
Unknown CIT Problems
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
CIT Problem !CharacteristicsCIT Solutions
unconstrainted problems
constrainted problems
Weighted problems
AETG
IPOG
Simulated Annealing
Tabu Hill Climbing
GA
Specific structures
General CIT Solution
Unknown CIT Problems
A hyperheuristic SBSTI for CIT
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
General CIT Solution
Unknown CIT Problems
Learning Combinatorial Interaction Test Generation Strategies using Hyperheuristic Search. Yue Jia, Myra Cohen, Mark Harman and Justyna Petke. ICSE 2015
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
General CIT Solution
Unknown CIT Problems
Learning Combinatorial Interaction Test Generation Strategies using Hyperheuristic Search. Yue Jia, Myra Cohen, Mark Harman and Justyna Petke. ICSE 2015
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
Predator Prey
Testing Bugs
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
Evolving
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
Evolving
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanSBSTI
Higher order mutantsTest data
A co-evolutionary approach for SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Multi-Objective Search Based Testing (MOSBAT)
increasingly prevalent regression testing was early adopter
e.g. Yoo and Harman: ISSTA 2007
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Multi-Objective Search Based Testing (MOSBAT)
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Multi-Objective Search Based Testing (MOSBAT)
test case generation is still mostly single objective
Coverage
Test data Generation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Multi-Objective Search Based Testing (MOSBAT)
Coverage
Security
Energy consumption
Usability
Execution Time
Test data Generation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Multi-Objective Search Based Testing (MOSBAT)
Multi-objective Understanding: Debug Security policies
SecurityUsability
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation techniques (MoSBaT).
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanMOSBAT
Let me ask you something …
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Genetic Improvement: searching for
improving modifications guided by testing
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Bowtie2 GP ProgramsProgramsProgramsBowtie2 Improved
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Genetic Improvement of Programs
W. B. Langdon and M. Harman Optimising Existing Software with Genetic Programming. TEC 2015
70 times faster 30+ interventions
HC clean up: 7 slight semantic improvement
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Genetic Improvement of Programs
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Cuda GP ProgramsProgramsProgramsCuda Improved
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Genetic Improvement of Programs
W. B. Langdon and M. Harman Genetically Improved CUDA C++ Software, EuroGP 2014
7 times faster updated for new hardware
automated updating
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Inter version transplantation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP ProgramsProgramsProgramsMiniSat Improved
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Justyna Petke, Mark Harman, William B. Langdon and Westley Weimer Using Genetic Improvement & Code Transplants to Specialise a C++ program to a Problem Class (EuroGP’14)
Multi-doner transplant Specialized for CIT
17% faster
MiniSat
MiniSat
MiniSat
v1
v2
vn
GECCO Humie !
silver medal
Inter version transplantation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Real world cross system transplantation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Real world cross system transplantation
Earl T. Barr, Mark Harman, Yue Jia, Alexandru Marginean, and Justyna Petke Automated Software Transplantation (Tech Report). To appear.
Doner
Host
featureHost’
feature
Successfully autotransplanted new functionality and passed all regression tests for 12 out of 15 real world systems
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Memory speed trade offs
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Memory speed trade offs
System
malloc
System
optimised malloc
Fan Wu, Westley Weimer, Mark Harman, Yue Jia and Jens KrinkeDeep Parameter OptimisationConference on Genetic and Evolutionary Computation (GECCO'15), To appear
Improve execution time by 12% or achieve a 21% memory consumption reduction
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Memory speed trade offs
System
malloc
System
optimised malloc
Fan Wu, Westley Weimer, Mark Harman, Yue Jia and Jens KrinkeDeep Parameter OptimisationConference on Genetic and Evolutionary Computation (GECCO'15), To appear
Improve execution time by 12% or achieve a 21% memory consumption reduction
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Reducing energy consumption
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Reducing energy consumption
Ensemble
AProVE
MiniSat
CITMiniSat
MiniSat
Ensemble
AProVE
Improved MiniSat
CITImproved MiniSat
Improved MiniSat
Bobby R. Bruce Justyna Petke Mark Harman Reducing Energy Consumption Using Genetic Improvement Conference on Genetic and Evolutionary Computation (GECCO'15), To appear
Energy consumption can be reduced by as much as 25%
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
? GP Programs
Non-functional property Test
harness
Fitness
Test data
Sensitivity Analysis
Grow and graft new functionality
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
GP
Non-functional property Test
harness
Fitness
Test data
Grow and graft new functionality
GP
Non-functional property Test
harness
Fitness
Test data
Sensitivity AnalysisFeature
Grow Graft
Human!Knowledge Feature
Host System
Mark Harman, Yue Jia and Bill Langdon, Babel Pidgin: SBSE can grow and graft entirely new functionality into a real world system Symposium on Search-Based Software Engineering SSBSE 2014. (Challenge track) Chall
enge T
rack !
Award
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Another keynote paper at ASE 2012
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Pareto Front
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Pareto Front
each circle is a program found by a machine
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Pareto Front
different non functional
properties have different pareto program fronts
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Failed Test Cases
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Why can’t functional properties be optimisation objectives ?
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Optimisation
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Optimisation
2.5 times faster but failed 1 test case?
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
ICST’15 Achievements, Open Problems and Challenges for SBST Mark HarmanGI
Optimisation
double the battery life but failed 2 test cases?
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
Summary
Isn’t testing all about searching?!!
Searching for test cases!Searching for test application orders!
Searching for patches!!
Searching for better programs guided by tests!Genetic Improvement
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
Summary
Isn’t testing all about searching?!!
Searching for test cases!Searching for test application orders!
Searching for patches!!
Searching for better programs guided by tests!Genetic Improvement
ICST’15 Achievements, Open Problems and Challenges for SBST Mark Harman
Picture Copyrights
http://en.wikipedia.org/wiki/Colossus_computer#/media/File:Colossus.jpghttp://en.wikipedia.org/wiki/Thomas_Edison#/media/File:Edison_bulb.jpg
http://beatles.wikia.com/wiki/The_Beatles_Wiki
http://en.wikipedia.org/wiki/Hippie#/media/File:Woodstock-kids.jpg
top related