oliver krauss msc dynamic fitness functions for genetic ... · genetic improvement in compilers and...
TRANSCRIPT
Oliver Krauss MSc
Dynamic Fitness Functions forGenetic Improvement in Compilersand InterpretersKyoto 16.7.2018
Advisors: Prof. Dr. Dr. h.c. Hanspeter Mössenböck,Prof. (FH) Priv.-Doz. Dipl.-Ing. Dr. Michael Affenzeller
AbstractGenetic Improvement on Interpreter / Compiler AbstractSyntax Tree (AST)
– Improve non-functional software features (performance)– Deal with large fitness landscapes– Dynamic fitness functions ...
– split test suites by the complexity of the cases– Sequential or Parallel
– ... don't perform well– ... show promise
Page 1 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 2 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 3 | 26
Search Space Size
∗
∗ ∗
∗ ∗
∗ ∗
depth
width
– n node types (MiniC -> 158)– depth 5, width 5
– All Options -> n3901
– MiniC -> 466.638– depth 10, width 10
– All Options -> n11.111.111.101
– MiniC -> 80.814.253.363
Page 4 | 26
Search Space Size
∗
∗ ∗
∗ ∗
∗ ∗
depth
width
– n node types (MiniC -> 158)– depth 5, width 5
– All Options -> n3901
– MiniC -> 466.638
– depth 10, width 10– All Options -> n11.111.111.101
– MiniC -> 80.814.253.363
Page 4 | 26
Search Space Size
∗
∗ ∗
∗ ∗
∗ ∗
depth
width
– n node types (MiniC -> 158)– depth 5, width 5
– All Options -> n3901
– MiniC -> 466.638– depth 10, width 10
– All Options -> n11.111.111.101
– MiniC -> 80.814.253.363
Page 4 | 26
Search Space Size
∗
∗ ∗
∗ ∗
∗ ∗
depth
width
– n node types (MiniC -> 158)– depth 5, width 5
– All Options -> n3901
– MiniC -> 466.638– depth 10, width 10
– All Options -> n11.111.111.101
– MiniC -> 80.814.253.363
Page 4 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 5 | 26
JIT Compilation with GI ICode
void main() {int i,
n,now ,prev ,next;
prev = 0;now = 1;i = 0;n = 30;while (i
< n){if (i
==0 )print (0);
elseif(i==1)print (1);
else {next
=now+prev;
prev=now;
now=next;
print(next);}i = i
+1;
}}
GI Truffle Graal
Coco/R Optimize Compile
Figure: Code Interpretation and Compilation with GI (Truffle and Graalimages from [1])
Page 6 | 26
JIT Compilation with GI II
MiniC– Subset of C11 [2]– Coco/R -> Creates Scanner & Parser in Java– 342 node classes (103 operators, 55 operands)
Page 7 | 26
JIT Compilation with GI IIITruffle [3, 1]
– Interpreter based on AST– Generalized and Spezialized AST - Nodes– Self optimizing by AST-Rewriting– Can run any Guest Language on the JVM– Python, Ruby, JavaScript, ...
Page 8 | 26
JIT Compilation with GI IV
Graal [4, 5, 6]
– Just-in-time (JIT) compiler– Agressive optimizations– Part of OpenJDK
Page 9 | 26
JIT Compilation with GI VAST from Coco/R Select subtree
for Optimization Run in TruffleCollect TestdataIN OUT
0 01 12 13 24 35 5
Create Stub
GI Population Evaluate Populationwith Truffle Hand best solution
to Truffle
clone & build
Profile Get Testdata clone AST
evaluate
cross & mutate
after x Generations
Figure: GP Optimization in depthPage 10 | 26
Architectural Overview
Figure: The architecture of the optimization framework in combinationwith Truffle and Graal
Page 11 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 12 | 26
Node Visitationfib
{}read
n if ret
ret||
==
n 1
==
0nn
+
fib
−
n 1
fib
−
n 2
Visited Previously VisitedSpezialisation
Goal: visit all nodes– n = 0 [v=14,s=0]
– n = 1 [v=11,s=0]– n = 2 [v=23,s=6]
Page 13 | 26
Node Visitationfib
{}read
n if ret
ret||
==
n 1
==
0nn
+
fib
−
n 1
fib
−
n 2
Visited Previously VisitedSpezialisation
Goal: visit all nodes– n = 0 [v=14,s=0]– n = 1 [v=11,s=0]
– n = 2 [v=23,s=6]
Page 13 | 26
Node Visitationfib
{}read
n if ret
ret||
==
n 1
==
0nn
+
fib
−
n 1
fib
−
n 2
Visited Previously VisitedSpezialisation
Goal: visit all nodes– n = 0 [v=14,s=0]– n = 1 [v=11,s=0]– n = 2 [v=23,s=6]
Page 13 | 26
Test Metrics ICorrectness - Percentage of successful tests
correctness(AST,tests) =∑tests
n=1 succeeded(AST,test)∑tests (1)
Complexity - Percentage of nodes visited by a testcomplexity(AST,test) =
∑nodesn=1 visited(AST,test)∑
nodes(AST) (2)
Page 14 | 26
Test Metrics IIOverlap - Percentage of nodes visited by two tests
overlap(A,B) =
∑nodesn=1 visited(A) ∩ visited(B)
max(∑nodes
n=1 visited(A),∑nodes
n=1 visited(B))(3)
Confidence - Percentage of nodes visited by test suiteconfidence(testsuite,AST) =
∑nodesn=1 visited(testsuite)∑
nodes(AST) (4)
Page 15 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 16 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Sequential Fitness Function
Figure: Sequential algorithm, iteratively increasing the amount of testsused by the genetic algorithm (GA) to verify ASTsPage 17 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
Parallel Fitness Function
Figure: Parallel algorithm, splitting a test suite into parts, and iterativelyrecombining them into bigger suites until the original suite is reached
Page 18 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 19 | 26
Test SetUpGA vs. Sequential FFN vs. Parallel FFN
– Same configuration for all– No grafting– Comparison of solved test cases
Table: Use Case statisticsUse Case No. Nodes Tests Nodes visitedx*2 8 5 5, 5, 5, 5, 5Sqrt 60 5 14, 42, 42, 42, 53Fibonacci 32 5 11, 15, 30, 30, 30Constructed 31 5 11, 15, 20, 20, 21
Page 20 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)Constructed 3 of 5 tests
solved
Fitness Functions
GA BaselineSequential FFN Solved more
testsParallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2
Fibonacci only first 2(Sequential FFN)
Constructed 3 of 5 testssolved
Fitness Functions
GA BaselineSequential FFN Solved more
testsParallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)
Constructed 3 of 5 testssolved
Fitness Functions
GA BaselineSequential FFN Solved more
testsParallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)Constructed 3 of 5 tests
solved
Fitness Functions
GA BaselineSequential FFN Solved more
testsParallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)Constructed 3 of 5 tests
solved
Fitness FunctionsGA Baseline
Sequential FFN Solved moretests
Parallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)Constructed 3 of 5 tests
solved
Fitness FunctionsGA Baseline
Sequential FFN Solved moretests
Parallel FFN Total failure
Page 21 | 26
ResultsUseCases
x*2 always solved,except ParallelFFN
Sqrt return 2Fibonacci only first 2
(Sequential FFN)Constructed 3 of 5 tests
solved
Fitness FunctionsGA Baseline
Sequential FFN Solved moretests
Parallel FFN Total failure
Page 21 | 26
OutlineMotivationBackgroundMethodology - Analyze TestdataMethodology - Dynamic EvaluationResultsDiscussion & Outlook
Page 22 | 26
Discussion IConcepts show promise
– Sequential FFN outperformed GA– Known why all solutions failed– Can be improved
Stuck in local optima– Sequential - solutions dominate from previous
– Additive mutator– Parallel - one previous repeat dominates
– Merge crossover
Page 23 | 26
Discussion IICorrectness measurement is binary 0|1
– Increase amount of tests– Switch correctness measure from 0|1 to 0..1
Search Space Size may be too big– Use indicators from original AST
– Operators / Operands– Types
– Restrict size with loop/branch counting
Page 24 | 26
Outlook
– Sequential FFN -> Age Layered Population Structure (ALPS)– Parallel FFN -> Island GA– Implement Mutator & Crossover Operators– Update FFN to also include hot path– Your Suggestion Here
Page 25 | 26
Bibliography I[1] T. Würthinger, C. Wimmer, A. Wöß, L. Stadler, G. Duboscq, C. Humer,
G. Richards, D. Simon, and M. Wolczko, “One vm to rule them all”, inProceedings of the 2013 ACM International Symposium on New Ideas, NewParadigms, and Reflections on Programming & Software, ser. Onward! 2013,Indianapolis, Indiana, USA: ACM, 2013, pp. 187–204. [Online]. Available:http://doi.acm.org/10.1145/2509578.2509581.
[2] ISO, ISO/IEC 9899:2011 Information technology — Programming languages— C. Geneva, Switzerland: International Organization for Standardization,Dec. 2011, 683 (est.) [Online]. Available: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=57853.
Page 1 | 7
Bibliography II[3] C. Wimmer and T. Würthinger, “Truffle: A self-optimizing runtime system”,
in Proceedings of the 3rd Annual Conference on Systems, Programming,and Applications: Software for Humanity, ser. SPLASH ’12, Tucson, Arizona,USA: ACM, 2012, pp. 13–14. [Online]. Available:http://doi.acm.org/10.1145/2384716.2384723.
[4] OpenJDK. (2018), Graal project, Last Accessed - 2018-04-23, [Online].Available: http://openjdk.java.net/projects/graal/.
[5] D. Simon, C. Wimmer, B. Urban, G. Duboscq, L. Stadler, and T. Würthinger,“Snippets: Taking the high road to a low level”, ACM Trans. Archit. CodeOptim., vol. 12, no. 2, 20:20:1–20:20:25, Jun. 2015. [Online]. Available:http://doi.acm.org/10.1145/2764907.
Page 2 | 7
Bibliography III[6] L. Stadler, G. Duboscq, H. Mössenböck, and T. Würthinger, “Compilation
queuing and graph caching for dynamic compilers”, in Proceedings of theSixth ACM Workshop on Virtual Machines and Intermediate Languages,ser. VMIL ’12, Tucson, Arizona, USA: ACM, 2012, pp. 49–58. [Online].Available: http://doi.acm.org/10.1145/2414740.2414750.
[7] R. Olsen and B. Ostman, Using fitness landscapes to visualize evolution inaction, https://www.youtube.com/watch?v=4pdiAneMMhU, [LastAccessed 14.07.2018], 2014.
Page 3 | 7
Appendix A - Run ConfigurationCrossover A single point crossover was selected
Mutator A single point mutator was selected. The point to bemutated is selected randomly by the mutator.
Mutation Probability was set to 57%Selector crossover and mutator use a tournament selection
PopulationSize was set to 200 individualsGenerations each run was 20 generations. GA, and the sequential
FFN took 80 generations over 4 restarts. Parallel FFNtook 60 generations initially and 20 generations afterrestart = 80 generations.
Elitism per generation the 3 best individuals were preservedRestart-Elitism 10 best individuals. Parallel FFN had 30 individuals
carried over (10 from each group).Operators and Operands all AST nodes of the MiniC language were
selected as operators and operands for each test case.Page 4 | 7
Appendix B - Constructed Use CaseListing 1: Constructed Use Case
int constructed(int n) {if (n == 0) {
return 1;}if (n == 1) {
return 2;}if (n == 2) {
return n + 1;}return n * n;
}
Page 5 | 7
Appendix C - Threats to Validity
Performance Measurements - Graal Specific– Runs in same JVM influence each following run– 200.000 runs for measurement due to warm up phase
Page 6 | 7
Appendix D - Fitness Landscape Vi-sualization
Figure: Regular Static FitnessLandscape [7]
Figure: Test-based FitnessLandscape
Page 7 | 7