algorithm selection and portfolios · 2016-06-24 · performance differences 0.1 1 10 100 1000 0.1...
TRANSCRIPT
![Page 1: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/1.jpg)
Algorithm Selection andPortfolios
(and Algorithm Configuration)
Lars KotthoffUniversity of British Columbia
ACP Summer School, Cork, June 20-24 2016
1 / 85
![Page 2: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/2.jpg)
Outline
▷ Motivation▷ Algorithm Configuration▷ Algorithm Configuration Exercises (after break)▷ Algorithm Selection and Portfolios (tomorrow morning)
2 / 85
![Page 3: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/3.jpg)
Big Picture
▷ advance the state of the art through meta-algorithmictechniques
▷ rather than inventing new things, use existing things better
3 / 85
![Page 4: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/4.jpg)
Prominent Application
Fréchette, Alexandre, Neil Newman, Kevin Leyton-Brown. “Solving theStation Packing Problem.” In Association for the Advancement of ArtificialIntelligence (AAAI), 2016.
4 / 85
![Page 5: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/5.jpg)
Performance Differences
0.1
1
10
100
1000
0.1 1 10 100 1000
Virt
ual B
est S
AT
Virtual Best CSP
Hurley, Barry, Lars Kotthoff, Yuri Malitsky, and Barry O’Sullivan. “Proteus:A Hierarchical Portfolio of Solvers and Transformations.” In CPAIOR, 2014.
5 / 85
![Page 6: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/6.jpg)
Leveraging the Differences
Xu, Lin, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown.“SATzilla: Portfolio-Based Algorithm Selection for SAT.” J. Artif. Intell. Res.(JAIR) 32 (2008): 565–606.
6 / 85
![Page 7: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/7.jpg)
Performance Improvements
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
Ran FocusedILS, 2 days × 10 machines
– On a training set from each benchmark
Compared to manually-engineered default
– 1 week of performance tuning
– Competitive with the state of the art
– Comparison on unseen test instances
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, optim
ized for
IBM
−B
MC
(s)
4.5-fold speedup
on hardware verification
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, optim
ized for
SW
V (
s)
500-fold speedup won category
QF BV in 2007 SMT competitionHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 17
Hutter, Frank, Domagoj Babic, Holger H. Hoos, and Alan J. Hu.“Boosting Verification by Automatic Tuning of Decision Procedures.” InFMCAD ’07: Proceedings of the Formal Methods in Computer Aided Design,27–34. Washington, DC, USA: IEEE Computer Society, 2007. 7 / 85
![Page 8: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/8.jpg)
Algorithm Configuration
8 / 85
![Page 9: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/9.jpg)
Algorithm Configuration
Given a (set of) problem(s), find the best parameter configuration.
9 / 85
![Page 10: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/10.jpg)
Parameter Configurations?
▷ anything you can change▷ e.g. search heuristic, variable ordering, type of global
constraint decomposition▷ some will affect performance, others will have no effect at all
10 / 85
![Page 11: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/11.jpg)
Examples
▷ Spear SAT solver, 26 parameters▷ CPLEX MIP solver, 76 parameters▷ WEKA machine learning package, 3 feature search methods, 8
feature evaluators, 39 classifiers (of which 12 can be combinedwith other classifiers)…
11 / 85
![Page 12: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/12.jpg)
Automated Algorithm Configuration
▷ no background knowledge on parameters▷ algorithm treated as a “black box”▷ as little manual intervention as possible
12 / 85
![Page 13: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/13.jpg)
Motivation
“Unlike our human subjects, [the system] experimentedwith a wide variety of combinations of heuristics. Ourhuman subjects rarely had the inclination or patience totry many alternatives, and on at least one occasionincorrectly evaluated alternatives that they did try […].”
Minton, Steven. “Automatically Configuring Constraint SatisfactionPrograms: A Case Study.” Constraints 1 (1996): 7–43.
13 / 85
![Page 14: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/14.jpg)
Algorithm Configuration
Frank Hutter and Marius Lindauer, “Algorithm Configuration: A Hands onTutorial”, AAAI 2016
14 / 85
![Page 15: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/15.jpg)
In Context
Frank Hutter and Marius Lindauer, “Algorithm Configuration: A Hands onTutorial”, AAAI 2016
15 / 85
![Page 16: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/16.jpg)
Parameter Spaces
▷ numeric – 1, 2, 3…▷ ordinal – a, b, c…▷ categoric – ACP, UBC, UCC…▷ conditional dependencies – e.g. if A is 1, B can’t be 2, C is
only active if A is >2→ not every tool suitable for every type of parameter
16 / 85
![Page 17: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/17.jpg)
General Approach
▷ evaluate algorithm as black box function▷ observe effect of parameters without knowing the inner
workings▷ balance diversification/exploration and
intensification/exploitation
17 / 85
![Page 18: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/18.jpg)
Choosing Instances
▷ we want configurations that generalise, i.e. good for morethan one instance
▷ similar problem to machine learning – want models thatgeneralise
▷ split instances into training set (which we configure on) andtest set (which we only evaluate performance on)
▷ need to balance easy/hard instances in both sets
18 / 85
![Page 19: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/19.jpg)
When are we done?
▷ most approaches incomplete▷ cannot prove optimality, not guaranteed to find optimal
solution (with finite time)▷ performance highly dependent on configuration space
→ How do we know when to stop?
19 / 85
![Page 20: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/20.jpg)
Time Budget
How much time/how many function evaluations?▷ too much → wasted resources▷ too little → suboptimal result▷ use statistical tests▷ evaluate on parts of the data▷ for runtime: adaptive capping
20 / 85
![Page 21: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/21.jpg)
Grid and Random Search
▷ evaluate certain points in parameter space
Bergstra, James, and Yoshua Bengio. “Random Search forHyper-Parameter Optimization.” J. Mach. Learn. Res. 13, no. 1 (February2012): 281–305.
21 / 85
![Page 22: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/22.jpg)
Population-Based Methods
▷ e.g. Racing and Genetic Algorithms▷ start with population of random configurations▷ eliminate “weak” individuals▷ generate new population from “strong” individuals▷ iterate
Birattari, Mauro, Zhi Yuan, Prasanna Balaprakash, and Thomas Stützle.“F-Race and Iterated F-Race: An Overview.” In Experimental Methods for theAnalysis of Optimization Algorithms, 311–36. 2010.
Ansótegui, Carlos, Meinolf Sellmann, and Kevin Tierney. “A Gender-BasedGenetic Algorithm for the Automatic Configuration of Algorithms.” In CP,142–57, 2009.
22 / 85
![Page 23: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/23.jpg)
Local Search
▷ start with random configuration▷ change a single parameter (local search step)▷ if better, keep the change, else revert▷ repeat▷ optional (but important): restart with new random
configurations
Hutter, Frank, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle.“ParamILS: An Automatic Algorithm Configuration Framework.” J. Artif. Int.Res. 36, no. 1 (2009): 267–306.
23 / 85
![Page 24: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/24.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
24 / 85
![Page 25: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/25.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
25 / 85
![Page 26: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/26.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
26 / 85
![Page 27: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/27.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
27 / 85
![Page 28: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/28.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Perturbation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
28 / 85
![Page 29: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/29.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
29 / 85
![Page 30: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/30.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
30 / 85
![Page 31: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/31.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
31 / 85
![Page 32: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/32.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
?
Selection (using Acceptance Criterion)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
32 / 85
![Page 33: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/33.jpg)
Model-Based Search
▷ build model of parameter-response surface▷ allows targeted exploration of new configurations▷ (can take instance features into account like algorithm
selection)
Hutter, Frank, Holger H. Hoos, and Kevin Leyton-Brown. “SequentialModel-Based Optimization for General Algorithm Configuration.” In LION 5,507–23, 2011.
33 / 85
![Page 34: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/34.jpg)
Model-Based Search Example
● ●
●
●
●
●
●
●
●
●0.00
0.25
0.50
0.75
0.000
0.025
0.050
0.075
yei
0.00 0.25 0.50 0.75 1.00x
type
● init
prop
type
y
yhat
ei
Iter = 1, Gap = 0.0000e+00
graphics by Bernd Bischl with mlrMBO R package 34 / 85
![Page 35: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/35.jpg)
Model-Based Search Example
● ●
●
●
●
●
●
●
●
●0.0
0.2
0.4
0.6
0.000
0.005
0.010
0.015
0.020
yei
0.00 0.25 0.50 0.75 1.00x
type
● init
prop
seq
type
y
yhat
ei
Iter = 6, Gap = 0.0000e+00
graphics by Bernd Bischl with mlrMBO R package 35 / 85
![Page 36: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/36.jpg)
Model-Based Search Example
● ●
●
●
●
●
●
●
●
●0.0
0.2
0.4
0.6
0.000
0.002
0.004
0.006
0.008
yei
0.00 0.25 0.50 0.75 1.00x
type
● init
prop
seq
type
y
yhat
ei
Iter = 13, Gap = 0.0000e+00
graphics by Bernd Bischl with mlrMBO R package 36 / 85
![Page 37: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/37.jpg)
Model-Based Search Example
● ●
●
●
●
●
●
●
●
●0.0
0.2
0.4
0.6
0.0000
0.0005
0.0010
0.0015
0.0020
0.0025
yei
0.00 0.25 0.50 0.75 1.00x
type
● init
prop
seq
type
y
yhat
ei
Iter = 20, Gap = 0.0000e+00
graphics by Bernd Bischl with mlrMBO R package 37 / 85
![Page 38: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/38.jpg)
Practical Considerations
▷ poorly-specified parameter spaces▷ incorrect results with some configurations▷ crashes▷ instances to configure on?▷ do not configure random seeds!
38 / 85
![Page 39: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/39.jpg)
Overtuning
▷ similar to overfitting in machine learning▷ performance improves on training instances, but not on test
instances▷ configuration is too “tailored”, e.g. specific to satisfiable
instances
39 / 85
![Page 40: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/40.jpg)
Analysis of Results
▷ ablation analysishttp://www.cs.ubc.ca/labs/beta/Projects/Ablation/
▷ functional ANOVA http://www.automl.org/fanova.html
40 / 85
![Page 41: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/41.jpg)
Tools and Resources
HPOlib http://www.automl.org/hpolib.htmliRace http://iridia.ulb.ac.be/irace/
mlrMBO https://github.com/mlr-org/mlrMBOSMAC http://www.cs.ubc.ca/labs/beta/Projects/SMAC/
Spearmint https://github.com/HIPS/SpearmintTPE https://jaberg.github.io/hyperopt/
Auto-WEKA http://www.cs.ubc.ca/labs/beta/Projects/autoweka/
Auto-sklearn https://github.com/automl/auto-sklearn
41 / 85
![Page 42: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/42.jpg)
Configurable SAT Solver Challenge
▷ http://aclib.net/cssc2014/▷ idea: make solvers as configurable as possible, let the machine
do the rest▷ avoids spurious results because of different configurations▷ mitigates impact of instance bias
42 / 85
![Page 43: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/43.jpg)
Algorithm Configuration Exercises
▷ install Numberjack (if you haven’t already)▷ install SMAC▷ get
http://www.cs.ubc.ca/~larsko/numberjack-tuning.tar.gzparams.pcs parameter space definitionwrapper.py wrapper that takes parameter definitions from
SMAC and outputs result for SMACNQueens.py actual Numberjack modelinstances-* problem instance (sizes) to run onscenario.txt SMAC configuration file
▷ configure!▷ smac --scenario scenario.txt
43 / 85
![Page 44: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/44.jpg)
When it’s working…
▷ check correctness in wrapper▷ find harder problem instances▷ define additional parameters▷ anything else you can think of!
44 / 85
![Page 45: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/45.jpg)
Algorithm Selection
45 / 85
![Page 46: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/46.jpg)
Motivation
Remember overtuning? What if we could leverage this?
46 / 85
![Page 47: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/47.jpg)
Motivation▷ different configurations (algorithms) good on different
instances▷ have a portfolio of different configurations/algorithms and a
selector
0.1
1
10
100
1000
0.1 1 10 100 1000
Virt
ual B
est S
AT
Virtual Best CSP
Hurley, Barry, Lars Kotthoff, Yuri Malitsky, and Barry O’Sullivan. “Proteus:A Hierarchical Portfolio of Solvers and Transformations.” In CPAIOR, 2014.
47 / 85
![Page 48: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/48.jpg)
Algorithm Selection
Given a problem, choose the best algorithm to solve it.
48 / 85
![Page 49: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/49.jpg)
Original Model
x ∈ PProblem space
A ∈ AAlgorithm space
p ∈ Rn
Performancemeasure space
‖p‖ = Algorithmperformance
S(x)
Selectionmapping
p(A,x)
Performancemapping
Normmapping
Rice, John R. “The Algorithm Selection Problem.” Advances in Computers15 (1976): 65–118.
49 / 85
![Page 50: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/50.jpg)
Contemporary Model
x ∈ P
Problem spaceA ∈ A
Algorithm space
S
Selection modelS(x) = A
Prediction
p ∈ Rn
Performancemeasure
∀x ∈ P′⊂ P ,
A ∈ A :p(A,x)
Training data
Featureextraction
Feedback
50 / 85
![Page 51: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/51.jpg)
Portfolios
▷ instead of a single algorithm, use several complementaryalgorithms
▷ idea from Economics – minimise risk by spreading it outacross several securities
▷ same for computational problems – minimise risk of algorithmperforming poorly
▷ in practice often constructed from competition winners
Huberman, Bernardo A., Rajan M. Lukose, and Tad Hogg. “An EconomicsApproach to Hard Computational Problems.” Science 275, no. 5296 (1997):51–54. doi:10.1126/science.275.5296.51.
51 / 85
![Page 52: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/52.jpg)
Evaluation of Portfolios
▷ single best algorithm▷ algorithm with the best performance across all instances▷ lower bound for performance of portfolio – hopefully we are
better!▷ virtual best algorithm
▷ choose the best algorithm for each instance▷ corresponds to oracle predictor or overhead-free parallel
portfolio▷ upper bound on portfolio performance (assuming robust
performance measurements) – note difference to configuration
52 / 85
![Page 53: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/53.jpg)
Contributions of Algorithms to Portfolios▷ stand-alone performance – ignore that there is a portfolio▷ marginal contribution – how much does the algorithm add to
the portfolio containing all other algorithms?▷ Shapley value – how much does the algorithm add on average
to all possible portfolios?
SATzilla2009_R−2009−03−22 435.05627
TNM−2009−03−22 379.04912gnovelty+2−2009−03−22 355.04639
hybridGM3−3 340.04505
adaptg2wsat2009++−2009−03−23 338.04435
T07 reference solver: gnovelty+−2007−02−08 318.04166
gNovelty+−T−2009−03−22 314.04094
march_hi−hi 313.03918
T07 reference solver: SATzilla−RANDOM 308.03953
T07 reference solver: March KS−2007−02−08 308.03854T07 reference solver: adaptg2wsat+−2007−02−08 298.03911
iPAWS−2009−03−22 288.03806
63.32536
55.25959
44.50843
49.234
39.62439
35.03323
33.73186
57.89701
50.13046
52.41559
32.79246
30.11838
9e−05
3.00046
2.00041
5.00122
1.00019
0
2e−05
4.00051
0
0.00011
1.00011
5e−05
Standalone performance Shapley value Marginal contribution
Fréchette, Alexandre, Lars Kotthoff, Talal Rahwan, Holger H. Hoos, KevinLeyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to AnalyzeAlgorithm Portfolios.” In 30th AAAI Conference on Artificial Intelligence, 2016.
53 / 85
![Page 54: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/54.jpg)
Parallel Porfolios
Why not simply run all algorithms in parallel?▷ not enough resources may be available / waste of resources▷ algorithms may be parallelized themselves▷ memory contention▷ …
54 / 85
![Page 55: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/55.jpg)
Key Components of an Algorithm Selection System
▷ feature extraction▷ performance model▷ prediction-based selector/scheduler
optional:▷ presolver▷ secondary/hierarchical models and predictors (e.g. for feature
extraction time)
55 / 85
![Page 56: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/56.jpg)
Features
▷ relate properties of problem instances to performance▷ relatively cheap to compute▷ specified by domain expert▷ syntactic – analyse instance description▷ probing – run algorithm for short time▷ dynamic – instance changes while algorithm is running
56 / 85
![Page 57: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/57.jpg)
Syntactic Features
▷ number of variables, number of clauses/constraints/…▷ ratios▷ order of variables/values▷ clause/constraints–variable graph or variable graph:
▷ node degrees▷ connectivity▷ clustering coefficient▷ …
▷ …
57 / 85
![Page 58: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/58.jpg)
Probing Features
▷ number of nodes/propagations within time limit▷ estimate of search space size▷ tightness of problem/constraints▷ …
58 / 85
![Page 59: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/59.jpg)
Dynamic Features
▷ change of variable domains▷ number of constraint propagations▷ number of failures a clause participated in▷ …
59 / 85
![Page 60: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/60.jpg)
What Features do we need in Practice?
▷ trade-off between complex features and complex models▷ in practice, very simple features (e.g. problem size) can
perform well
60 / 85
![Page 61: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/61.jpg)
Types of Performance Models
▷ models for entire portfolios▷ models for individual algorithms▷ models that are somewhere in between (e.g. pairs of
algorithms)
61 / 85
![Page 62: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/62.jpg)
Models for Entire Portfolios▷ predict the best algorithm in the portfolio▷ alternatively: cluster and assign best algorithms to clusters
optional (but important):▷ attach a “weight” during learning (e.g. the difference between
best and worst solver) to bias model towards the “important”instances
▷ special loss metric
Gent, Ian P., Christopher A. Jefferson, Lars Kotthoff, Ian Miguel, NeilMoore, Peter Nightingale, and Karen E. Petrie. “Learning When to Use LazyLearning in Constraint Solving.” In 19th European Conference on ArtificialIntelligence, 873–78, 2010.
Kadioglu, Serdar, Yuri Malitsky, Meinolf Sellmann, and Kevin Tierney.“ISAC – Instance-Specific Algorithm Configuration.” In 19th EuropeanConference on Artificial Intelligence, 751–56, 2010.
62 / 85
![Page 63: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/63.jpg)
Models for Individual Algorithms
▷ predict the performance for each algorithm separately▷ combine the predictions to choose the best one▷ for example: predict the runtime for each algorithm, choose
the one with the lowest runtime
Xu, Lin, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown.“SATzilla: Portfolio-Based Algorithm Selection for SAT.” J. Artif. Intell. Res.(JAIR) 32 (2008): 565–606.
63 / 85
![Page 64: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/64.jpg)
Hybrid Models
▷ get the best of both worlds▷ for example: consider pairs of algorithms to take relations into
account▷ for each pair of algorithms, learn model that predicts which
one is faster
Xu, Lin, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown.“Hydra-MIP: Automated Algorithm Configuration and Selection for MixedInteger Programming.” In RCRA Workshop on Experimental Evaluation ofAlgorithms for Solving Problems with Combinatorial Explosion at theInternational Joint Conference on Artificial Intelligence (IJCAI), 16–30, 2011.
Kotthoff, Lars. “Hybrid Regression-Classification Models for AlgorithmSelection.” In 20th European Conference on Artificial Intelligence, 480–85,2012.
64 / 85
![Page 65: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/65.jpg)
Overview – Machine Learning
classification regression clustering
Algorithm 1
Algorithm 2Algorithm 2
Algorithm 1
Algorithm 2
Algorithm 1
65 / 85
![Page 66: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/66.jpg)
Types of Predictions/Algorithm Selectors
▷ best algorithm▷ n best algorithms ranked▷ allocation of resources to n algorithms▷ change the currently running algorithm?
Kotthoff, Lars. “Ranking Algorithms by Performance.” In LION 8, 2014.
Kadioglu, Serdar, Yuri Malitsky, Ashish Sabharwal, Horst Samulowitz, andMeinolf Sellmann. “Algorithm Selection and Scheduling.” In 17th InternationalConference on Principles and Practice of Constraint Programming, 454–69,2011.
Stergiou, Kostas. “Heuristics for Dynamically Adapting Propagation inConstraint Satisfaction Problems.” AI Commun. 22, no. 3 (2009): 125–41.
66 / 85
![Page 67: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/67.jpg)
Time of Prediction
before problem is being solved▷ select algorithm(s) once▷ no recourse if predictions are bad
while problem is being solved▷ continuously monitor problem features and/or performance▷ can remedy bad initial choice or react to changing problem
67 / 85
![Page 68: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/68.jpg)
Types of Machine Learning
Lots!
▷ depends on the type of prediction to make (e.g. label –classification, number – regression)
▷ in general, all kinds of machine learning applicable▷ good results in practice with methods that use some kind of
meta-learning, e.g. random forests
68 / 85
![Page 69: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/69.jpg)
Example System – SATzilla
▷ 7 SAT solvers, 4811 problem instances▷ syntactic (33) and probing features (15)▷ ridge regression to predict log runtime for each solver, choose
the solver with the best predicted performance▷ later version uses random forests to predict better algorithm
for each pair, aggregation through simple voting scheme▷ pre-solving, feature computation time prediction, hierarchical
model▷ won several competitions
Xu, Lin, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown.“SATzilla: Portfolio-Based Algorithm Selection for SAT.” J. Artif. Intell. Res.(JAIR) 32 (2008): 565–606.
69 / 85
![Page 70: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/70.jpg)
Putting Configuration and Selection together – Hydra
▷ find best configuration▷ find configuration that complements existing configuration
best▷ iterate▷ stop when no further improvement
Xu, Lin, Holger H. Hoos, and Kevin Leyton-Brown. “Hydra: AutomaticallyConfiguring Algorithms for Portfolio-Based Selection.” In Twenty-FourthConference of the Association for the Advancement of Artificial Intelligence(AAAI-10), 210–16, 2010.
70 / 85
![Page 71: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/71.jpg)
Benchmark library – ASlib
▷ https://github.com/coseal/aslib_data▷ currently 17 data sets/scenarios with more in preparation▷ SAT, CSP, QBF, ASP, MAXSAT, OR▷ includes data used frequently in the literature that you may
want to evaluate your approach on▷ more scenarios in the pipeline▷ http://aslib.net
Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, YuriMalitsky, Alexandre Fréchette, Holger Hoos, Frank Hutter, KevinLeyton-Brown, Kevin Tierney, and Joaquin Vanschoren. “ASlib: A BenchmarkLibrary for Algorithm Selection.” To appear in Artificial Intelligence Journal.
71 / 85
![Page 72: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/72.jpg)
EDA – Overview of algorithm performance▷ runstatus▷ performance plots▷ (cumulative) density plots▷ scatter plot for pairs of algorithms▷ algorithm performance correlation
72 / 85
![Page 73: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/73.jpg)
EDA – Selector performance comparison
▷ (basic) classification, regression, clustering selectors▷ different machine learning techniques▷ comparison to virtual best and single best
73 / 85
![Page 74: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/74.jpg)
Algorithm Selection Challenge
▷ http://challenge.icon-fet.eu/▷ ran on ASlib data▷ all submissions and evaluation available
74 / 85
![Page 75: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/75.jpg)
Tools
autofolio https://bitbucket.org/mlindauer/autofolio/LLAMA https://bitbucket.org/lkotthoff/llamaSATzilla http://www.cs.ubc.ca/labs/beta/Projects/SATzilla/
75 / 85
![Page 76: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/76.jpg)
(Much) More Information
http://larskotthoff.github.io/assurvey/
Kotthoff, Lars. “Algorithm Selection for Combinatorial Search Problems: ASurvey.” AI Magazine 35, no. 3 (2014): 48–60.
76 / 85
![Page 77: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/77.jpg)
Demo(ish) – LLAMA
77 / 85
![Page 78: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/78.jpg)
LLAMA
▷ R package▷ access to most machine learning methods in R▷ implements the most common approaches from the literature
and a few extra ones▷ open source
78 / 85
![Page 79: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/79.jpg)
30 second tutorial – LLAMA and ASlib
library(llama)library(aslib)
scenario = parseASScenario("QBF -2011/")data = convertToLlamaCVFolds(scenario)
learner = makeLearner("classif.randomForest")model = classify(learner , data)
mean(parscores(data, model))> 9774.813mean(parscores(data, vbs))> 8337.099mean(parscores(data, singleBest))> 15228.53
79 / 85
![Page 80: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/80.jpg)
Getting Started – LLAMA
library(llama)
data(satsolvers)
folds = cvFolds(satsolvers)
learner = makeLearner("classif.JRip")model = classify(learner , folds)
mean(parscores(folds , model))> 5611.417mean(parscores(satsolvers , vbs))> 4645.169mean(parscores(satsolvers , singleBest))> 5779.526
80 / 85
![Page 81: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/81.jpg)
Example Data
▷ 19 SAT solvers▷ 2433 instances▷ 36 features
81 / 85
![Page 82: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/82.jpg)
Under the Hoodmodel$models[[1]]$learner.modelJRIP rules:===========
(dyn_log_propags <= 3.169925) and (log_ranges >=3.321928) and (sqrt_avg_domsize >= 3.162278) and(sqrt_avg_domsize <= 3.162278) => target=riss(12.0/0.0)
(log_values <= 8.366322) and (log_values >= 8.366322)and (dyn_log_nodes <= 6.066089) =>target=MPhaseSAT64 (17.0/6.0)
(dyn_log_propags >= 23.653658) and (log_values <=9.321928) => target=march_rw (253.0/108.0)
(dyn_log_avg_weight <= 5.149405) and (percent_global <=0.079239) and (sqrt_avg_domsize >= 3.872983) and(dyn_log_stdev_weight >= 3.961314) and(dyn_log_nodes <= 8.005625) => target=glucose(33.0/3.0)
(percent_global <= 0.048745) and (dyn_log_avg_weight <=5.931328) and (sqrt_avg_domsize >= 3.872983) and(dyn_log_nodes >= 5) and (dyn_log_propags <=14.391042) => target=glucose (42.0/10.0)
(sqrt_avg_domsize <= 2.475884) and (log_constraints >=11.920353) and (dyn_log_propags <= 21.202935) and(log_bits >= 7.459432) => target=picosat (41.0/6.0)
(dyn_log_avg_weight <= 4.766713) and (log_constraints>= 8.686501) and (dyn_log_nodes >= 2.807355) and(dyn_log_propags <= 14.038405) => target=picosat(59.0/23.0)
(dyn_log_avg_weight <= 5.422233) and (log_lists <=6.72792) and (dyn_log_nodes >= 6) and (log_bits >=6.459432) => target=picosat (24.0/5.0)
(percent_global >= 6.1207) and (dyn_log_nodes >=5.285402) and (percent_avg_continuity >= 44.27353)and (dyn_log_propags <= 20.543671) => target=picosat(23.0/6.0)
(log_values >= 10.643856) and (log_bits <= -1) and(dyn_log_avg_weight >= 7.939873) and(log_constraints <= 6.686501) =>target=cryptominisat (144.0/34.0)
(log_lists >= 8.214319) and (percent_global <=0.022831) and (dyn_log_nodes >= 6.459432) and(log_constraints >= 12.095067) =>target=cryptominisat (19.0/3.0)
(log_lists >= 8.214319) and (log_constraints <=12.005975) => target=cryptominisat (63.0/28.0)
(log_constraints <= 7.491853) and (log_values >=10.643856) and (dyn_log_avg_weight >= 11.663147) =>target=cryptominisat (52.0/16.0)
=> target=clasp (1651.0/1161.0)
Number of Rules : 14
82 / 85
![Page 83: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/83.jpg)
Other Modelsrflearner = makeLearner("classif.randomForest")rfmodel = classify(rflearner , folds)mean(parscores(folds , rfmodel))> 5598.554
rplearner = makeLearner("classif.rpart")rpmodel = classify(rplearner , folds)mean(parscores(folds , rpmodel))> 5561.635
# this will take a long time...rfrlearner = makeLearner("regr.randomForest")rfrmodel = regression(rfrlearner , folds)mean(parscores(folds , rfrmodel))> 5689.075
rfpmodel = classifyPairs(rflearner , folds)mean(parscores(folds , rfpmodel))> 5945.246
83 / 85
![Page 84: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/84.jpg)
Summary
Algorithm Selection choose the best algorithm for solving aproblem
Algorithm Configuration choose the best parameter configurationfor solving a problem with an algorithm
▷ mature research areas▷ can combine configuration and selection▷ effective tools are available
84 / 85
![Page 85: Algorithm Selection and Portfolios · 2016-06-24 · Performance Differences 0.1 1 10 100 1000 0.1 1 10 100 1000 Virtual Best SAT Virtual Best CSP Hurley, Barry, Lars Kotthoff, Yuri](https://reader036.vdocuments.us/reader036/viewer/2022070813/5f0c74337e708231d4357af8/html5/thumbnails/85.jpg)
COSEAL
Conf
igur
ation
and Selection of Algorithms
CO S E AL
Interested? Join the COSEAL group!
http://coseal.net 85 / 85