the black box optimization competition · is something new, and hopefully a lot of fun. tobias...

The True Black Box

Optimization Competition

NumBBO WorkshopMonday November 24, 2014

Tobias GlasmachersInstitut fur NeuroinformatikRuhr-Universitat Bochum, Germany

Joint work with Ilya Loshchilov

Tobias Glasmachers The Black Box Optimization Competition 1

Introduction

What is this about?

The “Black Box Optimization Competition”(BBComp)

is the first platform for a testing blackbox optimizers under realistic conditions:

the objective function is truly a black-box.

What does that mean? How is it differ from existingbenchmark platforms?

Wow does it work?

(Why) do we need this?

Tobias Glasmachers The Black Box Optimization Competition 2

Introduction

What is this about?

The “Black Box Optimization Competition”(BBComp)

is the first platform for a testing blackbox optimizers under realistic conditions:

the objective function is truly a black-box.

What does that mean? How is it differ from existingbenchmark platforms?

Wow does it work?

(Why) do we need this?

Tobias Glasmachers The Black Box Optimization Competition 2

Introduction

What is this about?

The “Black Box Optimization Competition”(BBComp)

is the first platform for a testing blackbox optimizers under realistic conditions:

the objective function is truly a black-box.

What does that mean? How is it differ from existingbenchmark platforms?

Wow does it work?

(Why) do we need this?

Tobias Glasmachers The Black Box Optimization Competition 2

Introduction

What is this about?

The “Black Box Optimization Competition”(BBComp)

is the first platform for a testing blackbox optimizers under realistic conditions:

the objective function is truly a black-box.

What does that mean? How is it differ from existingbenchmark platforms?

Wow does it work?

(Why) do we need this?

Tobias Glasmachers The Black Box Optimization Competition 2

Introduction

What is this about?

The “Black Box Optimization Competition”(BBComp)

is the first platform for a testing blackbox optimizers under realistic conditions:

the objective function is truly a black-box.

What does that mean? How is it differ from existingbenchmark platforms?

Wow does it work?

(Why) do we need this?

Tobias Glasmachers The Black Box Optimization Competition 2

Introduction

Platforms for benchmarking black box optimizers do exist:BBOB/COCO, CEC competition.

General proceeding: the problems to be optimized are public,you run the algorithm yourself and only submit a performancesummary.

You are expected to play fair. Of course, after all this isscience, not competition, right?

The hope is that the benchmark collection is sufficientlygeneral and diverse.

In particular, knowing the problems in advance should not bea (significant) advantage.

Tobias Glasmachers The Black Box Optimization Competition 3

Introduction

Platforms for benchmarking black box optimizers do exist:BBOB/COCO, CEC competition.

General proceeding: the problems to be optimized are public,you run the algorithm yourself and only submit a performancesummary.

You are expected to play fair. Of course, after all this isscience, not competition, right?

The hope is that the benchmark collection is sufficientlygeneral and diverse.

In particular, knowing the problems in advance should not bea (significant) advantage.

Tobias Glasmachers The Black Box Optimization Competition 3

Introduction

Platforms for benchmarking black box optimizers do exist:BBOB/COCO, CEC competition.

General proceeding: the problems to be optimized are public,you run the algorithm yourself and only submit a performancesummary.

You are expected to play fair. Of course, after all this isscience, not competition, right?

The hope is that the benchmark collection is sufficientlygeneral and diverse.

In particular, knowing the problems in advance should not bea (significant) advantage.

Tobias Glasmachers The Black Box Optimization Competition 3

Introduction

Platforms for benchmarking black box optimizers do exist:BBOB/COCO, CEC competition.

General proceeding: the problems to be optimized are public,you run the algorithm yourself and only submit a performancesummary.

You are expected to play fair. Of course, after all this isscience, not competition, right?

The hope is that the benchmark collection is sufficientlygeneral and diverse.

In particular, knowing the problems in advance should not bea (significant) advantage.

Tobias Glasmachers The Black Box Optimization Competition 3

Introduction

Platforms for benchmarking black box optimizers do exist:BBOB/COCO, CEC competition.

General proceeding: the problems to be optimized are public,you run the algorithm yourself and only submit a performancesummary.

You are expected to play fair. Of course, after all this isscience, not competition, right?

The hope is that the benchmark collection is sufficientlygeneral and diverse.

In particular, knowing the problems in advance should not bea (significant) advantage.

Tobias Glasmachers The Black Box Optimization Competition 3

Introduction

Is this really true? Or do we already see over-fitting to BBOBfunctions? How can we know?

It is common practice to test algorithms on well-knownfunctions as if these problems were black boxes. However,they are not.

This is good for understanding algorithm behavior. Is it goodfor comparison?

As an extreme example, think of a model based optimizer thattries to fit the BBOB functions as meta models...

Tobias Glasmachers The Black Box Optimization Competition 4

Introduction

Is this really true? Or do we already see over-fitting to BBOBfunctions? How can we know?

It is common practice to test algorithms on well-knownfunctions as if these problems were black boxes. However,they are not.

This is good for understanding algorithm behavior. Is it goodfor comparison?

As an extreme example, think of a model based optimizer thattries to fit the BBOB functions as meta models...

Tobias Glasmachers The Black Box Optimization Competition 4

Introduction

Is this really true? Or do we already see over-fitting to BBOBfunctions? How can we know?

It is common practice to test algorithms on well-knownfunctions as if these problems were black boxes. However,they are not.

This is good for understanding algorithm behavior. Is it goodfor comparison?

As an extreme example, think of a model based optimizer thattries to fit the BBOB functions as meta models...

Tobias Glasmachers The Black Box Optimization Competition 4

Introduction

Is this really true? Or do we already see over-fitting to BBOBfunctions? How can we know?

It is common practice to test algorithms on well-knownfunctions as if these problems were black boxes. However,they are not.

This is good for understanding algorithm behavior. Is it goodfor comparison?

As an extreme example, think of a model based optimizer thattries to fit the BBOB functions as meta models...

Tobias Glasmachers The Black Box Optimization Competition 4

Introduction

How does BBComp differ?

We do not publish our functions!(Please don’t ask me how they look like, it is complicated, and the

function generator is nearly a black box to me.)

We challenge you to actually deal with the black box!

We give you exactly one trial, or in other words, a fixedbudget of black box queries for each function.

Tobias Glasmachers The Black Box Optimization Competition 5

Introduction

How does BBComp differ?

We do not publish our functions!

(Please don’t ask me how they look like, it is complicated, and the

function generator is nearly a black box to me.)

We challenge you to actually deal with the black box!

We give you exactly one trial, or in other words, a fixedbudget of black box queries for each function.

Tobias Glasmachers The Black Box Optimization Competition 5

Introduction

How does BBComp differ?

We do not publish our functions!(Please don’t ask me how they look like, it is complicated, and the

function generator is nearly a black box to me.)

We challenge you to actually deal with the black box!

We give you exactly one trial, or in other words, a fixedbudget of black box queries for each function.

Tobias Glasmachers The Black Box Optimization Competition 5

Introduction

How does BBComp differ?

We do not publish our functions!(Please don’t ask me how they look like, it is complicated, and the

function generator is nearly a black box to me.)

We challenge you to actually deal with the black box!

We give you exactly one trial, or in other words, a fixedbudget of black box queries for each function.

Tobias Glasmachers The Black Box Optimization Competition 5

Introduction

How does BBComp differ?

We do not publish our functions!(Please don’t ask me how they look like, it is complicated, and the

function generator is nearly a black box to me.)

We challenge you to actually deal with the black box!

We give you exactly one trial, or in other words, a fixedbudget of black box queries for each function.

Tobias Glasmachers The Black Box Optimization Competition 5

Introduction

Feel free to do whatever:

use a commercial solver (publishing your code is notmandatory),optimize interactively or even manually (if you truly believe inyourself),run your own algorithm,pick the best you can find in the literature,do online algorithm selection, ...

There are only two rules:1 Don’t try to open the black box, e.g., by reverse engineering

our software or decrypting its network traffic.2 Don’t share function evaluations with colleagues.

Tobias Glasmachers The Black Box Optimization Competition 6

Introduction

Feel free to do whatever:

use a commercial solver (publishing your code is notmandatory),optimize interactively or even manually (if you truly believe inyourself),run your own algorithm,pick the best you can find in the literature,do online algorithm selection, ...

There are only two rules:1 Don’t try to open the black box, e.g., by reverse engineering

our software or decrypting its network traffic.2 Don’t share function evaluations with colleagues.

Tobias Glasmachers The Black Box Optimization Competition 6

Introduction

Hence—just like in real life—you cannot submit 10 differentalgorithm variants and see which performs best (unless yousplit your budget).

This means that we compare participants. BBOB comparesalgorithms.

That’s why BBComp is a competition, not (primarily) abenchmark platform.

We do not aim to replace existing algorithm comparisonplatforms. We have a different setup and different aims. Thisis something new, and hopefully a lot of fun.

Tobias Glasmachers The Black Box Optimization Competition 7

Introduction

Hence—just like in real life—you cannot submit 10 differentalgorithm variants and see which performs best (unless yousplit your budget).

This means that we compare participants. BBOB comparesalgorithms.

That’s why BBComp is a competition, not (primarily) abenchmark platform.

We do not aim to replace existing algorithm comparisonplatforms. We have a different setup and different aims. Thisis something new, and hopefully a lot of fun.

Tobias Glasmachers The Black Box Optimization Competition 7

Introduction

Hence—just like in real life—you cannot submit 10 differentalgorithm variants and see which performs best (unless yousplit your budget).

This means that we compare participants. BBOB comparesalgorithms.

That’s why BBComp is a competition, not (primarily) abenchmark platform.

We do not aim to replace existing algorithm comparisonplatforms. We have a different setup and different aims. Thisis something new, and hopefully a lot of fun.

Tobias Glasmachers The Black Box Optimization Competition 7

Introduction

Hence—just like in real life—you cannot submit 10 differentalgorithm variants and see which performs best (unless yousplit your budget).

This means that we compare participants. BBOB comparesalgorithms.

That’s why BBComp is a competition, not (primarily) abenchmark platform.

We do not aim to replace existing algorithm comparisonplatforms. We have a different setup and different aims. Thisis something new, and hopefully a lot of fun.

Tobias Glasmachers The Black Box Optimization Competition 7

Where to find BBComp?

BBComp has an official website:http://bbcomp.ini.rub.de

It was accepted as a competition to CEC 2015.

It is submitted as a competition to GECCO 2015(expensive setting).

Tobias Glasmachers The Black Box Optimization Competition 8

Where to find BBComp?

BBComp has an official website:http://bbcomp.ini.rub.de

It was accepted as a competition to CEC 2015.

It is submitted as a competition to GECCO 2015(expensive setting).

Tobias Glasmachers The Black Box Optimization Competition 8

Where to find BBComp?

BBComp has an official website:http://bbcomp.ini.rub.de

It was accepted as a competition to CEC 2015.

It is submitted as a competition to GECCO 2015(expensive setting).

Tobias Glasmachers The Black Box Optimization Competition 8

Where to find BBComp?

BBComp has an official website:http://bbcomp.ini.rub.de

It was accepted as a competition to CEC 2015.

It is submitted as a competition to GECCO 2015(expensive setting).

Tobias Glasmachers The Black Box Optimization Competition 8

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

The competition is based on a client server architecture.

Optimization algorithms are clients, the machinebbcomp.ini.rub.de is the server.

Clients don’t ever deal directly with the competition server(they actually cannot since all traffic is encrypted).

Instead they deal with a dynamic link library—this is the blackbox. During optimization it requires an internet connection tocontact the server.

We provide binary versions of this library for Linux, MacOS,and Windows, as well as a plain C header file with theinterfaces.

Tobias Glasmachers The Black Box Optimization Competition 9

Competition Architecture

Each participant needs an account.

Drop me an email and I’ll create one, this takes only a minute.

Then you will be able to log into the system. You can

log into the website,call the login library function and query the black box.

There is a public account with limited credentials:username demoaccount, password demopassword

Feel free to use it for exploration and testing.

Tobias Glasmachers The Black Box Optimization Competition 10

Competition Architecture

Each participant needs an account.

Drop me an email and I’ll create one, this takes only a minute.

Then you will be able to log into the system. You can

log into the website,call the login library function and query the black box.

There is a public account with limited credentials:username demoaccount, password demopassword

Feel free to use it for exploration and testing.

Tobias Glasmachers The Black Box Optimization Competition 10

Competition Architecture

Each participant needs an account.

Drop me an email and I’ll create one, this takes only a minute.

Then you will be able to log into the system. You can

log into the website,call the login library function and query the black box.

There is a public account with limited credentials:username demoaccount, password demopassword

Feel free to use it for exploration and testing.

Tobias Glasmachers The Black Box Optimization Competition 10

Competition Architecture

Each participant needs an account.

Drop me an email and I’ll create one, this takes only a minute.

Then you will be able to log into the system. You can

log into the website,call the login library function and query the black box.

There is a public account with limited credentials:username demoaccount, password demopassword

Feel free to use it for exploration and testing.

Tobias Glasmachers The Black Box Optimization Competition 10

Competition Architecture

Once you have an account you will see multiple tracks.

Each track consists of a number of black box problems.

For each problem you get to know the problem dimension dand the budget B.

For CEC the budget will be high, e.g., 100d2 or 1000d (to bedecided soon).

The feasible region for each problem is [0, 1]d . Querying aninfeasible point will fail, it will not count as a black box query.

Tobias Glasmachers The Black Box Optimization Competition 11

Competition Architecture

Once you have an account you will see multiple tracks.

Each track consists of a number of black box problems.

For each problem you get to know the problem dimension dand the budget B.

For CEC the budget will be high, e.g., 100d2 or 1000d (to bedecided soon).

The feasible region for each problem is [0, 1]d . Querying aninfeasible point will fail, it will not count as a black box query.

Tobias Glasmachers The Black Box Optimization Competition 11

Competition Architecture

Once you have an account you will see multiple tracks.

Each track consists of a number of black box problems.

For each problem you get to know the problem dimension dand the budget B.

For CEC the budget will be high, e.g., 100d2 or 1000d (to bedecided soon).

The feasible region for each problem is [0, 1]d . Querying aninfeasible point will fail, it will not count as a black box query.

Tobias Glasmachers The Black Box Optimization Competition 11

Competition Architecture

Once you have an account you will see multiple tracks.

Each track consists of a number of black box problems.

For each problem you get to know the problem dimension dand the budget B.

For CEC the budget will be high, e.g., 100d2 or 1000d (to bedecided soon).

The feasible region for each problem is [0, 1]d . Querying aninfeasible point will fail, it will not count as a black box query.

Tobias Glasmachers The Black Box Optimization Competition 11

Competition Architecture

Once you have an account you will see multiple tracks.

Each track consists of a number of black box problems.

For each problem you get to know the problem dimension dand the budget B.

For CEC the budget will be high, e.g., 100d2 or 1000d (to bedecided soon).

The feasible region for each problem is [0, 1]d . Querying aninfeasible point will fail, it will not count as a black box query.

Tobias Glasmachers The Black Box Optimization Competition 11

Competition Architecture

Currently there are two tracks: trial and BBComp2015CEC.

The trial track is special. Its budget can be reset arbitrarilyoften. It is meant for debugging client software beforepressing the red button. This is extremely important since youget only one attempt on competition tracks.

The trial track consists of dummy problems. They are verydifferent from competition problems. Don’t even try to tuneyour algorithms to this track, it will be misleading.

The library and the server will make sure that once yourbudget is exceeded you cannot query the black box any more.

The architecture is reasonably fail safe. It will deal withnetwork problems, and even with a power outage (hopefullynot on the server side). You can continue an optimization runany time, and you can even recover all previous functionevaluations.

Tobias Glasmachers The Black Box Optimization Competition 12

Competition Architecture

Currently there are two tracks: trial and BBComp2015CEC.

The trial track is special. Its budget can be reset arbitrarilyoften. It is meant for debugging client software beforepressing the red button. This is extremely important since youget only one attempt on competition tracks.

The trial track consists of dummy problems. They are verydifferent from competition problems. Don’t even try to tuneyour algorithms to this track, it will be misleading.

The library and the server will make sure that once yourbudget is exceeded you cannot query the black box any more.

The architecture is reasonably fail safe. It will deal withnetwork problems, and even with a power outage (hopefullynot on the server side). You can continue an optimization runany time, and you can even recover all previous functionevaluations.

Tobias Glasmachers The Black Box Optimization Competition 12

Competition Architecture

Currently there are two tracks: trial and BBComp2015CEC.

The trial track is special. Its budget can be reset arbitrarilyoften. It is meant for debugging client software beforepressing the red button. This is extremely important since youget only one attempt on competition tracks.

The trial track consists of dummy problems. They are verydifferent from competition problems. Don’t even try to tuneyour algorithms to this track, it will be misleading.

The library and the server will make sure that once yourbudget is exceeded you cannot query the black box any more.

The architecture is reasonably fail safe. It will deal withnetwork problems, and even with a power outage (hopefullynot on the server side). You can continue an optimization runany time, and you can even recover all previous functionevaluations.

Tobias Glasmachers The Black Box Optimization Competition 12

Competition Architecture

Currently there are two tracks: trial and BBComp2015CEC.

The trial track is special. Its budget can be reset arbitrarilyoften. It is meant for debugging client software beforepressing the red button. This is extremely important since youget only one attempt on competition tracks.

The trial track consists of dummy problems. They are verydifferent from competition problems. Don’t even try to tuneyour algorithms to this track, it will be misleading.

The library and the server will make sure that once yourbudget is exceeded you cannot query the black box any more.

The architecture is reasonably fail safe. It will deal withnetwork problems, and even with a power outage (hopefullynot on the server side). You can continue an optimization runany time, and you can even recover all previous functionevaluations.

Tobias Glasmachers The Black Box Optimization Competition 12

Competition Architecture

Currently there are two tracks: trial and BBComp2015CEC.

The trial track is special. Its budget can be reset arbitrarilyoften. It is meant for debugging client software beforepressing the red button. This is extremely important since youget only one attempt on competition tracks.

The trial track consists of dummy problems. They are verydifferent from competition problems. Don’t even try to tuneyour algorithms to this track, it will be misleading.

The library and the server will make sure that once yourbudget is exceeded you cannot query the black box any more.

The architecture is reasonably fail safe. It will deal withnetwork problems, and even with a power outage (hopefullynot on the server side). You can continue an optimization runany time, and you can even recover all previous functionevaluations.

Tobias Glasmachers The Black Box Optimization Competition 12

Competition Architecture

The website contains a lot of information on the competition.

On the top right it features a login area for participants.

The downloads section is still empty, coming soon.

[demo: website]

[demo: website login area]

Tobias Glasmachers The Black Box Optimization Competition 13

Competition Architecture

The website contains a lot of information on the competition.

On the top right it features a login area for participants.

The downloads section is still empty, coming soon.

[demo: website]

[demo: website login area]

Tobias Glasmachers The Black Box Optimization Competition 13

Competition Architecture

The website contains a lot of information on the competition.

On the top right it features a login area for participants.

The downloads section is still empty, coming soon.

[demo: website]

[demo: website login area]

Tobias Glasmachers The Black Box Optimization Competition 13

Competition Architecture

The website contains a lot of information on the competition.

On the top right it features a login area for participants.

The downloads section is still empty, coming soon.

[demo: website]

[demo: website login area]

Tobias Glasmachers The Black Box Optimization Competition 13

Competition Architecture

The website contains a lot of information on the competition.

On the top right it features a login area for participants.

The downloads section is still empty, coming soon.

[demo: website]

[demo: website login area]

Tobias Glasmachers The Black Box Optimization Competition 13

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.

(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

The competition interface is written in plain C.

This makes it easily accessible from most languages.

The central API function is evaluate. It evaluates the blackbox function in a given point and returns the function value.

signature:int evaluate(double* point, double* value);

The caller is responsible for providing a point of the correctdimension. Everything else results in undefined behavior.(give wrong results, crash your program, format your hard disk...)

Return: 0 on error, non-zero on success. In the latter case thefunction value is written into *value.

Tobias Glasmachers The Black Box Optimization Competition 14

Software Interface

Other API functions:

int configure(int history, const char* logfilepath);

int login(const char* username, const char* password);

int numberOfTracks();

const char* trackName(int trackindex);

int setTrack(const char* trackname);

int numberOfProblems();

int setProblem(int problemID);

int dimension();

int budget();

int evaluations();

int history(int index, double* point, double* value);

const char* errorMessage();

Tobias Glasmachers The Black Box Optimization Competition 15

Example Clients

We provide example clients in C, C++, Java, Python, andMatlab/Octave.

Bernd Bischl is about to contribute an R package.

If you prefer Lua, D, VB, C#, Fortran, Go, Scala or Javascriptwe are happy to receive further contributions.

We even have a TCP/IP proxy. This allows you to access theblack box through a plain text interface from any environmentthat can open a network connection.

[demo: client]

Tobias Glasmachers The Black Box Optimization Competition 16

Example Clients

We provide example clients in C, C++, Java, Python, andMatlab/Octave.

Bernd Bischl is about to contribute an R package.

If you prefer Lua, D, VB, C#, Fortran, Go, Scala or Javascriptwe are happy to receive further contributions.

We even have a TCP/IP proxy. This allows you to access theblack box through a plain text interface from any environmentthat can open a network connection.

[demo: client]

Tobias Glasmachers The Black Box Optimization Competition 16

Example Clients

We provide example clients in C, C++, Java, Python, andMatlab/Octave.

Bernd Bischl is about to contribute an R package.

If you prefer Lua, D, VB, C#, Fortran, Go, Scala or Javascriptwe are happy to receive further contributions.

We even have a TCP/IP proxy. This allows you to access theblack box through a plain text interface from any environmentthat can open a network connection.

[demo: client]

Tobias Glasmachers The Black Box Optimization Competition 16

Example Clients

We provide example clients in C, C++, Java, Python, andMatlab/Octave.

Bernd Bischl is about to contribute an R package.

If you prefer Lua, D, VB, C#, Fortran, Go, Scala or Javascriptwe are happy to receive further contributions.

We even have a TCP/IP proxy. This allows you to access theblack box through a plain text interface from any environmentthat can open a network connection.

[demo: client]

Tobias Glasmachers The Black Box Optimization Competition 16

Example Clients

We provide example clients in C, C++, Java, Python, andMatlab/Octave.

Bernd Bischl is about to contribute an R package.

If you prefer Lua, D, VB, C#, Fortran, Go, Scala or Javascriptwe are happy to receive further contributions.

We even have a TCP/IP proxy. This allows you to access theblack box through a plain text interface from any environmentthat can open a network connection.

[demo: client]

Tobias Glasmachers The Black Box Optimization Competition 16

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Summary

We have presented theBlack Box Optimization Competition.

It is the first of its kind that challenges participants to dealwith a real black box.

It is based on a client server architecture with a “black box”dynamic library in the middle.

Everything is ready to go: the server is up and running,example clients exist in major programming languages, and awebsite with extensive documentation is available.

You are cordially invited to try your luck and prove your skill.

Fingers crossed!

Tobias Glasmachers The Black Box Optimization Competition 17

Thank you!

Questions?

Tobias Glasmachers The Black Box Optimization Competition 18