a hitchhiker’s guide to neuroevolution in erlang

J e r o e n S o e t e r s

NEUROEVOLUTIONA hitchhiker’s guide to neuroevolution in

Erlang.

MEET GARY

2

WHAT IS MACHINE LEARNING?

• Artificial Neural Networks • Genetic Algorithms

3

ARTIFICIAL NEURAL NETWORKS

4

BIOLOGICAL NEURAL NETWORKS

5

Dendrites

SomaAxon

Synapse

A MODEL FOR A NEURON

6

Y

w1

w2

wn

x1

x2

xn

Dendrites Synapses Axon

Soma

A MODEL FOR A NEURON

6

Y

w1

w2

wn

x1

x2

xn

Input signals Weights Output signal

Neuron

HOW DOES THE NEURON DETERMINE IT’S OUTPUT?

Y =sign ∑ xiwi - ⍬

7

n=1

n

ACTIVATION FUNCTION

8

X

Y

MEET FRANK

9

PERCEPTRON LEARNING RULE

℮(p) = Yd(p) - Y(p)

wi(p + 1) = wi(p) + α • wi(p) • ℮(p) =

10

PERCEPTRON TRAINING ALGORITHM

11

weight training

start

stopweights converged? yes

no

set weights and threshold to random values [-0.5, 0.5]

activate the perceptron

LOGIC GATES

12

input values x1 x2 x1 AND x2 x1 OR x2 x1 XOR x2

0 0 0 0 0

0 1 0 1 1

1 0 0 1 1

1 1 1 1 0

TRAINING A PERCEPTRON TO PERFORM THE AND OPERATION

13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0

Threshold: ⍬ = 0.2 ; learning rate: α = 0.1 =


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0.3 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0 0.3


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


00


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0.3 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0 0.3


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


00


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0.3 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1 0.3


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


10


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


-1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0.2 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1 0.2


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1 -0.1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


01


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


1


13

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

1

0 0 0 0.3 -0.1 0 0 0.3 -0.1

0 1 0 0.3 -0.1 0 0 0.3 -0.1

1 0 0 0.3 -0.1 1 -1 0.2 -0.1

1 1 1 0.2 -0.1 0 1 0.3 0.0


0.3 0.0


14

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

2

0 0 0 0.3 0.0 0 0 0.3 0.0

0 1 0 0.3 0.0 0 0 0.3 0.0

1 0 0 0.3 0.0 1 -1 0.2 0.0

1 1 1 0.2 0.0 1 0 0.2 0.0


0.3 0.0


14

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

2

0 0 0 0.3 0.0 0 0 0.3 0.0

0 1 0 0.3 0.0 0 0 0.3 0.0

1 0 0 0.3 0.0 1 -1 0.2 0.0

1 1 1 0.2 0.0 1 0 0.2 0.0


0.2 0.0


15

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

3

0 0 0 0.2 0.0 0 0 0.2 0.0

0 1 0 0.2 0.0 0 0 0.2 0.0

1 0 0 0.2 0.0 1 -1 0.1 0.0

1 1 1 0.1 0.0 0 1 0.2 0.1


0.2 0.0


15

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

3

0 0 0 0.2 0.0 0 0 0.2 0.0

0 1 0 0.2 0.0 0 0 0.2 0.0

1 0 0 0.2 0.0 1 -1 0.1 0.0

1 1 1 0.1 0.0 0 1 0.2 0.1


0.1 0.0


15

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

3

0 0 0 0.2 0.0 0 0 0.2 0.0

0 1 0 0.2 0.0 0 0 0.2 0.0

1 0 0 0.2 0.0 1 -1 0.1 0.0

1 1 1 0.1 0.0 0 1 0.2 0.1


0.2 0.1


16

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

4

0 0 0 0.2 0.1 0 0 0.2 0.1

0 1 0 0.2 0.1 0 0 0.2 0.1

1 0 0 0.2 0.1 1 -1 0.1 0.1

1 1 1 0.1 0.1 1 0 0.1 0.1


0.2 0.1


16

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

4

0 0 0 0.2 0.1 0 0 0.2 0.1

0 1 0 0.2 0.1 0 0 0.2 0.1

1 0 0 0.2 0.1 1 -1 0.1 0.1

1 1 1 0.1 0.1 1 0 0.1 0.1


0.1 0.1


17

epochinputs

x1 x2

desired output

Yd

initial weights

w1 w2

actual output

Yerror ℮

final weights

w1 w2

5

0 0 0 0.1 0.1 0 0 0.1 0.1

0 1 0 0.1 0.1 0 0 0.1 0.1

1 0 0 0.1 0.1 0 0 0.1 0.1

1 1 1 0.1 0.1 1 0 0.1 0.1


0.1 0.1

A LITTLE GEOMETRY…

18

0 1

1

x2

x1 0 1

1

x2

x10 1

1

x2

x1

x1 AND x2 x1 OR x2 x1 XOR x2

WE NEED MORE LAYERS

19

3

4

5

1

2

Input layer Hidden layer Output layer

x1

x2

Y

PROBLEMS WITH BACK PROPAGATION

•a training set of sufficient size is required •topology of the network needs to be known in advance •no recurrent connections are allowed •activation function must be differentiable

Does not emulate the biological world

20

EVOLUTIONARY COMPUTATION

21

MEET JOHN

22

THE CHROMOSOME

23

10 11 101 0

1 1000 11 0

CROSSOVER

24

0 111 0 11 0 1 100 1 10 0

parents

1 1000 11 0

CROSSOVER

24

0 111 0 11 0 1 100 1 10 0✂

parents

✂

1 1000 11 0

1 10 00 111

offspring

CROSSOVER

24

0 111 1 10 0✂

parents

✂

MUTATION

25

A D10 11 101 0

MUTATION

25

A DA D10 11 101 01 0

MUTATION

25

A DA D10 11 101 01 00 1

EVOLUTIONARY ALGORITHM

•represent the candidate solution as a chromosome •chose the initial population size N, crossover probability (Pc) and mutation probability (Pm)

•define a fitness function to measure the performance of the chromosome •define the genetic operators for the chromosome

26


27

start

stop

generate a population

calculate fitness

termination criteria satisfied?

yes

no

new population size = N?

crossover and mutation

select pair for mating add to new population

replace population

no

yes

TRAVELING SALESMAN

28

A

B

D

F

G

C

E

H


•represent the candidate solution as a chromosome •define a fitness function to measure the performance of the chromosome •define the genetic operators for the chromosome •chose the initial population size N, crossover probability Pc and mutation probability Pm

29

THE CHROMOSOME

30

HG FE ABC D


•represent the candidate solution as a chromosome

•define a fitness function to measure the performance of the chromosome

•define the genetic operators for the chromosome •chose the initial population size N, crossover probability Pc and mutation probability Pm

31

FITNESS FUNCTION

Fitness = 1 / total distance

32



•define a fitness function to measure the performance of the chromosome •define the genetic operators for the chromosome •chose the initial population size N, crossover probability Pc and mutation probability Pm

33

CROSSOVER

34

G EBC H FA D A FEB H DC G

parents

CROSSOVER

34

G EBC H FA D A FEB H DC G✂ ✂

parents

H A D

offspring

CROSSOVER

34


parents

H A D

offspring

CROSSOVER

34


parents

B

H A D

offspring

CROSSOVER

34


parents

B E

H A D

offspring

CROSSOVER

34


parents

B E F

H A D

offspring

CROSSOVER

34


parents

B E F C

H A D

offspring

CROSSOVER

34


parents

B E F C G

MUTATION

35

HG FE ABC D

MUTATION

35

HG FE ABC DA D

MUTATION

35

HG FE ABC DA DAD




36

DEMO

37

NEUROEVOLUTION

38

MEET GENE

39

SIMULATION

•inputs (sensors) •outputs (actuators) •fitness function

40

CLEANING ROBOT

41

FOREX TRADING

42

AND LOTS MORE…

•data compression •training NPCs in a video game •cyber warfare

43


•represent the candidate solution as a chromosome •define a fitness function to measure the performance of the chromosome •define the genetic operators for the chromosome •chose the initial population size N, crossover probability Pc and mutation probability Pm

44

THE CHROMOSOME

45



•define a fitness function to measure the performance of the chromosome

•define the genetic operators for the chromosome •chose the initial population size N, crossover probability Pc and mutation probability Pm

46

FITNESS FUNCTION

Fitness = performance of network on an actual problem

47




48

CROSSOVER

Crossover doesn’t work for large neural nets!

49

MUTATE ACTIVATION FUNCTION

50

ADD CONNECTION

51

ADD NEURON

52

OUTSPLICE

53

MUTATION OPERATORS

and lots more…

54




55

RANDOM IMPACT MUTATION

Number of mutations = random(1, network size)

56

MEMETIC ALGORITHM

57

apply to a problem

start

generate a population

local search: Hill Climber calculate effective fitness

select fit organisms

create offspring

STOCHASTIC HILL CLIMBER (LOCAL SEARCH)

58

start

new fitness > old fitness?

yes

no

stopping condition reached?

stop

apply NN to a problem

backup and perturb weights restore backed-up weights

A LANGUAGE FOR NEUROEVOLUTION

•The system must be able to handle very large numbers of concurrent activities

•Actions must be performed at a certain point in time or within a certain time

•Systems may be distributed over several computers

•The system is used to control hardware

•The software systems are very large

59


59

•The system exhibits complex functionality such as, feature interaction.

•The systems should be in continuous operation for many years.

•Software maintenance (reconfiguration, etc) should be performed without stopping the system.

•There are stringent quality, and reliability requirements.

•Fault tolerance


59

•The system exhibits complex functionality such as, feature interaction.

•The systems should be in continuous operation for many years.

•Software maintenance (reconfiguration, etc) should be performed without stopping the system.

•There are stringent quality, and reliability requirements.

•Fault tolerance

Bjarne Dacker. Erlang - A New Programming Language. Ericsson Review, no 2, 1993.

MEET JOE

60

1:1 MAPPING

61

neuron

neuronneuron

neuron

process process

process process

processprocess

genotype

erlang

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

sync

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scapesense

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

percept

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

forward

forward

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scapeaction

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

{fitness, halt_flag}

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

sync

THE NEURAL NETWORK

62

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

cortex

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

{evaluation_completed, fitness}

cortex

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

cortex

fitness > best fitness

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

backup_weights

backup_weights

backup_weights

backup_weights

neuron neuron

neuronneuron

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

neuron

neuron

perturb_weights

perturb_weights

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

cortex

THE EXOSELF

63

neuron

neuronneuron

actuatorsensor

neuron

cortex

scape

exoself

cortex

reactivate

THE POPULATION MONITOR

64

population monitordatabase


64


private private privateprivate private


64



start

start start start

start


64




64



terminated

terminated terminated

terminated

terminated


64




64


private privateprivate


64


private privateprivate private private


64


private privateprivate private private

start

start start start

start

THE DEVIL IS IN THE DETAILS

•recurrent connections •newer generations get a higher chance for mutation •neural plasticity •public scapes and steady state evolution

65

POLE BALANCING

66

agent

cartactions

percepts

DEMO

67

BENCHMARK RESULTS

68

Method Single-Pole/Incomplete state information

Double-Pole/Partial Information W/O Damping

Double-Pole W/Damping

RWG 8557 415209 1232296

SANE 1212 262700 451612

CNE* 724 76906* 87623*

ESP 589 7374 26342

NEAT - - 6929

CMA-ES* - 3521* 6061*

CoSyNE* 127* 1249* 3416*

DXNN not performed 2359 2313

Our System 647 5184 4792

THE HANDBOOK

69

Feel free to reach out at:

e. [email protected] t. @JeroenSoeters

THANK YOU

mailto:[email protected]

DATA SCIENCE AND ENGINEERING

71

a hitchhiker’s guide to neuroevolution in erlang

Technology