1

Introduction to Bio-Inspired Models

During the last three decades, several efficient machine learning tools have been inspired in biology and nature:

Artificial Neural Networks (ANN) are inspired in the brain to automatically learn and generalize (model) observed data.

Evolutive and genetic algorithms offer a solution to standard optimization problems when no much information about the function to optimize is available.

Artificial ant colonies offer an alternative solution for optimization problems.All these methods share some common properties:

They are inspired in nature (not in human logical reasoning).

They are automatic (no human intervention) and nonlinear.

They provide efficient solutions to some hard NP-problems.

2

Introduction to Artificial Neural Networks

Artificial Neural Networks are inspired in the structure and functioning of the brain, which is a collection of interconnected neurons (the simplest computing elements performing information processing):

Each neuron consists of a cell body, that contains a cell nucleus. There are number of fibers, called dendrites, and a single long fiber called axon branching out from the cell body.The axon connects one neuron to others (through the dendrites). The connecting junction is called synapse.

3

Functioning of a “Neuron”

• The synapses releases chemical transmitter substances.• The chemical substances enter the dendrite, raising or lowering

the electrical potential of the cell body.• When the potential reaches a threshold, an electric pulse or

action potential is sent down to the axon affecting other neurons.(Therefore, there is a nonlinear activation).

• Excitatory and inhibitory synapses.

nonlinear activation function

neuron potential: mixed input of

neighboring neurons

weights (+ or -, excitatory or inhibitory)

(threshold)

4

cxe1

1)x(f

The neural activity (output) is given by a no linear function.

1. Init the neural weight with random values2. Select the input and output data and train it3. Compute the error associate with the output

4. Compute the error associate with the hidden neurons

5. Compute

and update the neural weight according to these values

Gradient descent

InputsOutputs

Multilayer perceptron. Backpropagation algorithm

5

Time Series Modeling and Forecast

1n2n1n x3.0x4.11x

Sometimes the chaotic time series have a stochastic look difficult to predict

An example is Henon map

6

Given a time series with 2000 points (T=20), generated from a Lorenz system (chaotic behavior). To check modeling power different parameters are tested .

Example: Supervised Fitting and Prediction

Three variables(x,y,z)

(xn,yn,zn)

(xn+1,yn+1,zn+1)

Continuous SystemNeural Network

3:k:3

h1 h2 hk

y1 yi

x2 x3 xj

Wik

wkj

3:k:3

xn yn zn

xn+1 yn+1

z i

zn+13:6:3

3:15:3

7

Dynamical Behavior

A simple model doesn’t capture the complete structure of the system , then the dynamics of the system is not reproduce.

3:6:3

3:15:3

A complex system it’s overfitting the problem and the dynamics of the system is not reproduce

Only a intermediate model with an appropriate amount of parameters can model the functional structure of the system and the dynamics

8

Time series from a infrared laser.

Xi-1 Xi-2 Xi-3 Xi-j

Xi

The Neural network reproduces laser behavior

Infrared laser intensity is modeled using a neural network. Only time lagged intensities are used.

Net 6:5:5:1

The Neural Network can be synchronized with the time series obtained from the laser.

9

Structural Learning: Modular Neural NetworksWith the aim of giving some flexibility to the network topology, modular

neural networks combine different neural blocks into a global topology.

Fully-connected topology (too many parameters).

Combining several blocks(parameter reduction).

Assigning different subnets to specific tasks we can simplify

the complexity of the model.

2*4+4*4+4*1+9=37 weights

2(2*2)+2(2*2)+4*1+9=29 weights

In most of the cases, block division is a heuristic task !!!

How to obtain an optimal “block division” for a given problem ?

10

Functional NetworksFunctional networks are a generalization of neural networks which combine both qualitative domain knowledge and data.

y

f

f

x

z

u

f

+ f-1

y

I

F

x

z

u

F

I

F

FQualitative knowledge:x3 = F(x1,x2),

Initial Topology

Theorem. The simplest functional form is:

Simplified Topology

Learning (least squares):

{n}

{a1, ..., an}

This is the optimal “block division” for this problem !!!Data: (x1i,x2i,x3i), i=1,2,...

11

Some FN Architectures

a

z

f1

f2

+

x

fn

y

g1

g2

gn

a

x

y

z+f1

f2

F(x, y) h 1( f (x) g(y))

Associative Model: F(x,y) is an associative operator.

Sliced-Conditioned Model:

F(x, y) fi (x)i * gi (y)

F(x0 , y) cx *(y)T

F(x, y0 ) cy *(x)T

where and are covenient basis for the x- and y-constant slices.

F(x, y) f (x) g(y)

Separable Model: A simple topology.

a

fx

u

g

+ h-1

f(x)

g(y)

f(x) + g(y)

h-1 ( f(x) + g(y))y

12

A First Example. Functional vs Neural

u x x2 y2 y3

Neural Network

Functional Network (separable model)

2:2:2:1 MLP 15 parameters RMSE=0.0074

2:3:3:1 MLP 25 parameters RMSE=0.0031

12 parameters RMSE=0.0024= {1,x ,x2 ,x3}

Knowledge of the network structure (separable).

Non-parametric approach to learn the neuron functions !!!!

100 points of Training Data with Uniform Noise in (-0.01,0.01).

25x25 points from the exact surface for Validation.

Appropriate family of functions (polynomial).

13

Functional Nets & Modular Neural Nets

Advantages and shortcomings of

Black-box topology with no problem connection.

Efficient non-parametric models for approximating functions.Neural Nets

Parametric learning techniques (supply basis functions).

Model driven optimal topology.Functional Nets

The topology of the network is obtained from the

Functional network.

The neuron functions are Approximated using MLPs.

Hybrid functional-neural networks (Modular networks)

14

Another example. Nonlinear Time Series

21 5.07.11 nnn xxx

Nonlinear Maps (the Lozi model)

Sensitivity to initial conditions

Fractal geometry0 10 20 30 40

0

0.2

0.4

0.6

0.8

n

xn,yn,zn

Time series modeling and forecasting is an important problem with many practical applications.

Goal: predicting the future using past values.

x1, x2,…, xn ¿¿¿ xn+1 ???

Modeling methods:

xn+1 = F(x1, x2,…, xn)

Nonlinear time series may exhibit complex seemingly stochastic behavior.

There are many well-known techniques for linear time series (ARMA, etc.).

Nonlinear time series modeling is a difficult task because:

Trajectories starting at very close initial points split away after a few iterates.

Evolve in a irregular fractal space.

X1=0.8

X1=0.8 + 10-3

X1=0.8 - 10-3

15

Functional Models (separation)

xn 1 1.7 xn 1 0.5xn 2

xn f (xn 1) g(xn 2 )

1 1.7 xn 1

0.5xn 2

500 training points 1000 validation points.

1

2

f

f

+

+

+

FN

MFNN 1

MFNN 2

Separable Functional Net:¿which basis family?={sin(x),…,sin(mx), cos(x),…,cos(mx)}With 4*m parameters

Symmetric Modular Functional Neural Net:

1:m:11:m:1

With 6*m parameters

Asymmetric Modular Functional Neural Net:

1:2m:11:2:1

With 2*m-2 parameters

m=11 (44 pars)

RMSE=5.3e-3

m=7 (42 pars) RMSE=1.5e-3

m=7 (42 pars) RMSE=4.0e-4

16

Minimum Description Length

The Minimum Description Length (MDL) algorithm has proved to be simple and efficient in several problems about Model Selection.

Description Length for a model