compressed sensing meets symbolic regression: sisso€¦ · luca m. ghiringhelli on-line course on...

Luca M. Ghiringhelli

On-line course on Big Data and Artificial Intelligence in Materials Sciences

Compressed sensingmeets

symbolic regression:SISSO

- Part 1 -

Reminder: Few bits of taxonomy

Machine learning

Representation learning

Learning algorithms that learn their representation and the predictive model.- symbolic regression- deep learning

Artificial intelligence

Compressed sensing meets symbolic regression

SymbolicregressionSymbolic

regressionCompressed

sensingCompressed

sensing

EvolutionaryprogrammingEvolutionary

programming

Sure-independence screeningcombined with

sparsifying operatorSISSO

Sure-independence screeningcombined with

sparsifying operatorSISSO

Featureselection/identification

Featureselection/identification

Linear regression

Symbolic regression

Linear regression

Symbolic regression

11111111

Linear regression

Symbolic regression

Linear regression

Symbolic regression

Linear regression

Symbolic regression

Linear regression Kernel regression

One-hidden-layer perceptron

Symbolic regression

Linear regression Kernel regression

One-hidden-layer perceptron

Symbolic regression

exp(x)

xn

ln(x)

Systematic construction of candidates

Length1 Length2

x + y

x·y

arctan(x)

Energy2 Energy1 Length1 Length2

| x - y | x + y

x / y

Length3

x3

Energy2 Energy1

| x - y |

exp(x)

xn

Energy2 Energy1

| x - y |

x / y

Length1 Length2

x / y

exp(-x)

ln(x)

Systematic construction of candidates

Length1 Length2

x + y

x·y

arctan(x)

Length1 Length2

x / y

exp(-x)

Energy2 Energy1

| x - y |

exp(x)

xn

Energy2 Energy1

| x - y |

x / y

Length1 Length2

x / y

exp(-x)

ln(x)

Length1 Length2

x + y

x·y

arctan(x)

Length1 Length2

x / y

exp(-x)

Energy2 Energy1

| x - y |

Symbolic regression

Evolutionary/genetic algorithmInitialize population

Representing individuals as binary genes

0 0 1 1 0 1 1 0 1 0 0

1 0 0 1 0 0 0 1 0 1 1

1 0 0 1 0 0 0 0 0 1 1

1 1 0 1 0 1 0 1 1 1 1

1 0 1 1 0 0 0 1 0 0 0

1 0 0 1 1 0 1 1 0 1 1

Representing individuals as binary genes

Evolutionary/genetic algorithm

0 0 1 1 0 1 1 0 1 0 0

1 0 0 1 0 0 0 1 0 1 1

1 1 0 1 0 1 0 1 1 1 1

1 0 0 1 0 0 0 0 0 1 1

1 0 0 1 1 0 1 1 0 1 1

1 0 1 1 0 0 0 1 0 0 0

0.89

0.55

0.34

0.21

0.13

0.08

Rank wrt fit function

Initialize population

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Initialize population

0 0 1 1 0 1 1 0 1 0 0

1 0 0 1 0 0 0 1 0 1 1

1 1 0 1 0 1 0 1 1 1 1

1 0 0 1 0 0 0 0 0 1 1

1 0 0 1 1 0 1 1 0 1 1

1 0 1 1 0 0 0 1 0 0 0

0.89

0.55

0.34

0.21

0.13

0.08

0 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Crossover

Initialize population

0 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1

1 0 0 1 0 0 0 0 1 0 00 0 1 1 0 1 1 1 0 1 1

Crossover

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Crossover

Initialize population

0 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1

1 0 0 1 0 0 0 0 1 0 00 0 1 1 0 1 1 1 0 1 1

0 0 1 1 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 1 1 0

Crossover

Mutation

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Crossover

Mutation

Initialize population

0 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1

1 0 0 1 0 0 0 0 1 0 00 0 1 1 0 1 1 1 0 1 1

0 0 1 1 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 1 1 0

Crossover

Mutation

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Crossover

Mutation

Initialize population

Happy?No

Rank wrt fit function

0 0 1 1 0 1 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1

1 0 0 1 0 0 0 0 1 0 00 0 1 1 0 1 1 1 0 1 1

0 0 1 1 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 1 1 0

Crossover

Mutation

Evolutionary/genetic algorithm

Rank wrt fit function

Randomly select “fittest first”

Crossover

Mutation

Initialize population

Happy?EndYes No

Rank wrt fit function

E.g., crossover for molecules/clusters

Rank wrt fit function

Randomly select “fittest first”

Crossover

Mutation

Initialize population

Happy?EndYes No

Evolutionary/genetic algorithm

Rank wrt fit function

Evolutionary programming

Energy2 Energy1

| x - y |

x / y

Length1 Length2

x / y

exp(-x)

Energy2 Energy1

x + y

x / y

Length1 Length2

x / y

ln(x)

Example of crossover between symbolic trees

Evolutionary programming

Energy2 Energy1

| x - y |

x / y

Length1 Length2

x / y

exp(-x)

Energy2 Energy1

x + y

x / y

Length1 Length2

x / y

ln(x)

Example of crossover between symbolic trees

Evolutionary programming

Energy2 Energy1

| x - y |

x / y

Length1 Length2

x / y

exp(-x)

Energy2 Energy1

x + y

x / y

Length1 Length2

x / y

ln(x)

Energy2 Energy1

x + y

x / y

Length1 Length2

x / y

exp(-x)ln(x)

Example of crossover between symbolic trees

Model selection: Pareto front

Objective 2

Obj

ectiv

e 1++++

++

+++++++

++

+++++++

++

+ + +

+

Multi-objective optimization:Points on the Pareto front are such that no point is found that simultaneously improve all the objective functions.

Model selection: Pareto front

Complexity (depth of the tree)

Accu

racy

(RM

SE) ++++

++

+++++++

++

+++++++

++

+ + +

+

Multi-objective optimization:Points on the Pareto front are such that no point is found that simultaneously improve all the objective functions.

A famous example: EUREQA

Distilling Free-Form Natural Laws from Experimental DataSchmidt M., Lipson H., Science, Vol. 324, No. 5923, (2009)EUREQA: genetic programming software.

EUREQA: Pareto front

Eureqa

In general, with symbolic regression:● If the exact equation is within reach of the searching/optimizing algorithm,

it is found. For other powerful ML methods (e.g., kernel regression, regression treesand forests, deep learning), this is not the case.

● The few fitting parameters yield stability with respect to noise (low complexity no overfitting)→

Compressing signals

Compressed sensing

Compressed sensing

Feature selection/identification vs extraction

Feature selection: selection of a subset among given features● Filters univariate ranking, i.e., each feature with the property● Wrappers search strategies, e.g., GA● Embedded (non-stochastic) optimization of objective function,

e.g., regularized regression, decision tree

Feature extraction: new (fewer) features are functions (e.g., linear combinations) of potentially all given features.● Dimension reduction● Autoencoders

Compressed sensing

D.L. Donoho, IEEE Trans. Inf. Theory 2006 DOI: 10.1109/TIT.2006.871582EJ Candès, J Romberg, T Tao, Trans. Inf. Theory 2006 DOI:10.1109/TIT.2005.862083R. Tibshirani J. Royal Stat. Soc. 1997 DOI: 10.1111/j.2517-6161.1996.tb02080.x

LASSO Least Absolute Shrinkage and Selection Operator

Compressed sensing

D.L. Donoho, IEEE Trans. Inf. Theory 2006DOI: 10.1109/TIT.2006.871582

EJ Candès, J Romberg, T Tao, Trans. Inf. Theory 2006 DOI:10.1109/TIT.2005.862083

Recovery possible when:

N: #features, M: #observations, Ω: sparsity

Compressed sensing

D.L. Donoho, IEEE Trans. Inf. Theory 2006 DOI: 10.1109/TIT.2006.871582EJ Candès, J Romberg, T Tao, Trans. Inf. Theory 2006 DOI:10.1109/TIT.2005.862083R. Tibshirani J. Royal Stat. Soc. 1997 DOI: 10.1111/j.2517-6161.1996.tb02080.x

LASSO Least Absolute Shrinkage and Selection Operator

Compressed sensing, or “sparse recovery”enables the recovering of a sparse signal from very few, non-adaptive measurements.

Compressed sensing, or “sparse recovery”enables the recovering of a sparse signal from very few, non-adaptive measurements.

Bonus slide: Suggested literature

Notable examples of other-than-SISSO compressed sensing applied to materials science.Actually, in this example LASSO is the applied method. Often LASSO and compressed sensing are thought to be equivalent, whereas compressed sensing includes LASSO as solution protocol.

V Ozoliņš, R Lai, R Caflisch, S Osher - PNAS, 2013 DOI: 10.1073/pnas.1318679110

LJ Nelson, GLW Hart, F Zhou, V Ozoliņš - PRB, 2013 DOI: 10.1103/PhysRevB.87.035125

LJ Nelson, V Ozoliņš, CS Reese, F Zhou, GLW Hart - PRB, 2013 DOI: 10.1103/PhysRevB.88.155105

F Zhou, W Nielson, Y Xia, V Ozoliņš - PRL, 2014 DOI: 10.1103/PhysRevLett.113.185501