introduction to machine learning and stochastic optimization · 2017-03-12 · introduction to...
TRANSCRIPT
![Page 1: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/1.jpg)
Introduction to Machine Learning and Stochastic Optimization
Robert M. Gower
Spring School on Optimization and Data Science, Novi Saad, March 2017
![Page 2: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/2.jpg)
An Introduction to Supervised Learning
![Page 3: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/3.jpg)
Some References
Understanding Machine Learning: From Theory to Algorithms
Stanford Machine Learning on Coursera by Andrew Ng
Graduate level Undergraduate level
![Page 4: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/4.jpg)
Is There a Cat in the Photo?
Yes
No
![Page 5: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/5.jpg)
Is There a Cat in the Photo?
Yes
![Page 6: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/6.jpg)
Is There a Cat in the Photo?
Yes
![Page 7: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/7.jpg)
Is There a Cat in the Photo?
No
![Page 8: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/8.jpg)
Is There a Cat in the Photo?
Yes
![Page 9: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/9.jpg)
Find mapping h that assigns the “correct” target to each input
Is There a Cat in the Photo?
Yes
No
x: Input/Feature y: Output/Target
![Page 10: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/10.jpg)
Labeled Data
![Page 11: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/11.jpg)
Labeled Data
y= -1 means no/false
![Page 12: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/12.jpg)
Labeled Data
Learning Algorithm
y= -1 means no/false
![Page 13: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/13.jpg)
Labeled Data
Learning Algorithm
y= -1 means no/false
![Page 14: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/14.jpg)
Labeled Data
Learning Algorithm
-1
y= -1 means no/false
![Page 15: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/15.jpg)
Example: Linear Regression for Height
Sex Male
Age 30
Height 1,72 cm
Sex Female
Age 70
Height 1,52 cm
Labeled data
![Page 16: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/16.jpg)
Example Hypothesis: Linear Model
Example: Linear Regression for Height
Sex Male
Age 30
Height 1,72 cm
Sex Female
Age 70
Height 1,52 cm
Labeled data
![Page 17: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/17.jpg)
Example Training Problem:
Example Hypothesis: Linear Model
Example: Linear Regression for Height
Sex Male
Age 30
Height 1,72 cm
Sex Female
Age 70
Height 1,52 cm
Labeled data
![Page 18: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/18.jpg)
Linear Regression for Height
Age
Height
![Page 19: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/19.jpg)
Linear Regression for Height
The Training Algorithm
Age
Height
![Page 20: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/20.jpg)
Linear Regression for Height
The Training Algorithm
Age
Height
Other options aside from linear?
![Page 21: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/21.jpg)
Parametrizing the HypothesisHeight
Age
Linear:
Polinomial:
Age
Height
Neural Net:
![Page 22: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/22.jpg)
Loss Functions
Why a SquaredLoss?
![Page 23: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/23.jpg)
Loss Functions
Why a SquaredLoss?
Loss Functions
The Training Problem
![Page 24: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/24.jpg)
Loss Functions
Why a SquaredLoss?
Loss Functions
The Training Problem
Typically a convex function
![Page 25: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/25.jpg)
Choosing the Loss Function
Quadratic Loss
Binary Loss
Hinge Loss
![Page 26: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/26.jpg)
Choosing the Loss Function
Quadratic Loss
Binary Loss
Hinge Loss
y=1 in all figures
![Page 27: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/27.jpg)
Choosing the Loss Function
Quadratic Loss
Binary Loss
Hinge Loss
EXE: Plot the binary and hinge loss function in when
y=1 in all figures
![Page 28: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/28.jpg)
Loss Functions
Is a notion of Loss enough?
What happens when we do not have enough data?
![Page 29: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/29.jpg)
Loss FunctionsThe Training Problem
Is a notion of Loss enough?
What happens when we do not have enough data?
![Page 30: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/30.jpg)
Overfitting and Model Complexity
Fitting 1st order polynomial
![Page 31: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/31.jpg)
Overfitting and Model Complexity
Fitting 1st order polynomial
![Page 32: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/32.jpg)
Overfitting and Model Complexity
Fitting 3rd order polynomial
![Page 33: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/33.jpg)
Overfitting and Model Complexity
Fitting 9th order polynomial
![Page 34: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/34.jpg)
Regularizor Functions
General Training Problem
Regularization
![Page 35: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/35.jpg)
Regularizor Functions
General Training Problem
Regularization
Goodness of fit, fidelity term ...etc
![Page 36: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/36.jpg)
Regularizor Functions
General Training Problem
Regularization
Goodness of fit, fidelity term ...etc
Penlizes complexity
![Page 37: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/37.jpg)
Regularizor Functions
General Training Problem
Regularization
Goodness of fit, fidelity term ...etc
Penlizes complexity
Controls tradeoff between fit and complexity
![Page 38: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/38.jpg)
Regularizor Functions
General Training Problem
Regularization
Exe:
Goodness of fit, fidelity term ...etc
Penlizes complexity
Controls tradeoff between fit and complexity
![Page 39: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/39.jpg)
Overfitting and Model Complexity
Fitting kth order polynomial
![Page 40: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/40.jpg)
Overfitting and Model Complexity
Fitting kth order polynomial
For big enough, λthe solution is a 2nd order polynomial
![Page 41: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/41.jpg)
Linear hypothesis
Exe: Ridge Regression
Ridge Regression
L2 loss
L2 regularizor
![Page 42: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/42.jpg)
Linear hypothesis
Exe: Support Vector Machines
SVM with soft margin
Hinge loss
L2 regularizor
![Page 43: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/43.jpg)
Linear hypothesis
Exe: Logistic Regression
Logistic Regression
Logistic loss
L2 regularizor
![Page 44: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/44.jpg)
The Machine Learners Job
![Page 45: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/45.jpg)
The Machine Learners Job
![Page 46: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/46.jpg)
The Machine Learners Job
![Page 47: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/47.jpg)
The Machine Learners Job
![Page 48: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/48.jpg)
The Machine Learners Job
![Page 49: Introduction to Machine Learning and Stochastic Optimization · 2017-03-12 · Introduction to Machine Learning and Stochastic Optimization Robert M. Gower Spring School on Optimization](https://reader034.vdocuments.us/reader034/viewer/2022042023/5e7ade92e60d670444374b02/html5/thumbnails/49.jpg)
The Machine Learners Job