applied machine learning in biomedicineenrigri/public/machine_learning/slides/...•beware of loss...
TRANSCRIPT
![Page 2: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/2.jpg)
Algorithm’s objective cost
Formal objective for algorithms: - minimize a cost function - maximize an objective function Proving convergence: - does objective monotonically improve? Considering alternatives: - does another algorithm score better?
![Page 3: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/3.jpg)
Loss function
![Page 4: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/4.jpg)
Choosing a loss function
• Motivated by the application – 0-1 error, achieving a tolerance, business cost
• Computational convenience: – Differentiability, convexity
• Beware of loss dominated by artifacts: – Outliers
– Unbalanced classes
![Page 5: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/5.jpg)
A step into linear regression
![Page 6: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/6.jpg)
A step into linear regression
![Page 7: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/7.jpg)
Vector form for RSS
![Page 8: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/8.jpg)
Least squares estimation
![Page 9: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/9.jpg)
Geometry of least squares
![Page 10: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/10.jpg)
Least square estimation
% ww = Dx1 weights
% X = NxD test cases
% Y = Nx1
ww = X\Y;
![Page 11: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/11.jpg)
Least square estimation (2)
![Page 12: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/12.jpg)
Least square estimation (2)
2. Update
![Page 13: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/13.jpg)
The importance of the step
![Page 14: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/14.jpg)
Least squares classifier
Why not using linear least squares to fit regressors on binary targets?
% fit yy = ww*xx
% ww = Dx1 weights
% xx = NxD test cases
% yy = Nx1
ww = xx\yy;
![Page 15: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/15.jpg)
Least squares classifier
![Page 16: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/16.jpg)
Least squares classifier
![Page 17: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/17.jpg)
Least squares classifier
Why not using linear least squares to fit regressors on binary targets?
% fit yy = ww*xx
% ww = Dx1 weights
% xx = NxD test cases
% yy = Nx1
ww = xx\yy;
![Page 18: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/18.jpg)
Least squares in practice (1)
Prostate cancer study (Stamey, 1989)
Mapping clinical measure with PSA marker
![Page 19: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/19.jpg)
Least squares in practice (2)
1) Normalize all data
![Page 20: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/20.jpg)
Least squares in practice (3)
2) Split the data into train and test set
3) Fit the regression on the training set
4) Estimate results on the test set
![Page 21: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/21.jpg)
Least squares in practice (5)
![Page 22: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/22.jpg)
Least squares in practice (6)
![Page 23: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/23.jpg)
Least squares in practice (6b)
![Page 24: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/24.jpg)
Multiclass linear classifier
K class, K>2 Building K-1 binary classifers
One versus all
![Page 25: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/25.jpg)
Multiclass linear classifier
K class, K>2 Building K(K-1)/2 binary classifers
One versus one
![Page 26: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/26.jpg)
Multiclass linear classifier
Least squares approach
![Page 27: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/27.jpg)
Linear regression (with features)
X = [ones(N,1), xx, xx.^2, xx.^3 , xx.^4 , xx.^5 , xx.^6];
W = X\yy;
% FunctionInterpolation
Xnew = [ones(N,1), xnew, xnew.^2, xnew.^3 , xnew.^4 , xnew.^5 , xnew.^6];
Ynew=Xnew*W;
![Page 28: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/28.jpg)
Neighbour-based regression
Take height from the nearest input
![Page 29: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/29.jpg)
Kernel smoothing
Weight points in proportion to a kernel
![Page 30: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/30.jpg)
Kernel smoothing
![Page 31: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/31.jpg)
Over fitting
We can make the empirical loss zero:
![Page 32: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/32.jpg)
Generalization
Want to do well on future, unknown data:
x
?
![Page 33: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/33.jpg)
Expected error
Training error Generalization error Expected test error
![Page 34: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/34.jpg)
Validation set
validation p=2 p=5
p=9
train
![Page 35: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/35.jpg)
Learning curves
Mea
n s
qu
ared
err
or
p, polynomial order
![Page 36: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/36.jpg)
Using validation set
Validation set used to estimate test error for fitted model
Can overfit the validation set
Tracking a validation set is also used during fitting of a single model
- ad hoc
- depends on optimizer
- sometimes fast
- sometimes can work annoyingly well
![Page 37: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/37.jpg)
Model selection and assessment Model selection:
Estimating the performance of different models in order to choose the best one
Model assessment:
having chosen a final model, estimating its prediction error on new data
Train Validation Test
![Page 38: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/38.jpg)
Cross validation
We want a procedure for estimating at least the average generalization error even when the data is scarce and we can not set aside a validation set
Data
Fold 5 - Test
Fold 4 - Train
Fold 3 - Train
Fold 2 - Train
Fold 1 - Train
Fold 4 - Train
Fold 4 - Train
Fold 2 - Train
Fold 1 - Train
Fold 3 - Test
![Page 39: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/39.jpg)
K-Fold Cross Validation
![Page 40: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/40.jpg)
Choosing K K=N leave one out Unbiased estimator for the prediction error High variance (training set are very similar)
Small-sized training set may overestimate the prediction error Rule of thumb: K=5 or K=10
![Page 41: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/41.jpg)
Cross validation scenario
![Page 42: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/42.jpg)
Cross validation scenario
![Page 43: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/43.jpg)
P1. Cleveland Heart Disease Data from V.A. Medical Center, Long Beach and Cleveland Clinic Foundation, 1988.
297 patients, 14 attributes per patient
Predict heart disease (possibly in a scale 1-4)
![Page 44: Applied Machine Learning in Biomedicineenrigri/Public/Machine_Learning/Slides/...•Beware of loss dominated by artifacts: –Outliers –Unbalanced classes A step into linear regression](https://reader033.vdocuments.us/reader033/viewer/2022052004/6017ebbe1f57b91477755941/html5/thumbnails/44.jpg)
P2. HIV cleavage site
Rögnvaldsson, You and Garwicz (2015) "State of the art prediction of HIV-1 protease cleavage sites", Bioinformatics, vol 31 (8), pp. 1204-1210. Kontijevskis, Wikberg and Komorowski (2007) "Computational Proteomics Analysis of HIV-1 Protease Interactome". Proteins: Structure, Function, and Bioinformatics, 68, 305–312. You, Garwicz and Rögnvaldsson (2005) "Comprehensive Bioinformatic Analysis of the Specificity of Human Immunodeficiency Virus Type 1 Protease". Journal of Virology, 79, 12477–12486.
Knowledge of the mechanism of HIV protease cleavage specificity is critical to the design of specific and effective HIV inhibitors. Searching for an accurate, robust, and rapid method to correctly predict the cleavage sites in proteins is crucial when searching for possible HIV inhibitors. Scope is to predict if a sequence of aminoacids will constitute a cleavage site