metrics and caffe - stanford university · roadmap evaluation metrics why metrics? confusion matrix...
TRANSCRIPT
![Page 1: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/1.jpg)
EVALUATION METRICS & CAFFE
Lyne Tchapmi
CS 231A
![Page 2: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/2.jpg)
Roadmap❖ Evaluation Metrics
❖ Why metrics?❖ Confusion Matrix❖ IOU❖ Positives vs Negatives
❖ ROC Curve❖ Precision/Recall❖ PR Curve
❖ Deep Learning & Caffe❖ NN and CNN Review❖ Caffe
![Page 3: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/3.jpg)
Why metrics?
❖ Objective evaluation of an algorithm’s performance
❖ Comparing different algorithms
❖ Choosing algorithm parameters
![Page 4: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/4.jpg)
Confusion MatrixDIGIT RECOGNITION TASK
![Page 5: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/5.jpg)
IOU(Intersection over Union)
GT
![Page 6: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/6.jpg)
IOU(Intersection over Union)
GT
![Page 7: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/7.jpg)
Positives vs Negatives
Actual Positive Actual Negative
Predicted Positive True Positive(TP)
False Positive(FP)
Predicted Negative False Negative(FN)
True Negative(TN)
FP
TP
FN
TN
![Page 8: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/8.jpg)
ROC Curve❖ ROC: Receiver Operating Characteristic
![Page 9: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/9.jpg)
ROC Curve❖ HOG + SVM threshold change
![Page 10: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/10.jpg)
Precision and Recall
Actual Positive Actual Negative
Predicted Positive True Positive(TP)
False Positive(FP)
Predicted Negative False Negative(FN)
True Negative(TN)
![Page 11: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/11.jpg)
PR Curve
![Page 12: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/12.jpg)
Metrics Summary❖ Confusion Matrix
❖ IOU
❖ Positives vs Negatives
❖ ROC Curve, Precision-Recall Curve
❖ Related
❖ MAP (Mean Average Precision): PASCAL VOC + ImageNet
❖ Segmentation accuracy: PASCAL VOC
❖ Miss rate: CALTECH Pedestrian Dataset
![Page 13: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/13.jpg)
Roadmap❖ Evaluation Metrics
❖ Why metrics?❖ Confusion Matrix❖ IOU❖ Positives vs Negatives
❖ ROC Curve❖ Precision/Recall❖ PR Curve
❖ Deep Learning & Caffe❖ NN and CNN Review❖ Caffe
![Page 14: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/14.jpg)
Neural Networks
![Page 15: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/15.jpg)
Convolutional Neural Networks
![Page 16: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/16.jpg)
Convolution Layer
![Page 17: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/17.jpg)
Activation Functions
![Page 18: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/18.jpg)
Pooling Layer
![Page 19: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/19.jpg)
Classification layer ❖ Softmax function
❖ Output between 0 and 1 for each class
![Page 20: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/20.jpg)
Convolutional Neural Networks
![Page 21: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/21.jpg)
Back-propagation
❖ SGD with momentum
❖ Nesterov momentum
❖ Adagrad
❖ RMSProp
Stochastic Gradient Descent (SGD)
![Page 22: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/22.jpg)
Caffe ❖ Deep Learning Framework from Berkeley
❖ C++, Python, MATLAB
❖ Available at https://github.com/BVLC/caffe
❖ CPU and GPU mode
❖ Platform-specific installation instructions available at http://caffe.berkeleyvision.org/installation.html
![Page 23: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/23.jpg)
Caffe❖ Blob: Storage and Communication of Data
❖ Data blobs are N x C x H x W
❖ Net: Contains all the layers in the networks
❖ Performs forward/backward pass through the entire network
❖ Solver: Used to set training/testing parameters
❖ Number of iterations,back propagation method,etc..
![Page 24: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/24.jpg)
Network Definition(train.prototxt)
![Page 25: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/25.jpg)
Network Definition(train.prototxt)
![Page 26: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/26.jpg)
Network Definition(train.prototxt)
![Page 27: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/27.jpg)
Network Definition(test.prototxt)Previously
![Page 28: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/28.jpg)
Previously
Network Definition(test.prototxt)
![Page 29: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/29.jpg)
Solver(solver.prototxt)
![Page 30: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/30.jpg)
Training from the Command Line
Command line output
OR
![Page 31: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/31.jpg)
Testing from the command line
![Page 32: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/32.jpg)
PyCaffe (Training in Python)❖ Add caffe python directory to path and import caffe
![Page 33: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/33.jpg)
Use NetSpec to define layers
![Page 34: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/34.jpg)
Define solver and train network
![Page 35: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/35.jpg)
Access Net data
![Page 36: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/36.jpg)
PyCaffe (Testing in Python)
![Page 37: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/37.jpg)
Caffe Summary
❖ Install Caffe
❖ Define training network in train.prototxt
❖ Define test network in test.prototxt
❖ Define solver parameters in solver.prototxt
❖ Train and test
![Page 38: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/38.jpg)
CNN Training tips
❖ Before running final/long training
❖ Make sure you can overfit on a small training set
❖ Make sure your loss decreases over first several iterations
❖ Otherwise adjust parameter until it does, especially learning rate
❖ Separate train/val/test data
![Page 39: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/39.jpg)
References❖ Metrics
❖ http://vase.essex.ac.uk/talks/performance-evaluation.pdf
❖ CNN❖ http://cs231n.github.io/
❖ Caffe❖ http://caffe.berkeleyvision.org/installation.html
❖ https://github.com/BVLC/caffe/tree/master/examples
❖ http://caffe.berkeleyvision.org/tutorial/interfaces.html
![Page 40: Metrics and Caffe - Stanford University · Roadmap Evaluation Metrics Why metrics? Confusion Matrix IOU Positives vs Negatives ROC Curve Precision/Recall PR Curve Deep Learning &](https://reader031.vdocuments.us/reader031/viewer/2022022010/5b067fe77f8b9a56408bbed6/html5/thumbnails/40.jpg)
QUESTIONS?