classification.net: efficient and accurate classification in c# jamie shotton toshiba corporate...
Post on 12-Jan-2016
214 Views
Preview:
TRANSCRIPT
Classification.NET: Efficient and Accurate Classification in C#
Jamie Shotton
Toshiba Corporate Research & Development Center, Japan
http://jamie.shotton.org/work/code/Classification.NET.zip
Introduction This tutorial gives
brief introduction to classification theory ideas for practical design and implementation of
classifiers examples of classifiers in Vision
Main technical reference [Bishop, 2006]
Programming references [Murach, 2005] [Liberty, 2005]
Structure
1) Introduction to classification
2) Library Design
3) Implementing classifiers
4) Real vision research
Infer discrete class label y from a set of measurements x
Mapping from a data point to a class label
This tutorial considers D-dimensional feature space binary labels
Supervised learning labeled training set
N training examples
Classification
Vending machine
Measurements x: material, diameter, weight
Class labels y : coin value
[Bishop, 2006]
Probabilistic Interpretation Discriminative models
model conditional probability directly
Generative models are an alternative use Bayes’ theorem to infer conditional probability
Decision theory is used to choose a single y e.g. maximum a-posteriori (MAP)
[Bishop, 2006]
Example Classifiers Nearest neighbour
[Shakhnarovich, 2006]
Linear discriminants decision stumps Fisher’s linear discriminant perceptrons
Decision trees
[Bishop, 2006]
Example Classifiers Boosting
http://www.boosting.org/
Support Vector Machines (SVMs) a ‘kernel’ method
http://www.kernel-machines.org/
And many more!DEMO
TIME
[Bishop, 2006]
Structure
1) Introduction to classification
2) Library Design
3) Implementing classifiers
4) Real vision research
Classification.NET Framework for classification in C#
general purpose extensible
A few example classifiers
Download library, demo, and slides from http://jamie.shotton.org/work/code/Classification
.NET.zip many thanks to Matthew Johnson for the demo
Why C#? Modern, object-oriented language
combines best of C++ and Java pointers, interpreted or compiled, garbage collected
.NET libraries rapid development and re-use of code
Freely available IDEs http://msdn.microsoft.com/vstudio/express/visualc
sharp/
http://www.mono-project.com/
[Scientific C#]
Representing Data
accuratedouble
fast, memory efficientfloat
flexibleno performance hit
<T>
generics
IDataPoint<T>flexible
v low performance hit
T[]fast, easy, but inflexible
double[] or float[]
Representing Data
fast, easyextensible to multi-class
int
Representing Data Sets
dim
en
sio
n d…
…
matrix
row vector…
…
……
……
……
example i
So just use T[,] or T[][] arrays? Not flexible
e.g. on-the-fly computation
Representing Custom class DataSet<T>
no interface changes needed for: on-the-fly computation sparse arrays sparse data points
void Increment(DataSet<double> dataSet){
for(int i = 0; i < dataSet.Count; i++) for(int d = 0; d < dataSet.Dimensionality; d++)
dataSet.Data[i][d] ++;}
Representing Data – Summary
IDataPoint<T>
DataSet<T>
LabeledDataSet<T>
int
Classifier<T> –Classifier Base Class
public abstract class Classifier<T>
{
// Train the classifier
public abstract void Train(LabeledTrainingSet<T> trainingSet);
// Return the classification for the data point
public abstract int Classify(IDataPoint<T> dataPoint);
…
}
Structure
1) Introduction to classification
2) Library Design
3) Implementing classifiers
4) Real vision research
Nearest-Neighbour Classification (NN) Find the nearest point in training set
distance metric (e.g. Euclidean)
Classify the point as
[Shakhnarovich, 2006]
Nearest-Neighbour Classification (NN)
[Shakhnarovich, 2006]
x1
x2
‘decision boundary’
Nearest-Neighbour Classification (NN)
[Shakhnarovich, 2006]
‘Voronoi’diagram
x1
x2
Implementing NN Classification
So let’s implement NN in Classification.NET
Naïve implementation very memory hungry training is instantaneous testing is very slow
Improvements to NN Classification Distance computation ‘trick’
Distances.Euclidean( IDataPoint<double> a,IDataPoint<double> b,double minDistance )
exact
kd-trees [Beis, 1997] class NearestNeighbourFast { … } exact or approximate
Parameter sensitive hashing [Shakhnarovich, 2006] approximate
Divide space into two halves division is axis-aligned classify each half differently
Examples
2D
3D
Decision Stumps (DS)
Classifier compares value xd
with threshold µ
Returns +1 or -1 based on sign s
Decision Stumps (DS)
x1
x2
x3
µ µ
Training Decision Stumps (DS)
x1
x2
x1
x2
x1-value threshold x2-value threshold
or
Training DS
But not always this easy! not usually linearly separable D dimensions
Search for best decision stump H dimensions d thresholds µ signs
Training set error
Training DS Efficiently Project onto each dimension successively
x1
x2
projection onto x1 axis
projectiononto x2 axis
Which Thresholds To Try? Fixed discrete set
perhaps wasteful does not adapt to data
Adaptive discrete set calculate mid-points between pairs of points
Efficient calculation of training set error ² algorithm
Efficient computation of error ² Recall
Consider decision stump with sign
Trivially,
decision stump training set error
Efficient computation of error ² Linear search over µ with update
4
µ
Efficient computation of error ² Linear search over µ with update
4 5
µ
Efficient computation of error ² Linear search over µ with update
4 5 6
µ
Efficient computation of error ² Linear search over µ with update
4 5 6 5
µ
Efficient computation of error ² Linear search over µ with update
4 5 6 5 4 3 2 3
µ
Efficient computation of error ² Linear search over µ with update
µ
4 5 6 5 4 3 2 34 4 5 6 7 65 3
DS Implementationpublic class DecisionStump : Classifier<double>
{private int _d; // The data dimensionprivate double _threshold; // The thresholdprivate int _sign; // The sign (+1 or -1)
// Train the classifier
public override void Train(LabeledTrainingSet<T> trainingSet) { … }
// Return the classification for the data point
public override int Classify(IDataPoint<T> dataPoint)
{
return dataPoint[_d] > _threshold ? _sign : -_sign;
}
…
DEMO
TIME
DS Summary Complexity
reasonable training time very low memory instantaneous classification time
Classification accuracy individually, not very powerful but in combination, much more powerful…
Boosting Many variants, e.g.
AdaBoost [Freund, 1999] LogitBoost & GentleBoost [Friedman, 1998] Cascade [Viola, 2001]
super-fast JointBoost [Torralba, 2007]
multi-class with shared features
Core ideas1. combine many simple classifiers2. weight the training data points
Core Idea 1 – Classifier Combine many simple classifiers (‘weak’ or
‘base’ learners)
1. computes classification score of weak learner2. multiplies by learned confidence value3. sums over T rounds4. compares sum to zero
gives discrete classification value, +1 or -1
Core Idea 2 – Training Weight the training data points
normalised distribution emphasise poorly classified examples
Learning is greedy iteration
At round (iteration) t choose optimal weak learner
under distribution
calculateto reflect updated accuracy
Weak Learners Can use almost any type of classifier
must adapt to weights distribution must give some classification advantage
Simple change allows DS to learn with weights:
AdaBoost Learning Algorithm
Initialise weights For
train weak learner using distribution
compute training set error
calculate confidence
update weights
[Freund, 1999]
AdaBoost with DS Example
1 round2 rounds3 rounds4 rounds5 rounds50 rounds
public class AdaBoost<WeakLearner>: Classifier<double>
where WeakLearner : Classifier<double>, IWeightedLearner
{
private List<WeakLearner> _h = new List<WeakLearner>(); // The learned weak learners
private List<double> _alpha = new List<double>(); // The learned alpha values
// Return the classification for the data point
public override int Classify(IDataPoint<T> dataPoint)
{
double classification = 0.0;
// Call the weak learner Classify() method and combine results
for (int t = 0; t < _h.Count; t++)
classification += _alpha[t] * _h[t].Classify(dataPoint);
// Return the thresholded classification
return classification > 0.0 ? +1 : -1;
}
…
AdaBoost Implementation DEMO
TIME
AdaBoost Summary Complexity
complexity of weak learners x T
Weak Learners stumps, trees, even AdaBoost classifiers
e.g. AdaBoost<AdaBoost<DecisionStump>>
Classification accuracy very flexible decision boundary good generalization
Support Vector Machines (SVMs) Maximize the margin
good generalization
Kernels allow complex decision boundaries linear, Gaussian, etc.
Classification.NET class SVM wrapper for [SVM.NET] library
[Bishop, 2006], [Burges, 1998]
smaller margin larger margin
DEMO
TIME
Structure
1) Introduction to classification
2) Library Design
3) Implementing classifiers
4) Real vision research
Contour Fragments for Object Detection We can recognise objects based fragments of
contour:
Can a computer?
[Shotton, 2007a]
Contour Fragments for Object Detection Clustering learns fragments
Labeled training data object bounding boxes
Boosted classifier learns is the object centre here?
[Shotton, 2007a]
sparse image locations
ears
hin
d legs
rear
tors
o
head
head
belly
contour fragmentsexample i
dim
en
sion
d…
……
……
……
……
DEMO
TIME
TextonBoost
Goal: semantically segment an image using texture (via ‘textons’) layout context
bicycle
road
building
TextonBoost Patterns of textons
‘texture-layout filters’
Labeled training data hand-drawn segmentations
Multi-class boosted classifier [Torralba, 2007] what class is this pixel?
[Shotton, 2007b]
dense image pixels
patterns of textons
dim
en
sion
d…
……
……
……
……
example i
sheep
gra
ss
TextonBoost
[Shotton, 2007b]
cow
grass grass
tree
body road
sky
airplane
sky
grass
building
object
classes
building grass tree cow sheep sky airplane water face car
bicycle flower sign bird book chair road cat dog body boat
Summary Classifiers very powerful for Vision research
many different options available
Classification.NET gives you AdaBoost, SVMs, Decision Stumps, Nearest
Neighbour
Careful high-level library design allows rapid classifier implementation quick experimentation
References [Beis, 1997]
J.S. Beis and D.G. Lowe.Shape Indexing Using Approximate Nearest-Neighbour Search in High-Dimensional Spaces.CVPR.
[Bishop, 2006]
C.M. Bishop.Pattern Recognition and Machine Learning.
[Burges, 1998]
C.J.C. Burges.A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery.
[Freund, 1999]
Y. Freund and R.E. Schapire.A Short Introduction to Boosting.Journal of Japanese Society for Artificial Intelligence.
[Friedman, 1998]
J. Friedman, T. Hastie, and R. Tibshirani.Additive Logistic Regression: A Statistical View of Boosting.The Annals of Statistics.
[SVM.NET]
M. Johnson.http://mi.eng.cam.ac.uk/~mj293/software_svm.html
[Liberty, 2005]
J. Liberty.Programming C#.
[Murach, 2005]
J. Murach.Murach’s C# 2005.
[Scientific C#]
F. Gilani.http://msdn.microsoft.com/msdnmag/issues/04/03/ScientificC/default.aspx
[Shakhnarovich, 2006]
G. Shakhnarovich, T. Darrel, and P. Indyk [Eds.].Nearest-Neighbor Methods in Learning and Vision.
[Shotton, 2007a]
J. Shotton, A. Blake, and R. Cipolla.Multi-Scale Categorical Object Recognition Using Contour Fragments.PAMI to appear (available on request).
[Shotton, 2007b]
J. Shotton, J.Winn, C. Rother, and A. Criminisi.TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context.IJCV to appear (available on request).
[Torralba, 2007]
A. Torralba, K.P. Murphy, and W.T. Freeman.Sharing Visual Features for Multiclass and Multiview Object Detection.PAMI.
[Viola, 2001]
P. Viola and M. Jones.Robust Real-time Object Detection.ICCV.
Thank You!jamie@shotton.org
code and slides available at
http://jamie.shotton.org/work/code/Classification.NET.zip
top related