lecture 10: lines from edges, binary operations
TRANSCRIPT
![Page 1: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/1.jpg)
Lecture 8: More on Classification,Lines from Edges, Interest Points,
Binary Operations
CAP 5415: Computer VisionFall 2009
![Page 2: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/2.jpg)
Non-Linear Classification
x
y
• We've focused on linear decision boundaries
• What if that's not good enough?
![Page 3: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/3.jpg)
Example from last time
![Page 4: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/4.jpg)
Non-Linear Classification
x
y
• What would a quadratic decision boundary look like?
• This is the decision boundary from x2 + 8xy + y2 > 0
![Page 5: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/5.jpg)
Non-Linear Classification
x
y
• We can generalize the form of the boundary
• ax2 + bxy + cy2+d > 0 Notice that this is linear in a,b,c, and d!
![Page 6: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/6.jpg)
Easy Non-Linear Classification
• Take our original training example (x,y), and make a new example
• (x,y) →(x2, xy, y2)• Now, plug these new vectors back into our
linear classification machinery• No new coding!
![Page 7: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/7.jpg)
(Assume this is the training data)
What's wrong with this decision boundary?
![Page 8: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/8.jpg)
What's wrong with this decision boundary?
● What if you tested on this data?● The classifier has over-fit the data
![Page 9: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/9.jpg)
How to tell if your classifier is overfitting
Strategy #1:Hold out part of your data as a test set
What if data is hard to come by?Strategy #2: k-fold cross-validation
Break the data set into k partsFor each part, hold a part out, then train the
classifier and use the held out part as a test setSlower than test-set methodMore efficient use of limited data
![Page 10: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/10.jpg)
● Boosted Classifiers and SVM's are probably the two most popular classifiers today
● I won't get into the math behind SVM's, if you are interested, you should take the pattern recognition course (highly recommended)
The Support Vector Machinecd ..
![Page 11: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/11.jpg)
The Support Vector Machine
![Page 12: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/12.jpg)
The Support Vector Machine• Last time, we talked about different criteria
for a good classifier• Now, we will consider a criterion called the
margin
![Page 13: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/13.jpg)
The Support Vector Machine● Margin – minimum distance from a data
point to the decision boundary
![Page 14: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/14.jpg)
The Support Vector Machine● The SVM finds the boundary that maximizes
the margin
![Page 15: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/15.jpg)
The Support Vector Machine● Data points that are along the margin are
called support vectors
![Page 16: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/16.jpg)
Non-Linear Classification in SVMs
• We will do the same trick as before
x
y
This is the decision boundary fromx2 + 8xy + y2 > 0
This is the same as making a new set of features, then doing linear classification
![Page 17: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/17.jpg)
Non-Linear Classification in SVMs
The decision function can be expressed in terms of dot-products
Each α will be zero unless the vector is a support vector
![Page 18: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/18.jpg)
Non-Linear Classification in SVMs
• What if we wanted to do non-linear classification?
• We could transform the features and compute the dot product of the transformed features.
• But there may be an easier way!
![Page 19: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/19.jpg)
The Kernel Trick
Let Φ(x) be a function that transforms x into a different space
A kernel function K is a function such that
![Page 20: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/20.jpg)
Example (Burges 98)
If
Then
This is called the polynomial kernel
![Page 21: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/21.jpg)
Gaussian RBF Kernel
One of the most commonly used kernels
Equivalent to doing a dot-product in an infinite dimensional space
![Page 22: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/22.jpg)
The Kernel TrickSo, with a kernel function K, the new classification
rule is
Basic Ideas:Computing the kernel function should be easier than
computing a dot-product in the transformed spaceOther algorithms, like logistic regression can also be
“kernelized”
![Page 23: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/23.jpg)
So what if I want to use an SVM?
There are well-developed packages with Python and MATLAB interfaceslibSVMSVMLightSVMTorch
![Page 24: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/24.jpg)
From Edges to Lines
We’ve talked about detecting Edges, but how can we extract lines from those edges?
We'll start with a very basic, least squares approach
![Page 25: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/25.jpg)
We have edge points
![Page 26: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/26.jpg)
We need a line
![Page 27: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/27.jpg)
Start with classic equation of line
Remember this?m – slopeb - y-intercept
![Page 28: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/28.jpg)
Fitting the line
In classification, we fit a separating boundary, or line, to maximize the probability of the data
Here, we will minimize error
![Page 29: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/29.jpg)
Convert this picture into numbers
Call every point (xi, y
i)
x
y
![Page 30: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/30.jpg)
Our new criterion
We want to fit a line so that the squared difference between the point and the line's “prediction” is minimized
![Page 31: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/31.jpg)
Mathematically
See why it's called least squares?
![Page 32: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/32.jpg)
A little calculus
Set these to zero and solve
![Page 33: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/33.jpg)
Let's take a matrix view of this
![Page 34: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/34.jpg)
Let's take a matrix view of this
So, we can write the prediction at multiple points as
![Page 35: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/35.jpg)
Going back
Notice that
Can be rewritten as
(I'm using MATLAB notation for A(:,1))
![Page 36: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/36.jpg)
Sketch it Out
![Page 37: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/37.jpg)
Now we can do the same thing again
Notice that
Can be rewritten as
(I'm using MATLAB notation for A(:,2))
![Page 38: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/38.jpg)
Now, setting everything to zero
![Page 39: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/39.jpg)
Also called pseudo-inverse
You will often see this appear as
Why all the matrix hassle?This will allow us to easily generalize to
higher-dimensions
![Page 40: Lecture 10: Lines from Edges, Binary Operations](https://reader035.vdocuments.us/reader035/viewer/2022071602/613d667b736caf36b75ce25f/html5/thumbnails/40.jpg)