tracking - carnegie mellon school of computer science16385/s18/lectures/lecture24.pdf · kernel...
TRANSCRIPT
![Page 1: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/1.jpg)
Tracking
16-385 Computer VisionSpring 2018, Lecture 24http://www.cs.cmu.edu/~16385/
![Page 2: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/2.jpg)
• Homework 6 has been posted and is due on April 20th. - Any questions about the homework?- How many of you have looked at/started/finished homework 6?
• This week Yannis’ office hours will be on Wednesday 3-6 pm.- Added an extra hour to make up for change.
Course announcements
![Page 3: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/3.jpg)
• KLT tracker.
• Mean-shift algorithm.
• Kernel density estimation.
• Mean-shift tracker.
• Modern trackers.
Overview of today’s lecture
![Page 4: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Kris Kitani (16-385, Spring 2017).
![Page 5: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/5.jpg)
http://iccp2018.ece.cmu.edu/
![Page 6: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/6.jpg)
http://iccp2018.ece.cmu.edu/
![Page 7: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/7.jpg)
http://iccp2018.ece.cmu.edu/
![Page 8: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/8.jpg)
All lectures and talks are free for CMU students!
(You only need to pay for registration to attend the free food events.)
![Page 9: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/9.jpg)
All lectures and talks are free for CMU students!
(You only need to pay for registration to attend the free food events.)
Everything in blue is free!
![Page 10: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/10.jpg)
cameras that see around corners
cameras that measure depth in real time
15-463/15-663/15-862 Computational Photography
Learn about this and other unconventional cameras – and build some on your own!
http://graphics.cs.cmu.edu/courses/15-463/
cameras that take video at the speed of light
cameras that capture entire focal stacks
ICCP covers all of these
![Page 11: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/11.jpg)
Kanade-Lucas-Tomasi
(KLT) tracker
![Page 12: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/12.jpg)
https://www.youtube.com/watch?v=rwIjkECpY0M
![Page 13: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/13.jpg)
Feature-based tracking
How should we select the ‘small images’ (features)?
How should we track them from frame to frame?
Up to now, we’ve been aligning entire images
but we can also track just small image regions too!(sometimes called sparse tracking or sparse alignment)
![Page 14: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/14.jpg)
An Iterative Image Registration Technique
with an Application to Stereo Vision.
1981
Lucas Kanade
Detection and Tracking of Feature Points.
1991
Kanade Tomasi
Good Features to Track.
1994
Tomasi Shi
History of the
Kanade-Lucas-Tomasi
(KLT) Tracker
The original KLT algorithm
![Page 15: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/15.jpg)
Method for aligning
(tracking) an image patch
Kanade-Lucas-Tomasi
Method for choosing the
best feature (image patch)
for tracking
Lucas-Kanade Tomasi-Kanade
How should we select features?How should we track them from frame to
frame?
![Page 16: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/16.jpg)
What are good features for tracking?
![Page 17: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/17.jpg)
What are good features for tracking?
Intuitively, we want to avoid smooth
regions and edges.
But is there a more is principled way to
define good features?
![Page 18: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/18.jpg)
Can be derived from the tracking algorithm
What are good features for tracking?
![Page 19: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/19.jpg)
Can be derived from the tracking algorithm
What are good features for tracking?
‘A feature is good if it can be tracked well’
![Page 20: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/20.jpg)
Recall the Lucas-Kanade image alignment method:
incremental update
error function (SSD)
![Page 21: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/21.jpg)
Recall the Lucas-Kanade image alignment method:
incremental update
error function (SSD)
linearize
![Page 22: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/22.jpg)
Recall the Lucas-Kanade image alignment method:
incremental update
error function (SSD)
linearize
Gradient update
![Page 23: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/23.jpg)
Recall the Lucas-Kanade image alignment method:
incremental update
error function (SSD)
linearize
Gradient update
Update
![Page 24: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/24.jpg)
Stability of gradient decent iterations depends on …
![Page 25: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/25.jpg)
Stability of gradient decent iterations depends on …
Inverting the Hessian
When does the inversion fail?
![Page 26: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/26.jpg)
Stability of gradient decent iterations depends on …
Inverting the Hessian
When does the inversion fail?
H is singular. But what does that mean?
![Page 27: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/27.jpg)
Above the noise level
Well-conditioned
both Eigenvalues are large
both Eigenvalues have similar magnitude
![Page 28: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/28.jpg)
Concrete example: Consider translation model
Hessian
How are the eigenvalues related to image content?
←when is this singular?
![Page 29: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/29.jpg)
interpreting eigenvalues
l1
l2
l2 >> l1
l1 >> l2
What kind of image patch does
each region represent?
![Page 30: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/30.jpg)
interpreting eigenvalues
‘horizontal’
edge
‘vertical’
edge
flat
l1
l2
l2 >> l1
l1 >> l2
l1 ~ l2
‘corner’
![Page 31: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/31.jpg)
interpreting eigenvalues
flat
‘corner’
l1
l2
l2 >> l1
l1 >> l2
l1 ~ l2
‘horizontal’
edge
‘vertical’
edge
![Page 32: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/32.jpg)
What are good features for tracking?
![Page 33: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/33.jpg)
What are good features for tracking?
‘big Eigenvalues means good for tracking’
![Page 34: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/34.jpg)
KLT algorithm
1. Find corners satisfying
2. For each corner compute displacement to next frame using
the Lucas-Kanade method
3. Store displacement of each corner, update corner position
4. (optional) Add more corner points every M frames using 1
5. Repeat 2 to 3 (4)
6. Returns long trajectories for each corner point
![Page 35: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/35.jpg)
Mean-shift algorithm
![Page 36: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/36.jpg)
![Page 37: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/37.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
![Page 38: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/38.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Find the region of
highest density
![Page 39: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/39.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Pick a point
![Page 40: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/40.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Draw a window
![Page 41: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/41.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the
(weighted) mean
![Page 42: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/42.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 43: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/43.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 44: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/44.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 45: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/45.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 46: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/46.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 47: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/47.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 48: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/48.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 49: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/49.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 50: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/50.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 51: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/51.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 52: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/52.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 53: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/53.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 54: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/54.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
![Page 55: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/55.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Compute the mean
![Page 56: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/56.jpg)
Mean Shift Algorithm
Fukunaga & Hostetler (1975)
A ‘mode seeking’ algorithm
Shift the window
To understand the theory behind this we need to understand…
![Page 57: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/57.jpg)
Kernel density estimation
![Page 58: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/58.jpg)
Kernel Density Estimation
A method to approximate an underlying PDF from samples
Put ‘bump’ on every sample to approximate the PDF
To understand the mean shift algorithm …
samples (+)
bumps
sum of bumps
![Page 59: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/59.jpg)
probability density function
1 2 3 4 5 6 7 8 9 10
cumulative density function
p(x)
Say we have some hidden PDF…
![Page 60: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/60.jpg)
ran
do
mly
sa
mp
le
0
1We can draw samples,
using the CDF…
![Page 61: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/61.jpg)
ran
do
mly
sa
mp
le
0
1
![Page 62: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/62.jpg)
ran
do
mly
sa
mp
le
0
1
samples
![Page 63: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/63.jpg)
samples
Now to estimate the ‘hidden’ PDF
place Gaussian bumps on the samples…
![Page 64: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/64.jpg)
samples
discretized ‘bump’
![Page 65: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/65.jpg)
samples
![Page 66: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/66.jpg)
samples
![Page 67: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/67.jpg)
1 2 3 4 5 6 7 8 9 10
samples
Kernel Density
Estimate
approximates the
original PDF
![Page 68: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/68.jpg)
Kernel Density Estimation
Approximate the underlying PDF from samples from it
Put ‘bump’ on every sample to approximate the PDF
Gaussian ‘bump’ aka ‘kernel’
but there are many types of kernels!
For example…
![Page 69: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/69.jpg)
Kernel Function
returns the ‘distance’ between two points
![Page 70: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/70.jpg)
Epanechnikov kernel
Uniform kernel
Normal kernel
These are all radially symmetric kernels
![Page 71: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/71.jpg)
Radially symmetric kernels
profile
…can be written in terms of its profile
![Page 72: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/72.jpg)
Connecting KDE and the
Mean Shift Algorithm
![Page 73: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/73.jpg)
Mean-Shift Tracking
Given a set of points:
and a kernel:
Find the mean sample point:
![Page 74: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/74.jpg)
Mean-Shift Algorithm
While
Initialize
1. Compute mean-shift
2. Update
Where does this algorithm come from?
shift values becomes really small
place we start
compute the ‘mean’
compute the ‘shift’
update the point
![Page 75: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/75.jpg)
While
Initialize
Where does this algorithm come from?
Where does this
come from?
Mean-Shift Algorithm
2. Update
1. Compute mean-shift
![Page 76: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/76.jpg)
Kernel density estimate (radially symmetric kernels)
Gradient of the PDF is related to the mean shift vector
How is the KDE related to the mean shift algorithm?
The mean shift vector is a ‘step’ in the direction of the gradient of the KDE
Recall:
We can show that:
can compute probability for any point using the KDE!
mean-shift algorithm is maximizing the objective function
![Page 77: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/77.jpg)
In mean-shift tracking, we are trying to find this
which means we are trying to…
![Page 78: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/78.jpg)
We are trying to optimize this:
usually non-linear
How do we optimize this non-linear function?
non-parametric
find the solution that has the highest probability
![Page 79: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/79.jpg)
We are trying to optimize this:
How do we optimize this non-linear function?
compute partial derivatives … gradient descent!
usually non-linear non-parametric
![Page 80: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/80.jpg)
Compute the gradient
![Page 81: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/81.jpg)
Gradient
Expand the gradient (algebra)
![Page 82: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/82.jpg)
Gradient
Expand gradient
![Page 83: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/83.jpg)
Gradient
Expand gradient
Call the gradient of the kernel function g
![Page 84: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/84.jpg)
Gradient
change of notation(kernel-shadow pairs)
Expand gradient
keep this in memory:
![Page 85: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/85.jpg)
multiply it out
too long!
(use short hand notation)
![Page 86: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/86.jpg)
multiply by one!
collecting like terms…
What’s happening here?
![Page 87: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/87.jpg)
The mean shift is a ‘step’ in the direction of the gradient of the KDE
mean shift!
mean shift
Can interpret this to be
gradient ascent with
data dependent step size
constant
Let
![Page 88: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/88.jpg)
Mean-Shift Algorithm
While
Initialize
1. Compute mean-shift
2. Update
gradient with
adaptive step size
Just 5 lines of code!
![Page 89: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/89.jpg)
Everything up to now has been about
distributions over samples…
![Page 90: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/90.jpg)
Mean-shift tracker
![Page 91: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/91.jpg)
Dealing with images
Pixels for a lattice, spatial density is the same everywhere!
What can we do?
![Page 92: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/92.jpg)
Consider a set of points:
Sample mean:
Mean shift:
Associated weights:
![Page 93: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/93.jpg)
Mean-Shift Algorithm(for images)
While
Initialize
1. Compute mean-shift
2. Update
![Page 94: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/94.jpg)
For images, each pixel is point with a weight
![Page 95: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/95.jpg)
For images, each pixel is point with a weight
![Page 96: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/96.jpg)
For images, each pixel is point with a weight
![Page 97: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/97.jpg)
For images, each pixel is point with a weight
![Page 98: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/98.jpg)
For images, each pixel is point with a weight
![Page 99: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/99.jpg)
For images, each pixel is point with a weight
![Page 100: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/100.jpg)
For images, each pixel is point with a weight
![Page 101: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/101.jpg)
For images, each pixel is point with a weight
![Page 102: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/102.jpg)
For images, each pixel is point with a weight
![Page 103: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/103.jpg)
For images, each pixel is point with a weight
![Page 104: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/104.jpg)
For images, each pixel is point with a weight
![Page 105: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/105.jpg)
For images, each pixel is point with a weight
![Page 106: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/106.jpg)
For images, each pixel is point with a weight
![Page 107: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/107.jpg)
For images, each pixel is point with a weight
![Page 108: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/108.jpg)
Finally… mean shift tracking in video!
![Page 109: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/109.jpg)
Frame 1 Frame 2
‘target’
center coordinate
of target
center coordinate of
candidate
Goal: find the best candidate location in frame 2
Use the mean shift algorithm
to find the best candidate location
‘candidate’there are many ‘candidates’ but only one ‘target’
![Page 110: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/110.jpg)
Non-rigid object tracking
hand tracking
![Page 111: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/111.jpg)
Target
Compute a descriptor for the target
![Page 112: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/112.jpg)
Target Candidate
Search for similar descriptor in neighborhood in next frame
![Page 113: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/113.jpg)
Target
Compute a descriptor for the new target
![Page 114: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/114.jpg)
Target Candidate
Search for similar descriptor in neighborhood in next frame
![Page 115: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/115.jpg)
How do we model the target and candidate regions?
![Page 116: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/116.jpg)
Modeling the target
M-dimensional target descriptor
A normalized
color histogram
(weighted by distance)
Kronecker delta
function
function of inverse
distance
(weight)
Normalization
factor
(centered at target center)
a ‘fancy’ (confusing) way to write a weighted histogram
sum over
all pixels
quantization
function
bin ID
![Page 117: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/117.jpg)
Modeling the candidate
M-dimensional candidate descriptor
(centered at location y)
bandwidth
a weighted histogram at y
![Page 118: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/118.jpg)
Similarity between
the target and candidate
Bhattacharyya Coefficient
Just the Cosine distance between two unit vectors
Distance function
![Page 119: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/119.jpg)
Now we can compute the similarity between a
target and multiple candidate regions
![Page 120: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/120.jpg)
target
similarity over imageimage
![Page 121: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/121.jpg)
target
similarity over imageimage
we want to find this peak
![Page 122: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/122.jpg)
Objective function
Assuming a good initial guess
Linearize around the initial guess (Taylor series expansion)
derivativefunction at specified value
same as
![Page 123: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/123.jpg)
Remember
definition of this?
Linearized objective
Fully expanded
![Page 124: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/124.jpg)
where
Does not depend on unknown y Weighted kernel density estimate
Weight is bigger when
Fully expanded linearized objective
Moving terms around…
![Page 125: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/125.jpg)
OK, why are we doing all this math?
![Page 126: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/126.jpg)
We want to maximize this
![Page 127: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/127.jpg)
where
Fully expanded linearized objective
We want to maximize this
![Page 128: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/128.jpg)
where
Fully expanded linearized objective
doesn’t depend on unknown y
We want to maximize this
![Page 129: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/129.jpg)
where
Fully expanded linearized objective
doesn’t depend on unknown y
We want to maximize this
only need to
maximize this!
![Page 130: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/130.jpg)
where
Fully expanded linearized objective
doesn’t depend on unknown y
what can we use to solve this weighted KDE?
Mean Shift Algorithm!
We want to maximize this
![Page 131: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/131.jpg)
the new sample of mean of this KDE is
(this was derived earlier)
(new candidate
location)
![Page 132: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/132.jpg)
Mean-Shift Object Tracking
1. Initialize location
Compute
Compute
2. Derive weights
3. Shift to new candidate location (mean shift)
4. Compute
5. If return
Otherwise and go back to 2
For each frame:
![Page 133: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/133.jpg)
Target
Compute a descriptor for the target
![Page 134: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/134.jpg)
Target Candidate
Search for similar descriptor in neighborhood in next frame
![Page 135: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/135.jpg)
Target
Compute a descriptor for the new target
![Page 136: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/136.jpg)
Target Candidate
Search for similar descriptor in neighborhood in next frame
![Page 137: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/137.jpg)
![Page 138: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/138.jpg)
Modern trackers
![Page 139: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/139.jpg)
![Page 140: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/140.jpg)
![Page 141: Tracking - Carnegie Mellon School of Computer Science16385/s18/lectures/lecture24.pdf · Kernel Density Estimation Approximate the underlying PDF from samples from it Put ‘bump’](https://reader035.vdocuments.us/reader035/viewer/2022081617/604c38c836786153b219a97b/html5/thumbnails/141.jpg)
References
Basic reading:• Szeliski, Sections 4.1.4, 5.3.