cs4495/6495 introduction to computer vision · 2015. 3. 12. · introduction to computer vision ....
TRANSCRIPT
![Page 1: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/1.jpg)
6B-L1 Dense flow: Brightness constraint
CS4495/6495 Introduction to Computer Vision
![Page 2: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/2.jpg)
Motion estimation techniques
Feature-based methods
Direct, dense methods
![Page 3: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/3.jpg)
Motion estimation techniques
Direct, dense methods
• Directly recover image motion at each pixel from spatio-temporal image brightness variations
• Dense motion fields, but sensitive to appearance variations
• Suitable for video and when image motion is small
![Page 4: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/4.jpg)
Motion estimation: Optic flow
Optic flow is the apparent motion of objects or surfaces
![Page 5: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/5.jpg)
Problem definition: Optic flow
How to estimate pixel motion from image I(x, y, t) to I(x, y, t+1)?
( , , )I x y t ( , , 1)I x y t
![Page 6: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/6.jpg)
=> Solve pixel correspondence problem
• Given a pixel in 𝐼(𝑥, 𝑦, 𝑡), look for nearby pixels of the same color in 𝐼(𝑥, 𝑦, 𝑡 + 1)
This is the optic flow problem.
How to estimate pixel motion from image I(x, y, t) to I(x, y, t+1)?
( , , )I x y t ( , , 1)I x y t
![Page 7: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/7.jpg)
Key assumptions
• color constancy: a point in 𝐼(𝑥, 𝑦, 𝑡) looks the same in 𝐼(𝑥′, 𝑦′, 𝑡 + 1)
– For grayscale images, this is brightness constancy
• small motion: points do not move very far
How to estimate pixel motion from image I(x, y, t) to I(x, y, t+1)?
( , , )I x y t ( , , 1)I x y t
![Page 8: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/8.jpg)
Optic flow constraints (grayscale images)
1) Brightness constancy constraint (equation)
( , , ) ( , , 1)I x y t I x u y v t
( , , )I x y t ( , , 1)I x y t
( , )u v
![Page 9: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/9.jpg)
Optic flow constraints (grayscale images)
1) Brightness constancy constraint (equation)
0 ( , , 1) ( , , )I x u y v t I x y t
( , , )I x y t ( , , 1)I x y t
( , )u v
![Page 10: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/10.jpg)
Optic flow constraints (grayscale images)
( , ) ( , ) [higher order terms]I I
I x u y v I x y u vx y
2) Small motion: (u and v are less than 1 pixel, or smooth)
Taylor series expansion of I:
( , , )I x y t ( , , 1)I x y t
( , )u v
![Page 11: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/11.jpg)
Optic flow constraints (grayscale images)
( , ) ( , )I I
I x u y v I x y u vx y
2) Small motion: (u and v are less than 1 pixel, or smooth)
Taylor series expansion of I:
( , , )I x y t ( , , 1)I x y t
( , )u v
![Page 12: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/12.jpg)
Combining these two equations:
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
0 ( , , 1) ( , , )
( , , 1) ( , , )x y
I x u y v t I x y t
I x y t I u I v I x y t
![Page 13: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/13.jpg)
Combining these two equations:
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
0 ( , , 1) ( , , )
( , , 1) ( , , )
[ ( , , 1) ( , , )]
x y
x y
t x y
I x u y v t I x y t
I x y t I u I v I x y t
I x y t I x y t I u I v
I I u I v
![Page 14: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/14.jpg)
Combining these two equations:
(Short hand: 𝐼𝑥 =𝜕𝐼
𝜕𝑥
for t or t+1)
0 ( , , 1) ( , , )
( , , 1) ( , , )
[ ( , , 1) ( , ,
,
)]
x y
x y
t
t x y
I x u y v t I x y t
I x y t I u I v I x y t
I x y t I x
I I
y t I u I v
I I u I v
u v
![Page 15: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/15.jpg)
Combining these two equations:
0 ,tI I u v
In the limit as u and v approaches zero, this becomes exact:
0 ,tI I u v
![Page 16: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/16.jpg)
Combining these two equations:
0 ,tI I u v
In the limit as u and v approaches zero, this becomes exact:
Brightness constancy constraint equation
0x y tI u I v I
0 ,tI I u v
![Page 17: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/17.jpg)
Gradient component of flow
Q: How many unknowns and equations per pixel?
2 unknowns (u,v) but 1 equation!
0x y tI u I v I 0 ,tI I u v or
![Page 18: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/18.jpg)
Gradient component of flow
Intuitively, what does this constraint mean?
• The component of the flow in the gradient direction is determined
• The component of the flow parallel to an edge is unknown
(u’,v’)
edge
(u,v)
gradient
(u+u’,v+v’)
0x y tI u I v I 0 ,tI I u v or
![Page 19: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/19.jpg)
Aperture problem
![Page 20: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/20.jpg)
Aperture problem
![Page 21: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/21.jpg)
Aperture problem
![Page 22: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/22.jpg)
Apparently an aperture problem
See: http://www.cfar.umd.edu/~fer/optical/movement2.html
![Page 23: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/23.jpg)
Gradient component of flow
Some folks say: “This explains the Barber Pole illusion” http://www.sandlotscience.com/Ambiguous/Barberpole_Illusion.htm http://www.liv.ac.uk/~marcob/Trieste/barberpole.html
Not quite… where do the vectors point? (See Hildreth, a long time ago…)
![Page 24: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/24.jpg)
No. of unknowns vs equations (pixels)
So if the brightness constraint equation gives us more unknowns than pixels, how do we recover motion?
![Page 25: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/25.jpg)
Smooth Optical Flow (Horn & Schunk, 1980)
• Formulate Error in Optical Flow constraint:
•We need additional constraints (pardon the integrals)
2( )c x y t
image
e I u I v I dx dy
![Page 26: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/26.jpg)
Smooth Optical Flow (Horn & Schunk, 1980)
• Smoothness constraint: Motion field tends to vary smoothly over the image
•Penalized for changes in 𝑢 and 𝑣 over image
2 2 2 2( ) ( )s x y x y
image
e u u v v dx dy
![Page 27: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/27.jpg)
Smooth Optical Flow (Horn & Schunk, 1980)
Given both terms:
2 2 2 2( ) ( )s x y x y
image
e u u v v dx dy
2( )c x y t
image
e I u I v I dx dy
![Page 28: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/28.jpg)
Smooth Optical Flow (Horn & Schunk, 1980)
Find (𝑢, 𝑣) at each image point that minimizes:
s ce e e
weighting factor
![Page 29: CS4495/6495 Introduction to Computer Vision · 2015. 3. 12. · Introduction to Computer Vision . Motion estimation techniques Feature-based methods Direct, dense methods . Motion](https://reader036.vdocuments.us/reader036/viewer/2022062403/5fd9c23841d73c6296184667/html5/thumbnails/29.jpg)
Dense Flow: Summary
• Impose a constraint on the flow field in general to make the problem solvable
• Strength: Allows you to bias your solution with a prior (if you have one)
• But there are better ways to increase the number of equations…