powerpoint presentation · title: powerpoint presentation author: guido gerig created date:...
TRANSCRIPT
![Page 1: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/1.jpg)
Optical Flow Ib
Guido Gerig
CS 6320, Spring 2012
(credits: slides modified from Marc Pollefeys
UNC Chapel Hill, Comp 256, and from K.H. Shafique, UCSF, CAP5415, and from S. Narasimhan, CMU)
![Page 2: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/2.jpg)
Materials
• Gary Bradski & Sebastian Thrun, Stanford CS223 http://robots.stanford.edu/cs223b/index.html
• S. Narasimhan, CMU: http://www.cs.cmu.edu/afs/cs/academic/class/15385-s06/lectures/ppts/lec-16.ppt
• M. Pollefeys, ETH Zurich/UNC Chapel Hill: http://www.cs.unc.edu/Research/vision/comp256/vision10.ppt
• K.H. Shafique, UCSF: http://www.cs.ucf.edu/courses/cap6411/cap5415/
– Lecture 18 (March 25, 2003), Slides: PDF/ PPT
• Jepson, Toronto: http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
• Original paper Horn&Schunck 1981: http://www.csd.uwo.ca/faculty/beau/CS9645/PAPERS/Horn-Schunck.pdf
• MIT AI Memo Horn& Schunck 1980: http://people.csail.mit.edu/bkph/AIM/AIM-572.pdf
• Bahadir K. Gunturk, EE 7730 Image Analysis II
• Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama
![Page 3: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/3.jpg)
3
Optical Flow
• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow
![Page 4: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/4.jpg)
Lucas & Kanade
•Assume single velocity for all pixels within a patch.
•Integrate over a patch.
![Page 5: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/5.jpg)
Lucas & Kanade
![Page 6: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/6.jpg)
Lucas & Kanade
![Page 7: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/7.jpg)
7
02),(
02),(
tyxy
tyxx
IvIuIIdv
vudE
IvIuIIdu
vudE
![Page 8: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/8.jpg)
8
![Page 9: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/9.jpg)
Discussion • Horn-Schunck: Add smoothness constraint.
• Lucas-Kanade: Provide constraint by minimizing over local neighborhood:
• Horn-Schunck and Lucas-Kanade optical methods work only for small motion.
• If object moves faster, the brightness changes rapidly, derivative masks fail to estimate spatiotemporal derivatives.
• Pyramids can be used to compute large optical flow vectors.
![Page 10: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/10.jpg)
Iterative Refinement
• Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation
• Warp one image toward the other using the estimated flow field
(easier said than done)
• Refine estimate by repeating the process
![Page 11: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/11.jpg)
Reduce the Resolution!
![Page 12: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/12.jpg)
12
Optical Flow
• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow
![Page 13: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/13.jpg)
![Page 14: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/14.jpg)
Results
![Page 15: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/15.jpg)
Revisiting the Small Motion Assumption
• Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem?
![Page 16: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/16.jpg)
image I image H
Gaussian pyramid of image H Gaussian pyramid of image I
image I image H u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine Optical Flow Estimation
![Page 17: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/17.jpg)
image I image J
Gaussian pyramid of image H Gaussian pyramid of image I
image I image H
run iterative OF
run iterative OF
upsample
.
.
.
Coarse-to-fine Optical Flow Estimation
![Page 18: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/18.jpg)
18
![Page 19: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/19.jpg)
J Jw I warp refine
ina
a
+
J Jw I warp refine
a
a
+
J
pyramid
construction
J Jw I warp refine
a
+
I
pyramid
construction
outa
Coarse-to-Fine Estimation
![Page 20: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/20.jpg)
Video Segmentation
![Page 21: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/21.jpg)
Next: Motion Field
Structure from Motion
![Page 22: PowerPoint Presentation · Title: PowerPoint Presentation Author: Guido Gerig Created Date: 4/18/2012 12:27:20 PM](https://reader035.vdocuments.us/reader035/viewer/2022062605/5fdab15e8006da6dc21973fe/html5/thumbnails/22.jpg)
Motion Field
X
Y
or
ir
'f
• Image velocity of a point moving in the scene
Perspective projection: Zf
o
o
oi
r
Zr
rr
ˆ'
1
22
''Zr
Zvr
Zr
rZvvZrrv
o
oo
o
ooooii ff
dt
d
Motion field
tov
tiv Scene point velocity:
Image velocity:
dt
d oo
rv
dt
d ii
rv
Z
Z