![Page 1: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/1.jpg)
Motion
![Page 2: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/2.jpg)
![Page 3: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/3.jpg)
Optical flowMeasurement of motion at every pixel
![Page 4: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/4.jpg)
Problem definition: optical flow
• How to estimate pixel motion from image H to image I?
• Solve pixel correspondence problem– given a pixel in H, look for nearby pixels of the same color in I
Key assumptions• color constancy: a point in H looks the same in I
– For grayscale images, this is brightness constancy• small motion: points do not move very far
This is called the optical flow problem
![Page 5: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/5.jpg)
![Page 6: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/6.jpg)
![Page 7: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/7.jpg)
![Page 8: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/8.jpg)
![Page 9: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/9.jpg)
![Page 10: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/10.jpg)
![Page 11: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/11.jpg)
Lukas-Kanade flow• Prob: we have more equations than unknowns
• The summations are over all pixels in the K x K window• This technique was first proposed by Lukas & Kanade (1981)
– described in Trucco & Verri reading
Solution: solve least squares problem• minimum least squares solution given by solution (in d) of:
![Page 12: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/12.jpg)
![Page 13: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/13.jpg)
![Page 14: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/14.jpg)
![Page 15: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/15.jpg)
Iterative RefinementIterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving Lucas-Kanade equations
2. Warp H towards I using the estimated flow field- use image warping techniques
3. Repeat until convergence
![Page 16: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/16.jpg)
![Page 17: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/17.jpg)
![Page 18: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/18.jpg)
image Iimage H
Gaussian pyramid of image H Gaussian pyramid of image I
image Iimage H u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
![Page 19: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/19.jpg)
image Iimage J
Gaussian pyramid of image H Gaussian pyramid of image I
image Iimage H
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
![Page 20: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/20.jpg)
Multi-resolution Lucas Kanade Algorithm
Compute Iterative LK at highest levelFor Each Level i
• Take flow u(i-1), v(i-1) from level i-1• Upsample the flow to create u*(i), v*(i)
matrices of twice resolution for level i.• Multiply u*(i), v*(i) by 2• Compute It from a block displaced by
u*(i), v*(i)• Apply LK to get u’(i), v’(i) (the correction
in flow)• Add corrections u’(i), v’(i) to obtain the
flow u(i), v(i) at ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)
![Page 21: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/21.jpg)
Optical Flow Results
![Page 22: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/22.jpg)
Optical Flow Results
![Page 23: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/23.jpg)
Optical flow Results
![Page 24: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/24.jpg)
Optical flow competition• http://vision.middlebury.edu/flow/eval/
![Page 25: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/25.jpg)
Global Flow• Dominant Motion in the image
– Motion of all points in the scene– Motion of most of the points in the scene– A Component of motion of all points in the scene
• Global Motion is caused by – Motion of sensor (Ego Motion)– Motion of a rigid scene
• Estimation of Global Motion can be used to– Video Mosaics– Image Alignment (Registration)– Removing Camera Jitter– Tracking (By neglecting camera motion)– Video Segmentation etc.
![Page 26: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/26.jpg)
Global Flow
Application: Image Alignment
![Page 27: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/27.jpg)
Global Flow• Special Case of General Optical Flow
Problem• Can be solved by using Lucas Kanade
algorithm.• Specialized algorithms exist that perform
better by further constraining the problem.
![Page 28: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/28.jpg)
Motion Models• First we look for a parametric form of
global flow vector.Global Flow occurs because of 3D rigid motion of either the sensor or the scene.
Z
Y
X
XYZ
T
T
T
Z
Y
X
T
Z
Y
X
RRRT
Z
Y
X
R
Z
Y
X
coscoscossinsin
cossinsinsincoscoscossinsinsinsincos
sinsincossincossincoscossinsincoscos
sin
1cos(If is small)
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
1
1
1
Also neglecting the higher order terms
3D Rigid Motion
![Page 29: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/29.jpg)
Iterative RefinementIterative Algorithm
1. Estimate global flow by solving linear system Aa=B
2. Warp H towards I using the estimated flow- use image warping techniques (to be covered later)
3. Repeat until convergence or a fixed number of iterations
![Page 30: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/30.jpg)
image Iimage H
Gaussian pyramid of image H Gaussian pyramid of image I
image Iimage H u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine global flow estimation
![Page 31: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/31.jpg)
image Iimage J
Gaussian pyramid of image H Gaussian pyramid of image I
image Iimage H
Coarse-to-fine global flow estimation
Compute Flow Iteratively
Compute Flow Iteratively
warp & upsample
.
.
.
![Page 32: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/32.jpg)
Coarse-to-fine global motion estimation
![Page 33: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/33.jpg)
Basic Components
• Pyramid Construction• Motion Estimation• Image Warping• Coarse to Fine Refinement
![Page 34: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/34.jpg)
Result of Global Motion Estimation
Affine Model4 Pyramids Level5 Iterations/Pyramid Level
Image ‘t’
Image ‘t+1’
output
![Page 35: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/35.jpg)
Video Mosaic
![Page 36: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/36.jpg)
Unwarp Mosaic• http://research.microsoft.com/en-us/um/ca
mbridge/projects/visionimagevideoediting/unwrap/
![Page 37: Motion. Optical flow Measurement of motion at every pixel](https://reader036.vdocuments.us/reader036/viewer/2022081506/56649c745503460f949272fa/html5/thumbnails/37.jpg)
Summary• Things to take away from this lecture
– Optical flow problem definition– Aperture problem and how it arises
• Assumptions– Brightness constancy, small motion, smoothness– Derivation of optical flow constraint equation– Lukas-Kanade equation– Conditions for solvability– meanings of eigenvalues and eigenvectors
• Iterative refinement– Newton’s method– Coarse-to-fine flow estimation
• Applications– Image alignment / Video mosaic