motion estimation using markov random fields hrvoje bogunović image processing group faculty of...
TRANSCRIPT
![Page 1: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/1.jpg)
Motion Estimation using Markov Random Fields
Hrvoje Bogunović
Image Processing Group
Faculty of Electrical Engineering and Computing
University of Zagreb
Summer School on Image Processing, Graz 2004
![Page 2: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/2.jpg)
Overview
• Introduction
• Optical flow
• Markov Random Fields
• OF+MRF combined
• Energy minimization techniques
• Results
![Page 3: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/3.jpg)
Introduction
• Input:– Sequence of images (Video)
• Problem– Extract information about motion
• Applications– Detection, Segmentation, Tracking, Coding
![Page 4: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/4.jpg)
Spatio-temporal spectrum
φ
f
![Page 5: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/5.jpg)
Motion – aliasing
φ
f
1/x
1/t
Large area flicker
Loss of spatialresolution
![Page 6: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/6.jpg)
Large motions - temporal aliasing
φ
fTemporal aliasing
Great loss of spatial resolution
![Page 7: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/7.jpg)
Temporal anti-aliasing
φ
f
• No more overlaping on the f axis. • filtering (anit-aliasing) is performed after sampling, hence the blurring
![Page 8: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/8.jpg)
Motion – eye tracking
φ
f
![Page 9: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/9.jpg)
Motion estimation
• Images are 2-D projections of the 3-D world.
• Problem is represented as a labeling one.– Assign vector to pixel
• Vector field field of movement• Low level vision
– No interpretation
![Page 10: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/10.jpg)
Example Ideal
![Page 11: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/11.jpg)
Problems
• Problem is inherently ill-posed– Solution is not unique
• Aperture problem– Specific to local methods
![Page 12: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/12.jpg)
Optical flow
• Main assumption: Intensity of the object does not change as it moves– Often violated
• First solved by Horn & Schunk– Gradient approach
• Other approaches include– Frequency based– Using corresponding features
![Page 13: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/13.jpg)
Image differencing
![Page 14: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/14.jpg)
Gradient approach
• Local by nature. Aperture problem is significant.
• Image understanding is not required– Very low level
![Page 15: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/15.jpg)
Horn & Schunk
• Intensity stays the same in the direction of movement. I(x,y,t)
• After derivation
![Page 16: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/16.jpg)
Horn & Schunk
• Spatial gradients Ix,Iy
– e.g. Sobel operator
• Temporal gradient It
– Image subtraction
( , ) ( , ) 0x y t
t
I I u v I
I I v
![Page 17: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/17.jpg)
Regularization
• Tikhonov regularization for ill-posed problems
• Add the smoothness term
• Energy function
![Page 18: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/18.jpg)
Result
![Page 19: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/19.jpg)
Problems of the H-S method
• Assumption: There are no discontinuities in the image– Optical flow is over-smoothed.
• Gradient method. Only the edges which are perpendicular to motion vector contribute
• Image regions which are uniform do not contribute.
• Difficulty with large motions (spatial filtering)
![Page 20: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/20.jpg)
Optical flow enhancement
• Optical flow can be piecewise smooth
• Discontinuities can be incorporated
• Solution: use the spatial context
• Problem is posed as a solution of the Bayes classifier. Solution in optimization sense. Search for optimum
![Page 21: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/21.jpg)
Bayes classifier
• Main equation
• Solution using MAP estimation
( , ) ( )( | )
( )
P hypothesis observation P hypothesisP hypothesis observation
P observation
![Page 22: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/22.jpg)
Markov Random Fields
• Suitable: Problems posed as a visual labeling problemn with contextual constraints
• Useful to encode a priori knowledge– required for bayes classifier (smoothness prior)– equvalence to Gibbs random fields (gibbs
distribution, exponential like)• Neighbourhoods• Cliques
– pairs,triples of neighbourhood points)– build the energy function
![Page 23: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/23.jpg)
MRF
• Define sites: rectangular lattice
• Define labels
• define neighbourhood: 4,8 point
• Field is MRF:– P(f)>0
– P(fi|f{S-i})=P(fi|Ni)
![Page 24: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/24.jpg)
Coupled MRF
• Field F is an optical flow field• Field L is a field of discontinuities
– line process
• Position of the two fields.
![Page 25: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/25.jpg)
Context
• neighbourhoods and cliques
![Page 26: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/26.jpg)
Motion estimation equations
![Page 27: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/27.jpg)
Energy for MAP estimation
Parameters are estimated ad hoc
![Page 28: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/28.jpg)
Energy minimization
• Global minimum– Simulated annealing– Genetic Algorithms
• Local minimum– Iterated Conditional Modes (ICM) (steepest
decent)– Highest Confidence First (HCF)
• specific site visiting
![Page 29: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/29.jpg)
Simulated annealing(1) Find the initial temperature of the system T.
(2) Assign initial values of the field to random
(3) For every pixel:
Assign random value to f(i,j)
Calculate the difference in energy before and after If the change is better (diff>0) keep it.
Else keep it with the probability exp(diff/T)
(4) Repeat (3) N1 times
(5) T = f(T) where f decreases monotono
(6) Repeat (3-5) N2 times
![Page 30: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/30.jpg)
Results (Square)
Horn-Schunk OF OF+MRF
![Page 31: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/31.jpg)
Taxi
![Page 32: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/32.jpg)
Results (Taxi)
![Page 33: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/33.jpg)
Line process result (Taxi)
![Page 34: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/34.jpg)
Cube
![Page 35: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/35.jpg)
Results (cube)
![Page 36: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/36.jpg)
Line process result (cube)
![Page 37: Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb](https://reader036.vdocuments.us/reader036/viewer/2022062410/5697c0301a28abf838cdaf4f/html5/thumbnails/37.jpg)
Q & A