![Page 1: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/1.jpg)
Motion & Tracking
![Page 2: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/2.jpg)
Motion is the only Cue
![Page 3: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/3.jpg)
G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Motion and perceptual organization
• Even “impoverished” motion data can evoke a
strong percept
![Page 4: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/4.jpg)
Motion Estimation & Tracking
Sequence of images contains information about the scene,We want to estimate motion
![Page 5: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/5.jpg)
5
Applications
• video compression
• 3D reconstruction
• segmentation
• object detection
• activity detection
• key frame extraction
• interpolation in time
![Page 6: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/6.jpg)
Tracking
Procerus Technologies
http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies
![Page 7: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/7.jpg)
Video Stabilization
![Page 8: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/8.jpg)
Structure From Motion
![Page 9: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/9.jpg)
Structure From Motion
![Page 10: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/10.jpg)
Motion Field & Optical Flow
Optical center
2D motion field
Projection on the image
plane of the 3D velocity
of the scene
3D motion field
Image intensity
I1
I2
Motion vector - ?
![Page 11: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/11.jpg)
Motion Field & Optical Flow
• Motion field
– The true 2d projection of the 3d motion
• Optical flow
– The measured 2d flow
![Page 12: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/12.jpg)
What we are able to perceive is just an apparent motion, called
Optical flow
(motion, observable only through intensity variations)
Intensity remains constant –
no motion is perceived
No object motion, moving light source
produces intensity variations
Optical Flow ≠ Motion Field
![Page 13: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/13.jpg)
Optical Flow vs. Motion Field
• Optical flow: the measured flow
• Motion field: the real motion
The Barberpole Illusion
![Page 14: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/14.jpg)
![Page 15: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/15.jpg)
Examples of Motion Fields
(a) Motion field of a pilot looking straight ahead while approaching a fixed
point on a landing strip. (b) Pilot is looking to the right in level flight.
(a) (b)
Global Motion
![Page 16: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/16.jpg)
Examples of Motion Fields
(a) (b)
(c) (d)
(a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background.
![Page 17: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/17.jpg)
)1( +tI
What is Optical Flow?
Optical Flow1p
2p
3p
4p
}{),( iptI
1vr
2vr
3vr
4vr
}{ ivr
Velocity vectors
Source: Stanford CS223B Computer Vision, Winter 2006, lecture notes
![Page 18: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/18.jpg)
Optical Flow Based Search
…
From Darya Frolova and Denis Simakov
![Page 19: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/19.jpg)
Optical flow-methods
Correlation-based techniques - compare
parts of the first image with parts of the
second in terms of the similarity in brightness
patterns in order to determine the motion
vectors
Feature-based methods - compute and
analyze Optical Flow at small number of
well-defined image features
Gradient-based methods - use
spatiotemporal partial derivatives to
estimate flow at each point
![Page 20: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/20.jpg)
Optical Flow - Assumptions
• Constant brightness assumption
The apparent brightness of moving objects remains
constant
• Small motion
Points do not move very far
• Spatial coherence
– Neighboring points in the scene typically belong to the
same surface and hence typically have similar motions
• Temporal coherence
– The image motion of a surface patch changes
gradually over time
![Page 21: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/21.jpg)
Estimating Optical Flow
• Constant brightness assumption
( )tyxI ,, ( )dttdyydxxI +++ ,,=
Time = t Time = t+dt
* From Lihi Zelnik-Manor
(x,y) (x+dx,y+dy)
![Page 22: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/22.jpg)
( ) tdt
dIv
dy
dIu
dx
dItyxIttvyuxI ∆+++=∆+++ ),,(,,
)tt,vy,ux(I)t,y,x(I ∆+++=
vIuII0 vxt ++=
Take the Taylor series expansion of I :
using brightness assumption:
Constant Brightness Assumption - 2D Case:
Optical Flow Equation
![Page 23: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/23.jpg)
vIuII0 vxt ++=
x
( )t,xI ( )tt,xI ∆+
u
tI
Optical Flow Equation- Intuition
uII xt =−
The change in value It at a pixel P is dependent on:
The distance moved (u).
x
II x ∆
∆=
x∆I∆
![Page 24: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/24.jpg)
tyx vu III −=+
Optical Flow Equation
tIvuI −=⋅∇ ],[
Only the component of the flow in the gradient direction can be determined
The component of the flow parallel to an edge is unknown
![Page 25: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/25.jpg)
Optical Flow Equation
Intuition - The Barber Pole Illusion.
true motion
vector
perp.
component
parallel.
component
![Page 26: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/26.jpg)
The Aperture Problem
• Different motions – classified as similar
source: Ran Eshel
![Page 27: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/27.jpg)
The Aperture Problem
• Similar motions – classified as different
source: Ran Eshel
![Page 28: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/28.jpg)
Frame 1
flow (1): true motionflow (2)
From Darya Frolova and Denis Simakov
Ambiguity of Optical Flow
![Page 29: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/29.jpg)
tyx vu III −=+
Optical Flow Equation
Shoot! One equation, two velocity unknowns (u,v) ...
Solving for u,v:
![Page 30: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/30.jpg)
Spatial coherence
• Impose additional constraints
– Assume the pixel’s neighbors have the same (u,v)
( ) ( )( ) ( )
( ) ( )
( )( )
( )
−=
−=+
Nt
t
t
NyNx
yx
yx
tyx
v
u
vu
p
p
p
pp
pp
pp
2
1
22
11
I
I
I
II
II
II
III
MMM
A
N×2
x
2×1b
N×1
p1
pN
bAx =
( ) bAAAxtt 1−
=
bAAxAtt =
p2
![Page 31: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/31.jpg)
Lukas Kanade Scheme
Equivalent to Solving least squares:
ATA ATb
bAx)AA( TT =
• The summations are over all pixels in the K x K window
• This technique was first proposed by Lukas & Kanade (1981)
x
![Page 32: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/32.jpg)
−=
∑∑
∑∑∑∑
ty
tx
yyx
yxx
v
u
II
II
III
III2
2
Lucas-Kanade Equation
I R IxIy It
22×
AAt
12×
x
12×
bAt
![Page 33: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/33.jpg)
When can we solve LK Eq ?Optimal (u, v) satisfies Lucas-Kanade equation
• ATA should be invertible
• The eigenvalues of ATA should not be too small (noise)
• ATA should be well-conditioned:
λ1/ λ2 should not be too large (λ1 = larger eigenvalue)
ATA is invertible when there is no aperture problem
![Page 34: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/34.jpg)
Hessian Matrix
Ix = 0
Iy = 0
M =
M =0 0
0 0Non Invertable
Ix = 0
Iy = kM =
0 0
0 k2Non Invertable
Ix = k
Iy = 0M =
k2 0
0 0Non Invertable
Ix = k1
Iy = k2RM =
k2 0
0 0Non Invertable
k1, k2 correlated(R = rotation)
Ix = k1
Iy = k2M =
k12 0
0 k22
Invertable
k1 * k2 = 0
![Page 35: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/35.jpg)
Classification of AtA
50 100 150 200
20
40
60
80
100
120
140
160
180
200
220
2 4 6 8 10 12
2
4
6
8
10
12
large λ1, small λ2
2 4 6 8 10 12
2
4
6
8
10
12
large λ1, large λ2
2 4 6 8 10 12
2
4
6
8
10
12
small λ1, small λ2
=
∑∑∑∑
2
2
yyx
yxxt
III
IIIΑA
The Hessian Matrix
![Page 36: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/36.jpg)
Optical Flow: Iterative Estimation
Optical flow is first order approximation (first order Taylor).
What if first order is not good approx of the area around a pixel?
( ) tdt
dIv
dy
dIu
dx
dItyxIttvyuxI ∆+++=∆+++ ),,(,,
![Page 37: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/37.jpg)
Optical Flow: Iterative Estimation
xx0
Estimate:
u0= 0
u1= u0+ û
estimate
update
û
f2(x)f1(x)
Initial guess:
![Page 38: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/38.jpg)
xx0
u1
u2= u1+ û
f1(x-u1) f2(x)
Estimate:
Initial guess: estimate
update
û
Optical Flow: Iterative Estimation
![Page 39: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/39.jpg)
xx0
f2(x)f1(x-u2)
u2
u3= u2+ ûEstimate:
Initial guess: estimate
update
û
Optical Flow: Iterative Estimation
![Page 40: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/40.jpg)
xx0
f1(x-u3) ≈ f2(x)
Optical Flow: Iterative Estimation
![Page 41: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/41.jpg)
( ) ( )∑∈
++=Ayx
tyxvuvu
vuvuSSD,
2
,,min,min III
0=++ tyx vu III
Iterative Refinement
Iterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving Lucas-Kanade
equations:
2. Warp I1 towards I2 using the estimated flow field
3. Repeat until convergence
Find
![Page 42: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/42.jpg)
Optical Flow: Iterative Estimation
• Some Implementation Issues:
– Warp one image, take derivatives of the other
so you don’t need to re-compute the gradient
after each iteration.
– Often useful to low-pass filter the images
before motion estimation (for better derivative
estimation, and linear approximations to
image intensity)
![Page 43: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/43.jpg)
Optical Flow
http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/
![Page 44: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/44.jpg)
Optical Flow: Iterative Estimation
![Page 45: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/45.jpg)
Optical Flow
![Page 46: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/46.jpg)
Revisiting the small motion assumption
• Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 47: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/47.jpg)
Aliasing in large motions
Temporal aliasing causes ambiguities in optical flow because
images can have many pixels with the same intensity.
I.e., how do we know which ‘correspondence’ is correct?
nearest match is
correct (no aliasing)
nearest match is
incorrect (aliasing)
To overcome aliasing: coarse-to-fine estimation.
actual shift
estimated shift
![Page 48: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/48.jpg)
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 49: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/49.jpg)
image It-1 image I
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow
estimation
![Page 50: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/50.jpg)
image Iimage J
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1
Coarse-to-fine optical flow
estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
![Page 51: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/51.jpg)
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 52: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/52.jpg)
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 53: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/53.jpg)
Optical Flow for Affine Transformation
+
=
f
c
y
x
ed
ba
v
u
tyx feydxcbyax III −=++⋅+++⋅ )()(
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( )
( )
−=
Nt
t
NyNyNyNxNxNx
yyyxxx
f
e
d
c
b
a
yxyx
yxyx
p
p
pppppp
pppppp 1111111
I
I
IIIIII
IIIIII
MM
![Page 54: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/54.jpg)
Optical Flow for Affine Transformation
1616
66
222
222
22222
22222
22222
22222
×××
−=
∑∑∑∑∑∑
∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑
bAx
AA
II
II
II
II
II
II
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
tt
ty
tx
ty
tx
ty
tx
yyyyxyxyx
yxyxyxxxx
yyyyxyxyx
yxyxyxxxx
yyyyxyxyx
yxyxyxxxx
x
y
y
x
f
e
d
c
b
a
yxyx
yxyx
xxyxxxyx
yyxyyyxy
yyxyyyxy
xxyxxxyx
![Page 55: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/55.jpg)
Errors in Lukas-Kanade
Suppose ATA is easily invertible
Suppose there is minimal noise in the image.
When our assumptions are violated:
– Brightness constancy is not satisfied
– The motion is not small
– A point does not move like its neighbors
• window size is too large
• what is the ideal window size?
What are the potential causes of errors in the LK approach?
![Page 56: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/56.jpg)
Horn and Schunck (1981)
• Two criteria:
– Small error in optical flow constraint equation, Fh(u,v)
– Optical flow is smooth, Fs(u,v)
• Minimize a combined error functional
Fc(u,v) = Fh(u,v) + λ Fs(u,v)
λ is a weighting parameter
Horn & Schunck Optical Flow - Extension
![Page 57: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/57.jpg)
Horn & Schunck Optical Flow - Extension
Fc(u,v) = Fh(u,v) + λ Fs(u,v)
Solve by means of calculus of variation (iteratively).
( ) ( ) smoothness from departure :dydxvvuue2
y2
x
D
2y
2xs +++= ∫∫
( ) equation flow opticalin error :2
∫∫ ++=D
tyxc dydxIvIuIe
( ) ( ) min2
2
2
2
2 →∇+∇++⋅∇∫∫ dydxvuII t λu
Fs(u,v) =
Fh(u,v) =
Smoothness term
(regularization term)Data term
![Page 58: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/58.jpg)
Horn and Schunck
• Assumptions
– brightness constancy
– neighboring velocities are nearly identical
• Properties
+ incorporates global information
+ image first derivatives only
- iterative
- smoothes across motion boundaries
![Page 59: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/59.jpg)
1 iteration 4 iteration
16 iteration 64 iteration
Horn & Schunck - Example
1 frame of sequence
![Page 60: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/60.jpg)
Discontinuities
But smoothing term does not allow to save discontinuities
Discontinuities near edges are lost
Synthetic example
(method of Horn and Schunck)
Use edge preserving approaches
![Page 61: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/61.jpg)
Reconsidering Smoothness Term
( ) ( ) min2
2
2
2
2 →∇+∇++⋅∇∫∫ dydxvuII t λu
Smoothness term
(regularization term)Data term
Use different
regularization in
smoothness term
![Page 62: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/62.jpg)
Homogeneous propagation
∫ ∇+∇video
vu22
min- flow in the x direction
- flow in the y direction
- gradient∇
),,( tyxu),,( tyxv
[Horn&Schunck 1981]
This constraint is not correct on motion boundaries => over-smoothing of the resulting flow
![Page 63: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/63.jpg)
Robustness to flow discontinuities
(also known as isotropic flow-driven regularization)
∫ ∇+∇video
vu )||||(min 22φ
:φ
[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]High accuracy optical flow estimation based on a theory for warping
22 ε+x
ε
![Page 64: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/64.jpg)
64
Brightness is not always constant
Rotating
cylinder
Brightness constancy
does not always hold
),,()1,,( tyxItvyuxI ≠+++
inte
nsity
position
),,()1,,( tyxItvyuxI ∇=+++∇
Gradient constancy holds
inte
nsity d
erivative
position
![Page 65: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/65.jpg)
http://cbia.fi.muni.cz/user_dirs/xulman/gtgen/gt_application.png
Horn & Schunck
![Page 66: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/66.jpg)
https://www.youtube.com/watch?v=Ox8oI7nzSPw&nohtml5=False
Navigation
![Page 67: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/67.jpg)
http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies
Navigation
![Page 68: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/68.jpg)
Car Tracking Using Optical Flow
![Page 69: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/69.jpg)
Automatic Plane landing
Using Horizon and Optical Flow
https://www.youtube.com/watch?v=U9iy1B5QG-0
![Page 70: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/70.jpg)
Tracking
Some slides are from Sebastian Thrun, Rick Szeliski, Hendrik Dahlkamp, Wolfram
Burgard, Kim Chule Hwon, D. Forsyth, M. Isard, T. Darrell Also From Yuri Rapaport
![Page 71: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/71.jpg)
Examples
![Page 72: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/72.jpg)
Tracking applications
• Tracking missiles
• Tracking heads/hands/drumsticks
• Extracting lip motion from video
• Lots of computer vision applications
• Economics
• Navigation
SIGGRAPH 2001
![Page 73: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/73.jpg)
Detect Foreground objects –
Background subtraction
Compare the current frame to a reference image and label all the different pixels as motion.
![Page 74: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/74.jpg)
Issues:
Detect Foreground objects –
Background subtraction
![Page 75: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/75.jpg)
Examples
![Page 76: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/76.jpg)
Tracking: First Idea!
estimateinitial position
x
y
x
y
state dynamics
x
y
measurement
x
y
Starting point Possible new positions Combine Measurement
and possibilities to obtain
estimate.
![Page 77: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/77.jpg)
Tracking
• Bayes Filtering
• Kalman Filtering
• Particle Filtering
![Page 78: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/78.jpg)
On-line posterior density estimation algorithms that
estimate the posterior density of the state-space by
directly implementing the Bayesian recursion equations.
Particle Filters Condensation Algorithm
Sequential Monte Carlo (SMC)
HUH???
![Page 79: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/79.jpg)
Partially Observable Chains
z2 z3 z4Measurements z1
state x4state x3state x2state x1
)( 0xp
( )tt zzxp ,,1 LWe have to estimate:
![Page 80: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/80.jpg)
Assumptions
• The system follows a Markovian process
• The observations are independent
( ) ( )111 ,, −− = tttt xxpxxxp L
( ) ( )tttt xzpxxzp =,,1 L
![Page 81: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/81.jpg)
System Dynamics
t-1 t t+1 time
zt-1 zt zt+1
xt-1 xt xt+1
Measurements(observed)
States(to be estimated)( )1−tt xxp
( )tt xzp
![Page 82: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/82.jpg)
82
Bayes Filters
System state dynamics (uncertain)
Observation dynamics (measurement error)
We are interested in: posterior probability
Estimating system state from noisy observations
( )tt zzxp ,,1 L
( )1−tt xxp
( )tt xzp
![Page 83: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/83.jpg)
Recursive Bayes Filters
• Sequential update of previous estimate
• Allow on-line processing of data
• Rapid adaptation to changing signals
characteristics
• Consists of two steps:
( ) ( )1:11:11 −−− → tttt zxpzxp
( ) ( )ttttt zxpzzxp :11:1 , →−
– Prediction step:
– Update step:
![Page 84: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/84.jpg)
84
Bayes Formula
evidence
prior likelihood
)(
)()|()(
)()|()()|(),(
⋅==
⇒
==
yP
xPxyPyxP
xPxyPyPyxPyxP
![Page 85: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/85.jpg)
85
1:( 1) 1:( 1)( | , ) ( | ) ( | )t t t t t t t tp x z z p z x p x zα− −=
1:( 1) 1 1 1:( 1) 1( | ) ( | ) ( | )t t t t t t tp x z p x x p x z dx− − − − −= ∫
1( | )
( | )
t t
t t
p x x
p z x
−Motion Model
Observation Model
Start from: 0 00 0 0
0
( | )( | ) ( )
( )
p z xp x z p x
p z=
Predict:
Update:
Recursive Bayes Filters
![Page 86: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/86.jpg)
Bayes Filters
Prior P(x)
Measurement
evidence
P(z|x)
Posterior
P(x|z)
)()|()|( xpxzpzxp ∝
dxxpxxpxp )()|'()'( ∫=
Update
Predict
Predict the new prior p(x’)
![Page 87: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/87.jpg)
On-line posterior density estimation algorithms that
estimate the posterior density of the state-space by
directly implementing the Bayesian recursion equations.
Particle Filters Condensation Algorithm
Sequential Monte Carlo (SMC)
Uses Particles (many of them) to represent the distributions.
![Page 88: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/88.jpg)
Want to estimate x position of plane flying across a Fiord
Demo – Particle Filtering
Particle Filter Explained without Equations
Andreas Svensson, Uppsala Univ.
![Page 89: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/89.jpg)
Demo – Particle Filtering
Want to estimate x position of plane flying across a Fiord
Particle Filter Explained without Equations
Andreas Svensson, Uppsala Univ.
![Page 90: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/90.jpg)
Demo – Particle Filtering
Zt = elevationt - heightt∈ [0; 1].
Xt ∈ [0; 5000].
Xt can be estimated from Xt-1.
Zt can be calculated from Xt .
Velocity distributes uniformly from 800 to 1000 km/h.
Assume have map of elevation.
Zt
Xt
![Page 91: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/91.jpg)
Demo – Particle Filtering
Particle Filter Explained without Equations
Andreas Svensson, Uppsala Univ.https://www.youtube.com/watch?v=aUkBa1zMKv4
![Page 92: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/92.jpg)
Particle Filtering
![Page 93: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/93.jpg)
Occlusion
![Page 94: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/94.jpg)
Challenges
![Page 95: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/95.jpg)
Resolving Occlusion
• Types of occlusions:
– Self occlusion
– Inter-object occlusion
– Occlusion by the background scene structure
![Page 96: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/96.jpg)
“Detection and Tracking of Occluded People”
Siyu Tang, Mykhaylo Andriluka, Bernt Schiele
Use double person detector
![Page 97: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/97.jpg)
“Detection and Tracking of Occluded People”
Siyu Tang, Mykhaylo Andriluka, Bernt Schiele
![Page 98: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/98.jpg)
Shadows
![Page 99: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/99.jpg)
Multiple Cameras
• Challenge : How to integrate all the detections from the different cameras
![Page 100: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/100.jpg)
![Page 101: Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I](https://reader035.vdocuments.us/reader035/viewer/2022071016/5fcec4104040c96d3a026353/html5/thumbnails/101.jpg)
source: Ran Eshel