visual motion analysis and representationyfwang/courses/cs281b/notes/7-motion2.… · visual motion...
TRANSCRIPT
![Page 1: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/1.jpg)
Visual Motion
Analysis and Representation
![Page 2: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/2.jpg)
Example
Ullman’s concentric counter-rotating cylinder experiment
Two concentric cylinders of different radii
W. a random dot pattern on both surfaces (cylinder
surfaces and boundaries are not displayed)
Stationary: not able to tell them apart
Counter-rotating: structures apparent
![Page 3: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/3.jpg)
Motion helps in
segmentation (two structures)
identification (two cylinders)
Example (cont.)
![Page 4: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/4.jpg)
Classes of Techniques
Feature-based methods
Extract visual features (corners, textured areas) and track them
Sparse motion fields, but possibly robust tracking
Suitable especially when image motion is large (10s of pixels)
Direct-methods (Pixel-based methods)
Directly recover image motion from spatio-temporal image brightness
variations
Global motion parameters directly recovered without an intermediate
feature motion calculation
Dense motion fields, but more sensitive to appearance variations
Suitable for video and when image motion is small (< 10 pixels)
Szeliski
![Page 5: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/5.jpg)
Optical flow and motion analysis
Now we move to considering images that vary over time –
image sequences
Typical case is video – images captured at 30 frames/second (or
15, or 60, or ...)
I(x, y, t) I1(x,y) = I(x, y, t1) , I2(x,y) = I(x, y, t2) , etc.
“Spatial-temporal space” describes (x, y, t)
t
x
y
What can change between It
and It+1?
What do images close in time
have in common?
![Page 6: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/6.jpg)
Spatio-temporal image data
(examples)
![Page 7: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/7.jpg)
Frames
x-t slice
t
x
![Page 8: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/8.jpg)
Optical flow and motion analysis
Optical flow is the apparent motion of brightness patterns
in the image sequence
A 2D vector at each point – a vector field
The motion field is the true motion (3D) at each point,
mapped onto the 2D image
A vector field
They are not always
the same
E.g., white, featureless ball?
In general, we estimate the motion field by
computing the optical flow
The motion field is not directly
observed
![Page 9: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/9.jpg)
Example
![Page 10: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/10.jpg)
Example
![Page 11: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/11.jpg)
![Page 12: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/12.jpg)
![Page 13: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/13.jpg)
Caveats
Motion analysis a very important and popular area in
computer vision
A large body of literature exits with maybe hundreds of
different formulations (At CVPR, you will find at least 2 or 3 sessions on
motion)
Many of them can be very mathematical
Apparent motion != True motion
![Page 14: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/14.jpg)
Rigid vs. nonrigid motion
Camera motion is 6 DOF rigid motion
Object motion may be rigid or nonrigid
Rigid: coffee mugs, silverware, baseballs, jets, ...
Nonrigid: humans, face, medical imagery, beach
balls, scissors, grass, ...
Includes articulated motion
![Page 15: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/15.jpg)
Nonrigid motion
Nonrigid motion is complicated and difficult, especially
with little prior knowledge on what is being viewed
Typical problem: What are the parameters of the known nonrigid
model of the object being viewed?
We’ll just focus on rigid motion
![Page 16: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/16.jpg)
The barber’s pole illusion
Motion
field
Optical
flow
Web example
![Page 17: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/17.jpg)
The aperture problem
In local processing, we can only measure
motion perpendicular to the image
gradient
![Page 18: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/18.jpg)
First steps
Motion processing starts with estimating optical flow from frame to frame, either densely or sparsely
The typical approaches are:
Dense correspondence:
Differential methods, local area/correlation based
This could be hierarchical (coarse-to-fine approach)
Sparse correspondence
Matching methods, feature based
Asumption: Points/features can be matched in nearby images
![Page 19: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/19.jpg)
Brightness constancy equation
0)),(),((
==td
ttytxId
td
Id
0)),(),((
=∂
∂+
∂
∂+
∂
∂=
dt
dt
t
I
dt
dy
y
I
dt
dx
x
I
td
ttytxId
0,, =∂
∂+
∂
∂
∂
∂
t
I
dt
dy
dt
dx
y
I
x
IT
0=+⋅∇ tII v
I∇
v
tI
Image gradient
Optical flow
Time difference
For a given scene pointTotal
derivative
![Page 20: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/20.jpg)
Brightness constancy equation
(method #2)
),,(),,( ttyyxxItyxI δ+δ+δ+= For a given scene point
0),,(),,( =−δ+δ+δ+ tyxIttyyxxI
dtt
tyxIdy
y
tyxIdx
x
tyxIzyxI
∂
∂+
∂
∂+
∂
∂+
),,(),,(),,(),,(
0),,(),,(),,(
=∂
∂+
∂
∂+
∂
∂dt
t
tyxIdy
y
tyxIdx
x
tyxI
by Taylor expansion
≈
0=∂
∂+
∂
∂+
∂
∂
t
I
dt
dy
y
I
dt
dx
x
I0=+⋅∇ tIvI
![Page 21: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/21.jpg)
Brightness constancy equation
(method #2)
),,( ttyyxxI δ+δ+δ+),,( tyxI
Image at time t Image at time t+δt
=
![Page 22: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/22.jpg)
Back to the aperture problem
0=+⋅∇ tIvI
I∇
2v
tI
1v
tIvI −=⋅∇
Many vectors v satisfy this
Only the normal direction is constrained
3v
![Page 23: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/23.jpg)
On images…
t
x
yx
I
∂
∂
y
I
∂
∂
t
I
∂
∂
t
x
yx
I
∂
∂
y
I
∂
∂
t
I
∂
∂
dt
dy
dt
dx,
0=+⋅∇ tIvI
This equation defines and constrains the optical flow v (x, y)
![Page 24: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/24.jpg)
What is the image gradient?
Image gradient – the first derivative (slope) of the intensity variation in (x, y)
![Page 25: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/25.jpg)
What is the temporal gradient?
![Page 26: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/26.jpg)
t1 t2
t
I
∂
∂
![Page 27: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/27.jpg)
Brightness constancy of a point
Scene
I1 I2 I3
I(x(t1),y(t1),t1)
I(x(t2),y(t2),t2)
I(x(t2),y(t2),t2)
I(x(t3),y(t3),t3)
= =
Image
sequence
![Page 28: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/28.jpg)
Difficulty
One equation with two unknowns
Aperture problem
spatial derivatives use only a few adjacent pixels (limited aperture
and visibility)
many combinations of (u,v) will satisfy the equation
Constraint line
u
v
![Page 29: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/29.jpg)
intensity gradient is zero
no constraints on (u,v)
interpolated from other places
intensity gradient is nonzero
but is constant
one constraints on (u,v)
only the component along the gradient
are recoverable
intensity gradient is nonzero
and changing
multiple constraints on (u,v)
motion recoverable
( , ) ( , )0 0 0⋅ =u v
( , ) ( , )∂
∂
∂
∂
∂
∂
I
x
I
yu v
I
t⋅ = −
( , ) ( , )
( , ) ( , )
( , )
( , )
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
I
x
I
yu v
I
t
I
x
I
yu v
I
t
x y
x y
1 1
2 2
1 1
2 2
⋅ = −
⋅ = −
![Page 30: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/30.jpg)
( )∑Ω∈
++=yx
tyx IvyxIuyxIvuE,
2),(),(),(
Minimizing
Assume a single velocity for all pixels within an image patch
−=
∑∑
∑∑∑∑
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
( ) t
TIIUII ∑∑ ∇−=∇∇
r
LHS: sum of the 2x2 outer product of the gradient vector
Patch Translation [Lucas-Kanade]
Szeliski
![Page 31: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/31.jpg)
Image motion
How do we determine correspondences?
Assume all change between frames is due to motion:
),(),( ),(),( yxyx vyuxIyxJ ++≈
I J
![Page 32: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/32.jpg)
The Aperture Problem
( )( )∑ ∇∇=T
IIMLet
• Algorithm: At each pixel compute by solving
• M is singular if all gradient vectors point in the same direction
• e.g., along an edge
• of course, trivially singular if the summation is over a single pixelor there is no texture
• i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
and
−
−=
∑∑
ty
tx
II
IIb
U bMU=
Szeliski
![Page 33: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/33.jpg)
SSD Surface – Textured area
![Page 34: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/34.jpg)
SSD Surface -- Edge
![Page 35: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/35.jpg)
SSD – homogeneous area
![Page 36: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/36.jpg)
Limits of the gradient method
Fails when intensity structure in window is poor
Fails when the displacement is large (typical operating range is
motion of 1 pixel)
Linearization of brightness is suitable only for small displacements
Also, brightness is not strictly constant in images
actually less problematic than it appears, since we can pre-filter images
to make them look similar
Szeliski
![Page 37: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/37.jpg)
image Iimage J
av
Jwwarp refine
av
aΔv
+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Szeliski
![Page 38: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/38.jpg)
Iterative Refinement
Estimate velocity at each pixel using one iteration of Lucas
and Kanade estimation
Warp one image toward the other using the estimated flow
field
(easier said than done)
Refine estimate by repeating the process
Szeliski
![Page 39: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/39.jpg)
Optical Flow: Iterative Estimation
xx0
Initial guess:
Estimate:
estimate
update
(using d for displacement here instead of u)
Szeliski
![Page 40: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/40.jpg)
Optical Flow: Iterative Estimation
xx0
estimate
updateInitial guess:
Estimate:
Szeliski
![Page 41: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/41.jpg)
Optical Flow: Iterative Estimation
xx0
Initial guess:
Estimate:
Initial guess:
Estimate:
estimate
update
Szeliski
![Page 42: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/42.jpg)
Temporal coherency
t t t+ δ t t+ 2δ
( , )u v( , )u v
( , ) ( , )∂
∂
∂
∂
∂
∂
I
x
I
yu v
I
t⋅ = − ( , ) ( , )
∂
∂
∂
∂
∂
∂
I
x
I
yu v
I
t⋅ = −
• Caveat:– (u,v) must stay the same across several frames
– scenes highly textured
– (u,v) at the same location actually refers to different object points
![Page 43: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/43.jpg)
Spatial coherency
neighboring pixels should have “similar” flow vector
Q: What do you mean by “similar”
A1: identical
A2: change slowly ∂
∂
∂
∂
∂
∂
∂
∂
u
x
u
y
v
x
v
y= = = = 0
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
u
x
u
y
v
x
v
y
u
x
u
y
v
x
v
y
, , ,
( ) ( ) ( ) ( )
≅
+ + +
0
2 2 2 2 sm all
![Page 44: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/44.jpg)
Mathematical formulation
Based on Lagrange Multiplier
Incorporate smoothness as an additional constraint
Can be thought of as a weighting of two terms:
optical flow constraint
smoothness constraint
( , ) ( , )∂
∂
∂
∂
∂
∂
I
x
I
yu v
I
t⋅ = −
( ) ( ) ( ) ( )∂
∂
∂
∂
∂
∂
∂
∂
u
x
u
y
v
x
v
y
2 2 2 2+ + +
![Page 45: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/45.jpg)
EI
xu
I
yv
I
t
u
x
u
y
v
x
v
ydxdy= + + + + + +∫∫ ( ) [( ) ( ) ( ) ( ) ]
∂
∂
∂
∂
∂
∂λ
∂
∂
∂
∂
∂
∂
∂
∂
2 2 2 2 2
Optimize over all image plane:
Discretize the governing equation, at (i,j):
∂
∂
∂
∂∂
∂
∂
∂
u
xu u
u
yu u
v
xv v
v
yv v
i j i j i j i j
i j i j i j i j
= − = −
= − = −
+ +
+ +
1 1
1 1
, , , ,
, , , ,
Discretized expression:
EI
xu
I
yv
I
t
u u u u v v v v
i j
i j
i j
i j
i jji
i j i j i j i j i j i j i j i j
= + +
+ − + − + − + −
∑∑
+ + + +
( )
[( ) ( ) ( ) ( ) ]
,
,
,
,
,
, , , , , , , ,
∂
∂
∂
∂
∂
∂
λ
2
1
2
1
2
1
2
1
2
![Page 46: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/46.jpg)
• At a pixel location (k,l):
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
λ
E
u
I
xu
I
yv
I
t
I
x
u u u u u u u u
k l k l
k l
k l
k l
k l k l
k l k l k l k l k l k l k l k l
, ,
,
,
,
, ,
, , , , , , , ,
( )
[( ) ( ) ( ) ( )]
= + +
− − + − + − + − =− − + +
2
2 01 1 1 1
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
λ
E
v
I
xu
I
yv
I
t
I
y
v v v v v v v v
k l k l
k l
k l
k l
k l k l
k l k l k l k l k l k l k l k l
, ,
,
,
,
, ,
, , , , , , , ,
( )
[( ) ( ) ( ) ( )]
= + +
− − + − + − + − =− − + +
2
2 01 1 1 1
∂
∂ λ
E=L
![Page 47: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/47.jpg)
• Putting it all together:
4/)(
4/)(
0)(4)(
0)(4)(
1,,11,,1
1,,11,,1
,
,,
,
2
,
,
,,
,
,,
,
,,
,
2
,
++−−
++−−
+++=
+++=
=−−++
=−−++
lklklklk
lklklklk
lk
lklk
lk
lk
lk
lklk
lk
lklk
lk
lklk
lk
lk
vvvvv
uuuuu
vvt
I
y
Iv
y
Iu
y
I
x
I
uut
I
x
Iv
y
I
x
Iu
x
I
λ∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
![Page 48: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/48.jpg)
• Or:
[ ( ) ]
[ ( ) ]
,
,
, ,
,
, ,
, ,
,
,
,
, ,
4 4
4 4
2
2
λ∂
∂
∂
∂
∂
∂λ
∂
∂
∂
∂
∂
∂
∂
∂λ
∂
∂λ
∂
∂
∂
∂
+ + = −
+ + = −
I
xu
I
x
I
yv u
I
x
I
t
I
x
I
yu
I
yv v
I
y
I
t
k l
k l
k l k l
k l
k l k l
k l k l
k l
k l
k l
k l k l
u u
I
xu
I
yv
I
t
I
x
I
y
I
x
v v
I
xu
I
yv
I
t
I
x
I
y
I
y
k l
k l k l k l
k l k l
k l
k l
k l k l k l
k l k l
k l
,
, , ,
, ,
,
,
, , ,
, ,
,
( ) ( )
( ) ( )
= −
+ +
+ +
= −
+ +
+ +
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
4
4
2 2
2 2
![Page 49: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/49.jpg)
u u
I
xu
I
yv
I
t
I
x
I
y
I
x
v v
I
xu
I
yv
I
t
I
x
I
y
I
y
k l
k l k l k l
k l k l
k l
k l
k l k l k l
k l k l
k l
,
, , ,
, ,
,
,
, , ,
, ,
,
( ) ( )
( ) ( )
= −
+ +
+ +
= −
+ +
+ +
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
4
4
2 2
2 2
•estimate based on smoothness
•how much does the smooth estimate violate
optical flow constraint
•how much does the optical flow constraint matters
•direction for correction
![Page 50: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/50.jpg)
Algorithms
1 . C o m p u te fro m a p a ir o f in p u t im ag es
2 . C h o o se a w eigh tin g fac to r
3 . C o m p u te (
A t each p ix e l lo ca tio n ( ), d o
∂
∂
∂
∂
∂
∂
λ
∂
∂
∂
∂
∂
∂
λ∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
I
x
I
y
I
t
u v
k ,l
u u
I
xu
I
yv
I
t
I
x
I
y
I
x
v v
I
xu
I
y
k l
n n k l
n
k l
n
k l
k l k l
k l
k l
n n k l
n
, ,
, )
.
( ) ( ),
( ) ( ) ,
( )
,
( )
,
, ,
,
,
( ) ( ) ,
( )
4
4
1
2 2
1
+
+
= −
+ +
+ +
= −
+k l
n
k l
k l k l
k l
vI
t
I
x
I
y
I
y
,
( )
,
, ,
,( ) ( )
.
+
+ +
∂
∂
λ∂
∂
∂
∂
∂
∂4
5
2 2
Ite ra te s tep s 3 an d 4 u n til n o ch an g e o r co u n t ex ceed s
![Page 51: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/51.jpg)
Results
![Page 52: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/52.jpg)
Motion representations
How can we describe this scene?
Szeliski
![Page 53: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/53.jpg)
Block-based motion prediction
Break image up into square blocks
Estimate translation for each block
Use this to predict next frame, code difference (MPEG-2)
Szeliski
![Page 54: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/54.jpg)
Layered motion
Break image sequence up into “layers”:
÷ =
Describe each layer’s motion
Szeliski
![Page 55: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/55.jpg)
Layered motion
Advantages:
• can represent occlusions / disocclusions
• each layer’s motion can be smooth
• video segmentation for semantic processing
Difficulties:
• how do we determine the correct number?
• how do we assign pixels?
• how do we model the motion?
Szeliski
![Page 56: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/56.jpg)
Layers for video summarization
Szeliski
![Page 57: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/57.jpg)
Background modeling (MPEG-4)
Convert masked images into a background sprite for
layered video coding
+ + +
=
Szeliski
![Page 58: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/58.jpg)
![Page 59: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/59.jpg)
![Page 60: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/60.jpg)
![Page 61: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/61.jpg)
![Page 62: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/62.jpg)
Optical flow summary
Optical flow techniques:
Techniques that estimate the motion field from the image
brightness constancy equation
Optical flow:
Is best estimated (least noisy) at image points with high spatial
image gradients. (Why?)
Works best for Lambertian surfaces (Why?)
Works best for very high frame rates (Why?)
From optical flow, we can compute shape/structure/depth,
motion parameters, segmentation, etc.
But if you primarily want to track an object, other methods may be
preferred
![Page 63: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/63.jpg)
Tracking
Tracking is the process of updating an object’s position
(and orientation, and articulation?) over time through a
video sequence
Estimate the object pose at each time point
“Pose” – position and orientation
Applications
Surveillance
Targeting
Motion-based recognition (e.g., gesture recognition, computation
of egomotion)
Motion analysis (golf swing, gait, character animation)
……..
![Page 64: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/64.jpg)
Tracking vs. optical flow
In tracking, we are generally acknowledging that some sparse features are the points to track
Corners, lines, regions, patterns, contours....
Rather than computing the full motion field from optical flow, let’s keep track of the time-varying position of these sparse features
Then compute egomotion, object pose, etc. from this
This typically involves a loop of prediction, measurement, and correction
Often with presumed models of motion dynamics and measurement noise
![Page 65: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/65.jpg)
Tracking vs. Matching
Tracking requires
videos
Small displacement is
assumed
Simple features
Use image constraint
(similar to optical flow
constraint)
Matching can be done
on discrete frames
Displacement can be
large (>10 pixels)
Often more elaborate
features
Independent detection
in each frame and then
match
![Page 66: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/66.jpg)
Examples LKT tracker
F(x)G(x)
![Page 67: Visual Motion Analysis and Representationyfwang/courses/cs281b/notes/7-motion2.… · Visual Motion Analysis and Representation . Example Ullman’s concentric counter-rotating cylinder](https://reader033.vdocuments.us/reader033/viewer/2022043023/5f3edbb1bd6a934c8a7f3d46/html5/thumbnails/67.jpg)
KLT tracker
An iterative update algorithm
The estimate is more accurate if F is indeed linear
Penalize pixels with large 2nd derivatives (w(x))