dr. j. shanbehzadeh shanbehzadeh@gmail sara omid
DESCRIPTION
Machine Vision. Lecture 06 Computing Optical Flow. DR. J. Shanbehzadeh [email protected] Sara Omid. Science and Research Branch of Azad University. Table of Contents. Optical Flow Examples. Optical Flow Examples. Optical Flow Examples Optical Flow Horn & Schunck Optical Flow - PowerPoint PPT PresentationTRANSCRIPT
DR. J. [email protected]
Sara Omid
Science and Research Branch of Azad University
Lecture 06Computing Optical Flow
Machine Vision
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Table of Contents2
Optical Flow ExamplesOptical FlowHorn & Schunck Optical Flow
• Interpretation Of Optical Flow eq• Horn & Schunck• Derivative Masks• Laplacian• Algorithm-1• Synthetic Images• Horn & Schunck Results
Lucas & KanadeComments
3
Optical Flow Examples
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples4
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples5
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples6
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples7
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples8
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples9
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples10
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples11
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples12
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples13
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
14
Optical Flow
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow15
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Applications
• Motion based segmentation
• Structure from Motion(3D shape and Motion)
• Alignment (Global motion compensation) Camcorder video stabilization UAV Video Analysis
• Video Compression
16
Horn & Schunck
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck Optical Flow17
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Brightness constancy assumption𝑓 (𝑥 , 𝑦 , 𝑡 )= 𝑓 (𝑥+𝑑𝑥 , 𝑦+𝑑𝑦 , 𝑡+𝑑𝑡 )
𝑓 (𝑥 , 𝑦 , 𝑡 )= 𝑓 (𝑥 , 𝑦 ,𝑡 )+𝜕 𝑓𝜕𝑥 𝑑𝑥+𝜕 𝑓𝜕 𝑦 𝑑𝑦+
𝜕 𝑓𝜕𝑡 𝑑𝑡
𝑓 𝑥𝑑𝑥+ 𝑓 𝑦𝑑𝑦+ 𝑓 𝑡 𝑑𝑡=0
𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡=0
Taylor Series
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Interpretation Of Optical Flow eq18
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡=0
d = normal flowP = parallel flow𝑑=
𝑓 𝑡√ 𝑓 𝑥
2+ 𝑓 𝑦2
𝑣=−𝑓 𝑥
𝑓 𝑦𝑢−
𝑓 𝑡𝑓 𝑦
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck19
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
∬ { ( 𝑓 𝑥𝑢+ 𝑓 𝑦𝑣+ 𝑓 𝑡 )2+λ (𝑢𝑥2+𝑢𝑦
2 +𝑣𝑥2+𝑣𝑦
2 ) }𝑑𝑥𝑑𝑦Brightness Constancy
Smoothness Constraint
min
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck20
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Laplacian
=
=
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Derivative Masks21
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
fx fy
ft
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Derivative Masks (Roberts)22
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Apply first mask to 1st image
Second mask to 2nd image
Add the responses to get fx, fy, ft
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian23
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian24
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Discrete version
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian25
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
variational calculus
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Algorithm-126
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
• k=0
• Initialize uk and vk
• Repeat until some error measure is satisfied(converges)
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Synthetic Images27
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck Results28
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
29
Lucas & Kanade
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade30
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
• Optical Flow eq
A u = ft
𝑓 𝑥𝑢+ 𝑓 𝑦𝑣=− 𝑓 𝑡
• Consider 3 by 3 window
. . .
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade31
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
A U = ft
AT A U = AT ft
U = (ATA)-1 AT ft
Pseudo Inverse
Least Squares Fit
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade32
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade33
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
[ ∑ 𝑓 𝑥𝑖2 ∑ 𝑓 𝑥𝑖 𝑓 𝑦𝑖
∑ 𝑓 𝑥𝑖 𝑓 𝑦𝑖 ∑ 𝑓 𝑦𝑖2 ][𝑢𝑣 ]=[−∑ 𝑓 𝑥𝑖 𝑓 𝑡𝑖
−∑ 𝑓 𝑦𝑖 𝑓 𝑡𝑖]
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade34
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade35
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade36
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade37
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
38
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Comments39
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
• Horn‐Schunck and Lucas‐Kanade optical methods work only for small motion.
• If object moves faster, the brightness changes rapidly,
• 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.
• Pyramids can be used to compute large optical flow vectors.