shape from shading
DESCRIPTION
Shape From Shading. Mica Arie-Nachimson. May 2011. What is Shading?. Well… not shadow… We can’t reconstruct shape from one shadow…. Image from www.moolf.com. What is Shading?. Variable levels of darkness Gives a cue for the actual 3D shape - PowerPoint PPT PresentationTRANSCRIPT
SHAPE FROM SHADING
Mica Arie-Nachimson May 2011
What is Shading?
Well… not shadow…
We can’t reconstruct shape from one shadow…
Image from www.moolf.com
What is Shading?
Variable levels of darkness
Gives a cue for the actual 3D shape There is a relation between intensity and shape
Images from S. Narasimhan, Carnegie Mellon; www.erco.com; www.alesmav.com; H. Wang, University of California
Shading Examples
These circles differ only in grayscale intensity
Intensities give a strong “feeling” of scene structure
Talk Outline
Introduction and Basics Notations Reflectance map Photometric stereo
Main Approaches Propagation Solutions
Basic Solution by Horn [85] Using Fast Marching, Kimmel & Sethian [2001]
Global energy minimization solution using Belief Propagation – only if time permits
Problem definitionNotationsReflectance MapPhotometric Stereo
Introduction and Basics
What determines scene radiance?
The amount of light that falls on the surface The fraction of light that is reflected (albedo) Geometry of light reflection
Shape of surface Viewpoint
n (𝜃𝑖 ,𝜑𝑖)
(𝜃𝑒 ,𝜑𝑒)
𝜌
Albedo
A property of the surface Usually normalized
Can be modeled as a single scalar for the entire surface or dependent on location
Usually assumed to be a known scalar Is this a ball with uniform albedo or is this a 2D circle with
varying albedo?
We’re not going to talk about the albedo
Surface Normal
Convenient notation for surface orientation A smooth surface has a tangent plane at every
point
We can model the surface using the normal at every point
The Shape From Shading Problem
Given a grayscale image And albedo And light source direction
Reconstruct scene geometry Can be modeled by surface normals
Lambertian Surface
Appears equally bright from all viewing directions Reflects all light without absorbing Matte surface, no “shiny” spots Brightness of the surface as seen from camera is
linearly correlated to the amount of light falling on the surface
n (𝜃𝑖 ,𝜑𝑖)
(𝜃𝑒 ,𝜑𝑒)
Today we will discuss only Lambertian surfaces under point-source illumination
Some Notations:Surface Orientation
A smooth surface has a tangent plane at every point Mark
Parametrize surface orientation by first partial derivatives of
𝑝𝛿𝑥𝑞𝛿 𝑦
Some Notations:Surface Orientation
Surface normal ,
Normalize
𝑝𝛿𝑥𝑞𝛿 𝑦
𝑝=𝜕 𝑧𝜕 𝑥 ;𝑞=
𝜕 𝑧𝜕 𝑦
𝑟 𝑥
𝑟 𝑦𝑧
Reflectance Map
Relationship between surface orientation and brightness
Lambertian surface
Image irradiance (brightness) is proportional to
Reflectance Map Example
Brightness as a function of surface orientation
0.1
3.0
0.0
9.08.0
7.0, qpR
p
q
90i 01 SS qqpp
SS qp ,
iso-brightnesscontour
Illustration from S. Narasimhan, Carnegie Mellon
Lambertian surface
Reflectance Map of a Glossy Surface
Brightness as a function of surface orientation
Surface with diffuse and glossy components
Reflectance Map Examples
Brightness as a function of surface orientation
Graphics with a 3D Feel
Given a 3D surface , lighting and viewing direction, we can compute the gray level of pixel of the surface
Find the gradient of the surface Use to determine the gray level
Shape From Shading?
Can we reconstruct shape from a single image?
Two variables with one equation… So what do we do?
p
q
Illustration from S. Narasimhan, Carnegie Mellon
Shape From Shading!
Use more images Photometric stereo
Shape from shading Introduce constraints Solve locally Linearize problem
Images fromR. Basri et al. “Photometric Stereo with General, Unknown Lighting“, IJCV 2006
Photometric Stereo
Take several pictures of same object under same viewpoint with different lighting
p
q
11 ,SSqp
Photometric Stereo
Take several pictures of same object under same viewpoint with different lighting
p
q
11 ,SSqp
22 ,SSqp
Photometric Stereo
Take several pictures of same object under same viewpoint with different lighting
p
q
11 ,SSqp
22 ,SSqp
33 ,SSqp
A Quick Review
Main Approaches forShape From Shading
Main Approaches
Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …
Main Approaches
Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …
Propagation: Grow a solution around an initial known point or boundary
Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance
function
Main Approaches
Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …
Propagation: Grow a solution around an initial known point or boundary
Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance
function
Main Approaches
Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …
Propagation: Grow a solution around an initial known point or boundary
Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance
function
Horn [85]Solution by Characteristic Curves
Basic Propagation Solution
Propagating Solution
Suppose that we know the depth of some point on the surface
Extend the solution by taking a small step But and are unknown, and the image irradiance
equation gives only one constraint If we knew and , can compute changes in using
second partial derivatives and
𝐻=(𝜕2𝑧𝜕 𝑥2
𝜕2𝑧𝜕 𝑥𝜕 𝑦
𝜕2𝑧𝜕 𝑥𝜕 𝑦
𝜕2𝑧𝜕 𝑦2
)
Propagating Solution
, what else can we use? Image irradiance equation
Propagating Solution
; We note that for a small step
For these specific
If you take a small step in the image plane parallel to the direction of the gradient of , then can compute change in
Propagating Solution
; Get ODEs:
; ; ; ;
These equations describe a characteristic curve
(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)
(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)
Propagating Characteristic Curve
Characteristic curve
Need to initialize every curve at some known point Singular points Occluding boundaries
(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)
(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)
Image by Dejan Todorović
Propagating Characteristic Curve
Characteristic curve
Need to initialize every curve at some known point Singular points Occluding boundaries
Curves are “grown” independently, very instableImage by Dejan Todorović
(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)
(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)
“Optimal Algorithm for Shape from Shading and Path Planning”, R. Kimmel and J. A. Sethian [2001]
Shape From Shading by Fast Marching
Vertical Light Source Case
Recall reflectance map
Assume light source located near the viewer =(0,0,1)
This is an Eikonal equation Can be solved by an numerical algorithm
Fast Marching
Fast Marching
Expanding Dijkstra’s shortest path algorithm for general surfaces Represented as triangulated mesh Flat domains
Many computer vision problems can be set into flat domains Every pixel is a node, edges between adjacent nodes
Dijkstra’s Shortest Paths
Connected graphs Start with the source node
s 7
1510914
9
211
6
Dijkstra’s Shortest Paths
Connected graphs Start with the source node Update all neighbors
s7
9
14
7
1510914
9
211
6
Dijkstra’s Shortest Paths
Connected graphs Start with the source node Update all neighbors Go to the closest neighbor
Set its values
7
9
14
7
1510914
9
211
6
Dijkstra’s Shortest Paths
Connected graphs Start with the source node Update all neighbors Go to the closest neighbor
Set its value Compute its neighbors Update smaller scores
s7
9
1423
7
1510914
9
211
6
Dijkstra’s Shortest Paths
Connected graphs Start with the source node Update all neighbors Go to the closest neighbor
Set its value Compute its neighbors Update smaller scores
Continue with smallest value node Remember path
s7
9
1423
7
1510914
9
211
6
Flat Domains: Why Does Dijkstra Fail?
Dijkstra will not find the diagonal path Need to examine triangles
Fast Marching: Problem Definition
Suppose there is a forest fire with multiple sources Every point that was touched by the fire is burnt
and will not be visited by the fire again Firemen register the time T(x) at which the fire
arrives to location x. The fast marching algorithm simulates this
scenario
Fast Marching: Problem Definition
Multiple sources Advancing forward Advances either at a constant rate or at varying
rate What is the arrival time at every location?
Image from G. Rosman, Technion
Fast Marching Algorithm
Set and mark the points as done
Set the rest of the points as and mark them as far
All far points adjacent to done points become verify points
Update all verify points using of the done set
The verify with the smallest becomes done
Continue until all points are done
Illustrations from selected publications of J. A. Sethian
Fast Marching on a Grid
Update step
T
i,j-1
i,j+1i-1,jij
i+1,j
i+1,j
i,j+1T
TT
T
i-1,j
i,j-1
ij
?
Slide based on slides by R. Kimmel, Technion
Fast Marching on a Grid
Update step
T
i,j-1
i,j+1i-1,jij
i+1,j
i+1,j
i,j+1T
TT
T
i-1,j
i,j-1
ij
?
Slide based on slides by R. Kimmel, Technion
Fast Marching on a Grid
Update step
T
i,j-1
i,j+1i-1,jij
i+1,j
i+1,j
i,j+1T
TT
T
i-1,j
i,j-1
ij
?𝑇 1
𝑇 2
Slide based on slides by R. Kimmel, Technion
Fast Marching on a Grid
Initialization: all or known initial value Update step:
Fit a triangle with gradient and two determined values at neighboring grid points T
i,j-1
i,j+1i-1,jij
i+1,j
i+1,j
i,j+1T
TT
T
i-1,j
i,j-1
ij
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Slide based on slides by R. Kimmel, Technion
Update Step Details
Update step is a quadratic equation:
Solution:
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Disregard the “minus” solutionwhich yields
Update Step Details
Update step is a quadratic equation:
Solution:
If , then and
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Update Step Details
Update step is a quadratic equation:
Solution:
If , then and
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Update Step Details
Update step is a quadratic equation:
Solution:
If , then and
and
This means that the wave front propagation comes from outside the triangle Only choice is to update
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Update Step Details
Update step is a quadratic equation:
If
Else
(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2
Shape From shadingVertical Light Source Case
Eikonal equation
Numerical approximation of this equation
Assume w.l.o.g. that
Shape From ShadingVertical Light Source Case
Eikonal equation
Numerical approximation of this equation
Assume w.l.o.g. that
Vertical Light Source CaseDerive Update Step
𝑧1
𝑧 2
Vertical Light Source CaseDerive Update Step
Where:
If
Otherwise
Vertical Light Source CaseFast Marching
Fast marching introduces order to these update steps
Points are updated from small to large Total complexity:
for the selection of smallest point and update of neighboring points
points (pixles)
General Light Source
Recall reflectance map
In this case The right hand side depends on
This is not an Eikonal equation anymore How can we solve this?
Solve in the light source coordinates, then change the variables back
General Light SourceChange Coordinate System
Before moving to light source coordinate system, choose w.l.o.g light source direction to be, where This will simplify our equations a bit
General Light SourceChange Coordinate System
Change coordinate system
Light source coordinates marked with ~
Eikonal equation!
𝑦
�⃗�=(𝒍𝟏𝟎𝒍𝟑)𝑙1
𝑙3
𝒙=( 𝒍𝟑𝟎− 𝒍𝟏)~𝑥
~𝑧�⃗� 𝒛
𝑥
𝑧
𝑦
General Light SourceSolution in New Coordinate System
But… We need to find in order to find the intensityUse the smallest value from all neighbors of
General Light Source SolutionUpdate Step
Where:
If
Otherwise
Results
Results shown on synthetic shading image
“Efficient Belief Propagation for Vision Using Linear Constraint Nodes”, B. Potetz [2007]
Minimization Using EfficientBelief Propagation
Belief Propagation
Estimate the marginals of a multivariate probability distribution
, potential function consists of several elements
Usually represented as a factor graph
Factor Graph
, Bipartite graph Every element of is represented by a variable
node Every potential function represented by a factor
node , connected to all variable nodes of circles: variable nodessquares: factor nodes
Belief Propagation on a Factor Graph
, Estimate the marginals
Iteratively compute messages along edges of the factor graph Every variable node computes its marginal probability
according to the probabilities its neighbors sent Usually converges to a good enough local
minimum
Shape From Shading Using aFactor Graph
Recall Constraint I: Consistent integral
Where Integral of the surface gradient along a closed curve should be zero
Illustration from Klaus & Horn, “Robot Vision”, MIT Press, 1986
Shape From Shading Using aFactor Graph
Constraint II: Lambertian constraint
Where could be changed for other lighting conditions
Constraint III: Shape prior Modeled by a Laplace distribution,
Shape From Shading Using aFactor Graph
Constraint I: Consistent integral
Constraint II: Lambertian constraint
Constraint III: Shape prior,
one pixel one pixel
one pixel one pixel one pixel
Variable nodesFactor node, consistent integralFactor node, Lambertian constraintFactor node, shape prior
Shape From Shading Using aFactor Graph
Constraint I: Consistent integral
Constraint II: Lambertian constraint
Constraint III: Shape prior,
Variable nodesFactor node, consistent integralFactor node, Lambertian constraintFactor node, shape prior
Results
Input:
Result:
original surface
Rendering of original surface with
Input image
recovered surfaceRendering of recovered surface with
Conclusion
Shape from shading problem Definition Main difficulties Main approaches
Propagating solutions Horn [85] Kimmel [2001]
Minimization with BP while introducing constraints
Thank You!