shape from shading

SHAPE FROM SHADING

Mica Arie-Nachimson May 2011

What is Shading?

Well… not shadow…

We can’t reconstruct shape from one shadow…

Image from www.moolf.com

What is Shading?

Variable levels of darkness

Gives a cue for the actual 3D shape There is a relation between intensity and shape

Images from S. Narasimhan, Carnegie Mellon; www.erco.com; www.alesmav.com; H. Wang, University of California

Shading Examples

These circles differ only in grayscale intensity

Intensities give a strong “feeling” of scene structure

Talk Outline

Introduction and Basics Notations Reflectance map Photometric stereo

Main Approaches Propagation Solutions

Basic Solution by Horn [85] Using Fast Marching, Kimmel & Sethian [2001]

Global energy minimization solution using Belief Propagation – only if time permits

Problem definitionNotationsReflectance MapPhotometric Stereo

Introduction and Basics

What determines scene radiance?

The amount of light that falls on the surface The fraction of light that is reflected (albedo) Geometry of light reflection

Shape of surface Viewpoint

n (𝜃𝑖 ,𝜑𝑖)

(𝜃𝑒 ,𝜑𝑒)

𝜌

Albedo

A property of the surface Usually normalized

Can be modeled as a single scalar for the entire surface or dependent on location

Usually assumed to be a known scalar Is this a ball with uniform albedo or is this a 2D circle with

varying albedo?

We’re not going to talk about the albedo

Surface Normal

Convenient notation for surface orientation A smooth surface has a tangent plane at every

point

We can model the surface using the normal at every point

The Shape From Shading Problem

Given a grayscale image And albedo And light source direction

Reconstruct scene geometry Can be modeled by surface normals

Lambertian Surface

Appears equally bright from all viewing directions Reflects all light without absorbing Matte surface, no “shiny” spots Brightness of the surface as seen from camera is

linearly correlated to the amount of light falling on the surface

n (𝜃𝑖 ,𝜑𝑖)

(𝜃𝑒 ,𝜑𝑒)

Today we will discuss only Lambertian surfaces under point-source illumination

Some Notations:Surface Orientation

A smooth surface has a tangent plane at every point Mark

Parametrize surface orientation by first partial derivatives of

𝑝𝛿𝑥𝑞𝛿 𝑦

Some Notations:Surface Orientation

Surface normal ,

Normalize

𝑝𝛿𝑥𝑞𝛿 𝑦

𝑝=𝜕 𝑧𝜕 𝑥 ;𝑞=

𝜕 𝑧𝜕 𝑦

𝑟 𝑥

𝑟 𝑦𝑧

Reflectance Map

Relationship between surface orientation and brightness

Lambertian surface

Image irradiance (brightness) is proportional to

Reflectance Map Example

Brightness as a function of surface orientation

0.1

3.0

0.0

9.08.0

7.0, qpR

p

q

90i 01 SS qqpp

SS qp ,

iso-brightnesscontour

Illustration from S. Narasimhan, Carnegie Mellon

Lambertian surface

Reflectance Map of a Glossy Surface

Brightness as a function of surface orientation

Surface with diffuse and glossy components

Reflectance Map Examples

Brightness as a function of surface orientation

Graphics with a 3D Feel

Given a 3D surface , lighting and viewing direction, we can compute the gray level of pixel of the surface

Find the gradient of the surface Use to determine the gray level

Shape From Shading?

Can we reconstruct shape from a single image?

Two variables with one equation… So what do we do?

p

q

Illustration from S. Narasimhan, Carnegie Mellon

Shape From Shading!

Use more images Photometric stereo

Shape from shading Introduce constraints Solve locally Linearize problem

Images fromR. Basri et al. “Photometric Stereo with General, Unknown Lighting“, IJCV 2006

Photometric Stereo

Take several pictures of same object under same viewpoint with different lighting

p

q

11 ,SSqp

Photometric Stereo

Take several pictures of same object under same viewpoint with different lighting

p

q

11 ,SSqp

22 ,SSqp

Photometric Stereo

Take several pictures of same object under same viewpoint with different lighting

p

q

11 ,SSqp

22 ,SSqp

33 ,SSqp

A Quick Review

Main Approaches forShape From Shading

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance

function

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance

function

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s): Brightness Smoothness Intensity gradient …

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type Linear: Make problem linear by a linearization of reflectance

function

Horn [85]Solution by Characteristic Curves

Basic Propagation Solution

Propagating Solution

Suppose that we know the depth of some point on the surface

Extend the solution by taking a small step But and are unknown, and the image irradiance

equation gives only one constraint If we knew and , can compute changes in using

second partial derivatives and

𝐻=(𝜕2𝑧𝜕 𝑥2

𝜕2𝑧𝜕 𝑥𝜕 𝑦

𝜕2𝑧𝜕 𝑥𝜕 𝑦

𝜕2𝑧𝜕 𝑦2

)

Propagating Solution

, what else can we use? Image irradiance equation

Propagating Solution

; We note that for a small step

For these specific

If you take a small step in the image plane parallel to the direction of the gradient of , then can compute change in

Propagating Solution

; Get ODEs:

; ; ; ;

These equations describe a characteristic curve

(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)

(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)

Propagating Characteristic Curve

Characteristic curve

Need to initialize every curve at some known point Singular points Occluding boundaries

(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)

(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)

Image by Dejan Todorović

Propagating Characteristic Curve

Characteristic curve

Need to initialize every curve at some known point Singular points Occluding boundaries

Curves are “grown” independently, very instableImage by Dejan Todorović

(𝑥 , 𝑦 , 𝑧 ,𝑝 ,𝑞)

(𝑥+𝛿 𝑥 , 𝑦+𝛿 𝑦 , 𝑧+𝛿𝑧 ,𝑝+𝛿𝑝 ,𝑞+𝛿𝑞)

“Optimal Algorithm for Shape from Shading and Path Planning”, R. Kimmel and J. A. Sethian [2001]

Shape From Shading by Fast Marching

Vertical Light Source Case

Recall reflectance map

Assume light source located near the viewer =(0,0,1)

This is an Eikonal equation Can be solved by an numerical algorithm

Fast Marching

Fast Marching

Expanding Dijkstra’s shortest path algorithm for general surfaces Represented as triangulated mesh Flat domains

Many computer vision problems can be set into flat domains Every pixel is a node, edges between adjacent nodes

Dijkstra’s Shortest Paths

Connected graphs Start with the source node

s 7

1510914

9

211

6

Dijkstra’s Shortest Paths

Connected graphs Start with the source node Update all neighbors

s7

9

14

7

1510914

9

211

6

Dijkstra’s Shortest Paths

Connected graphs Start with the source node Update all neighbors Go to the closest neighbor

Set its values

7

9

14

7

1510914

9

211

6

Dijkstra’s Shortest Paths

Connected graphs Start with the source node Update all neighbors Go to the closest neighbor

Set its value Compute its neighbors Update smaller scores

s7

9

1423

7

1510914

9

211

6

Dijkstra’s Shortest Paths

Connected graphs Start with the source node Update all neighbors Go to the closest neighbor

Set its value Compute its neighbors Update smaller scores

Continue with smallest value node Remember path

s7

9

1423

7

1510914

9

211

6

Flat Domains: Why Does Dijkstra Fail?

Dijkstra will not find the diagonal path Need to examine triangles

Fast Marching: Problem Definition

Suppose there is a forest fire with multiple sources Every point that was touched by the fire is burnt

and will not be visited by the fire again Firemen register the time T(x) at which the fire

arrives to location x. The fast marching algorithm simulates this

scenario

Fast Marching: Problem Definition

Multiple sources Advancing forward Advances either at a constant rate or at varying

rate What is the arrival time at every location?

Image from G. Rosman, Technion

Fast Marching Algorithm

Set and mark the points as done

Set the rest of the points as and mark them as far

All far points adjacent to done points become verify points

Update all verify points using of the done set

The verify with the smallest becomes done

Continue until all points are done

Illustrations from selected publications of J. A. Sethian

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1i-1,jij

i+1,j

i+1,j

i,j+1T

TT

T

i-1,j

i,j-1

ij

?

Slide based on slides by R. Kimmel, Technion

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1i-1,jij

i+1,j

i+1,j

i,j+1T

TT

T

i-1,j

i,j-1

ij

?

Slide based on slides by R. Kimmel, Technion

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1i-1,jij

i+1,j

i+1,j

i,j+1T

TT

T

i-1,j

i,j-1

ij

?𝑇 1

𝑇 2

Slide based on slides by R. Kimmel, Technion

Fast Marching on a Grid

Initialization: all or known initial value Update step:

Fit a triangle with gradient and two determined values at neighboring grid points T

i,j-1

i,j+1i-1,jij

i+1,j

i+1,j

i,j+1T

TT

T

i-1,j

i,j-1

ij

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Slide based on slides by R. Kimmel, Technion

Update Step Details

Update step is a quadratic equation:

Solution:

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Disregard the “minus” solutionwhich yields

Update Step Details

Update step is a quadratic equation:

Solution:

If , then and

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Update Step Details

Update step is a quadratic equation:

Solution:

If , then and

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Update Step Details

Update step is a quadratic equation:

Solution:

If , then and

and

This means that the wave front propagation comes from outside the triangle Only choice is to update

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Update Step Details

Update step is a quadratic equation:

If

Else

(𝑇 𝑖𝑗−𝑇 1 )2+(𝑇 𝑖𝑗−𝑇2 )2=h2 𝑓 𝑖𝑗2

Shape From shadingVertical Light Source Case

Eikonal equation

Numerical approximation of this equation

Assume w.l.o.g. that

Shape From ShadingVertical Light Source Case

Eikonal equation

Numerical approximation of this equation

Assume w.l.o.g. that

Vertical Light Source CaseDerive Update Step

𝑧1

𝑧 2

Vertical Light Source CaseDerive Update Step

Where:

If

Otherwise

Vertical Light Source CaseFast Marching

Fast marching introduces order to these update steps

Points are updated from small to large Total complexity:

for the selection of smallest point and update of neighboring points

points (pixles)

General Light Source

Recall reflectance map

In this case The right hand side depends on

This is not an Eikonal equation anymore How can we solve this?

Solve in the light source coordinates, then change the variables back

General Light SourceChange Coordinate System

Before moving to light source coordinate system, choose w.l.o.g light source direction to be, where This will simplify our equations a bit

General Light SourceChange Coordinate System

Change coordinate system

Light source coordinates marked with ~

Eikonal equation!

𝑦

�⃗�=(𝒍𝟏𝟎𝒍𝟑)𝑙1

𝑙3

𝒙=( 𝒍𝟑𝟎− 𝒍𝟏)~𝑥

~𝑧�⃗� 𝒛

𝑥

𝑧

𝑦

General Light SourceSolution in New Coordinate System

But… We need to find in order to find the intensityUse the smallest value from all neighbors of

General Light Source SolutionUpdate Step

Where:

If

Otherwise

Results

Results shown on synthetic shading image

“Efficient Belief Propagation for Vision Using Linear Constraint Nodes”, B. Potetz [2007]

Minimization Using EfficientBelief Propagation

Belief Propagation

Estimate the marginals of a multivariate probability distribution

, potential function consists of several elements

Usually represented as a factor graph

Factor Graph

, Bipartite graph Every element of is represented by a variable

node Every potential function represented by a factor

node , connected to all variable nodes of circles: variable nodessquares: factor nodes

Belief Propagation on a Factor Graph

, Estimate the marginals

Iteratively compute messages along edges of the factor graph Every variable node computes its marginal probability

according to the probabilities its neighbors sent Usually converges to a good enough local

minimum

Shape From Shading Using aFactor Graph

Recall Constraint I: Consistent integral

Where Integral of the surface gradient along a closed curve should be zero

Illustration from Klaus & Horn, “Robot Vision”, MIT Press, 1986

Shape From Shading Using aFactor Graph

Constraint II: Lambertian constraint

Where could be changed for other lighting conditions

Constraint III: Shape prior Modeled by a Laplace distribution,

Shape From Shading Using aFactor Graph

Constraint I: Consistent integral

Constraint II: Lambertian constraint

Constraint III: Shape prior,

one pixel one pixel

one pixel one pixel one pixel

Variable nodesFactor node, consistent integralFactor node, Lambertian constraintFactor node, shape prior

Shape From Shading Using aFactor Graph

Constraint I: Consistent integral

Constraint II: Lambertian constraint

Constraint III: Shape prior,

Variable nodesFactor node, consistent integralFactor node, Lambertian constraintFactor node, shape prior

Results

Input:

Result:

original surface

Rendering of original surface with

Input image

recovered surfaceRendering of recovered surface with

Conclusion

Shape from shading problem Definition Main difficulties Main approaches

Propagating solutions Horn [85] Kimmel [2001]

Minimization with BP while introducing constraints

Thank You!