Shape from Shading #1

Topics

irradiance and radiance basic concepts of reflection reflection map photometric stereo

Reflectance map and Photometric stereo

Brightness

irradiance amount of light falling on a surfacefalling energy measured by a unit surface area [watt/m2]

amount of light radiated from a surfaceemitting energy measured from a unit forshorted light source surface area to a unit solid angle [watt/ m2 ・ Sr]

solid angle --- steradian

radiance

2

cos

R

A R Θ

A

Reflection geometry

•irradiance at a pixel depends on•illumination•materials•geometry

•under the same illuminate condition, we observe irradiance difference on the same material surfacethere is a relationship between pixel irradiance and geometry•Reflectance geometry

L=illuminationN=normalV=viewer

VLg

VNe

LNi

cos

cos

cos

L

i=incidence anglee=emitting angleg=phase angle

NVi e

g

Gradient space

reflection functions are defined in the local coordinate system(e,i,g)

For our development, we will redefine the reflectance geometry in the gradient space viewers is always on the Z axis

1

)1,,(

,

22

qp

qpn

Y

Zq

X

Zp

q

p

Surface and body reflection Surface reflection and body reflection

surface reflection=gloss,highlights very directional(specular)body reflection =object color all direction(diffuse) plastic, paint have both metal has only surface reflection

body

air

incident light

surfacereflection

bodyreflection

internalpigment

Model for body reflection

Diffuse---scatters in all directionscommon approximation:

equal in all directions“lambertian”Lambertian’s cosine law“perfectly diffuse reflector”

reflectance=constant * geometric factorf(i,e,g) = Kb * cos i

why cos i ? angle of incidence affects “density” of illumination.(irradiance)

irradiance=light/arealight=1area=1/cos iirradiance = cos i

Calculating a reflection map (Lambertian)

for each(p,q), N=(p,q,1) light source direction, S= )1,,( ss qp

11

1

cos),,,(

2222

ss

ss

ss

qpqp

qqpp

LNiqpqpR

)0,0(),( ss qp

1

1),(

22

qpqpR

iso-brightness contour

p

q

0.5

0.80.9

Reflectance map(continue)

Lambertian

Self-shadow line

p

q

Surface reflection

metals have the only surface reflectiondielectrics(plastics,paint)have the surface reflection as well as the body reflectionsimplest approximation: perfect mirror

reflection is specular direction, S’S’ is coplanar with S,NSN = i = NS’: opposite sides

SN

S’

i i

Phong’s model

calculate angle between S’ and V ---α

f-surface(i,e,g)=Ks * cos α

typical : n = 10 to 500heuristic model

tells amount of light at each angle

n=1

Cos α

3

5

Real surfaces are rough : light scatters

n n

Reflectance map

brightdark

q

q

p

p

Better model of surface reflection

Phong’s model R=Ks cosnα not based on physics

just looks OK for graphics, not really accurate

off-specular effect

Torrance and Sparrow --- geometrical opticsBeckmann and Spizzichino --- physical optics

composite surface reflectionPhong’s model real surface

Phong’s model real surface

Torrance and Sparrow model

Geometrical optics– a collection of planar mirror-like facets– surface reflection caused only by these microfacets– their sizes are much larger than wave length

average normal direction

microfacet

facet slopes to be normally distributed

V-shaped valleys

facet normal

0

)/exp()( 22

cp

α

surface reflectance = constant for material* effect of one ray

* % not blocked by others (geometrical attenuation)

* % of all facets involved

Surface reflectance

Effect of one ray

incoming energy = A cos i

outgoing energy =(A cos i) / (A cos e) =cos i / cos e

i

e

Geometric attenuation

1) masking 2) shadowing

g(i,e)

% of all facets involved

idd cos/4'

eeigK s cos/),()/exp( 22

'd α

'dd

i

reflection distribution

facet normal distributionα

N

)/exp( 22

Beckmann and Spizzichino modelphysical optics

surface is continuous h(x,y)light is wave reflection off of surfaceroughness is amplitude and spatial frequency of variations in

h(x,y)E(x,y,z) “field” of light energysurface is assumed to be a perfect conductor(metal)--- > Maxwell’s equation exact solution is vicious integral

,!

cos'

1

4/22

222

222

22

22

m

mTm

gi xyemm

g

A

DTe

R

AEEE

where

22

y

2

)cos(coscos

cossinsincoscos1

)X)sinc(sinc(

)cos(cos2

yxxy

rii

iriri

x

rih

D

Y

g

Our model (Nayer,Ikeuchi,Kanade89)

Torrance and Sparrow + Beckmann and Spizzichino•diffuse lobe --- cosine function•specular spike --- delta function•specular lobe --- gaussian function

recall

VNe

VSa

em

aK

qpqp

qqppK

qpqpR

qqp

qpp

qp

qpS

V

qpN

qpL

s

ss

ssb

ss

ss

sss

ss

ss

ss

cos

cos

cos/exp11

1

),,,(

1,11

,11

then

)1,0,0(

)1,,(

)1,,(

2

2222

22

22

22

22

Calculating reflectance mapspecular lobe + diffuse lobe

Lambertian (diffuse lobe) contours Specular peak

p

q

Shape-from-shading

recover object shape (orientation) from image irradiance (brightness)

brightness surface orientation

E(x,y)=R(p,q) -- image irradiance equation

gives one constraint on the gradient space at each pixel--- > ill-posed problem (cannot solve !!!!!)

(p,q,1)0.8

p

q0.8

Photomotric stereo•one image irradiant equation gives only one constraint

--- > use multiple equations at each pixel.•take multiple images from the same points under different light source directions

•recall different light source directions give different reflectance map•at each pixel, multiple irradiance values

p

q

p

q

Photometric　 Stereo

Analytical solution

•real world gives complicated light source direction--- > look-up table method

),,(

),,(

1

1,

1,

1),,(

1

1,

1,

1),,(

3333

2222

21

21

21

21

1

21

21

11111

222222

zyx

zyx

ssss

s

ss

szyx

zyx

SSSS

SSSS

qpqp

q

qp

pSSSS

qpqp

q

qp

pnnnn

nAE

nSE

nSE

nSE

33

22

11

zyx

zyx

zyx

SSS

SSS

SSS

A

333

222

111

EAn

nAAEA

1

11

Look-up table method

Summary

Basic concepts of reflectionradiance and irradiancereflection geometrysurface reflection and body reflection

Shape-from-shading problemreflectance mapimage irradiance equationphotometric stereo

Shape-from-shading #2

Shape-from-shading #2

get a depth map from a needle map

get a needle map from a single image

Depth from surface orientation

1 dimensional caserecall

2 dimensional case

z1

)(1)( pdxdzz

dxdzP

x

0z

1zdyq

dxp

dy

dxqdypdxzz 01

Recovering depth map from a needle map(direct integration method)

Photometric stereo gives a needle map

assume a depth at the originget depth along the x of the needle map

get the depth map

pdxzz '

qdypdxzz '

Direct integration

rapid

accumulates errors

Relaxation method

observed orientation (p,q) should be same as those of the depth map (zx,zy)

reduce the total error within a boundary (the calculus of variations See Horn pp.469-474)

an iterative formula

22 )),(),(()),(),(( yxy

zyxqyx

x

zyxp

dxdyyxy

zyxqyx

x

zyxp

A

22 )),(),(()),(),((

),(),(4

1

)1,()1,(),1(),1(4

1),(1

yxy

qyx

x

p

yxzyxzyxzyxzyxz nnnnn

),(),(4

1

...),1(4

1),(1

yxy

qyx

x

p

yxzyxz nn

iterative method

needle map

brightness

depth map

Relaxation method (Example)

Jacobi’s method

yxnnnnn qpyxHyxHyxHyxHyxH

4

1)1,()1,(),1(),1(

4

1),(1

Iteratively computing the equation itself convergence is very slow

The number of iteration needed to converge is

when the error decreased to 10-p times, for the pixels whose size is J×J

2

2

1pJr

Eg. r=1200000 when J=400, p=15

Gauss-Seidel method

yxnnnnn qpyxHyxHyxHyxHyxH

4

1)1,()1,(),1(),1(

4

1),( 111

2

4

1pJr Eg. r=600000 when J=400, p=15

yxnnnnn qpyxHyxHyxHyxHyxH

4

1)1,()1,(),1(),1(

4

1),(1

Jacobi:

Gauss-Seidel:

scan-line order checkerboard pattern

Iteration number:

SOR (successive overrelaxation) method ),(1

4

1)1,()1,(),1(),1(

4

1),( 111 yxHqpyxHyxHyxHyxHyxH n

yxnnnnn

ω: overrelaxation parameter

Converge if 0<ω<2

0<ω<1 (underrelaxation): slower than Gauss-Seidel1<ω<2 (overrelaxation): faster than Gauss-Seidel

Optimal ω:J

1

2Eg. ω=1.984 when J=400

pJr3

1 Eg. r=2000 when J=400, p=15Iteration number:

Chebyshev acceleration (faster convergence): change ω properly for each iteration

same as Gauss-Seidel

ADI (alternating-direction implicit) method

yxnnnnnn qpyxHyxHyxHyxHyxHyxH )1,(),(2)1,(),1(),(2),1( 212121

yxnnnnnn qpyxHyxHyxHyxHyxHyxH ),1(),(2),1()1,(),(2)1,( 212121111

Jpr 10log3

8 Eg. r=104 when J=400, p=15Iteration number:

: changed properly for each iterationif =0 Gauss-Seidel

-1 +2 -1

……

H(1,y)

H(J,y)

……

*

…

…

*-1 +2 -1

-1 +2 -1

……

H(x,1)

H(x,J)

……

*

…

…

*-1 +2 -1

A: tridiagonal matrix (given)b: vector (given)x: solve Ax=b by using linear system solver (forward substitution) O(N)

Natural boundary condition

ds

dxds

dy

q

p

ds

dxds

dy

H

H

y

x

T

ds

dx

ds

dy

s… arc length of boundary

Natural boundary condition of "gradient-to-height problem" is

·… dot product … the normal of boundary

Algorithm:

[Truth] [Wrong result][Correct result]

Same height boundaryNatural boundary condition

Fast convergenceSlow convergence

Boundary:Height known just use it

Height unknown natural boundary condition

yxyxyx HHp ,1,1, 2

1 1,1,, 2

1 yxyxyx HHq

Calculate the height "H" at the boundary by

or for each iteration

Shape-from-shading with a single view

Photometric stereo uses multiple images.

Is there a way to recover shape from a single image?

Yes, there is a way.1. characteristics strip expansion method:obtain surface orientati

on along characteristics strips of image irradiance equation (Horn 75)

2. relaxation method:obtain surface orientation using image irradiance equation and smoothness constraint (Ikeuchi and Horn 81)

3. global method: assume a surface is a part of sphere (Pentland 83)

Characteristic strip expansion method

the steep descent direction of the reflectance map(gradient space)

the steep descent direction of the image brightness(image brightness)

p

q

x

y

x

y

SDD of RM

SDD of IB

SDD of RM

SDD of IB

p

q

move towards the SDD of the reflectance map on the image plane

move towards the SDD of the image brightness on the gradient space

Characteristic Strip

Proof1. Taylor expansion of p(x,y) and q(x,y)

2. derivative of the image irradiance equation

dyy

qdx

x

qyxqdyydxdq

dyy

pdx

x

pyxpdyydxxp

),(),(

),(),(

dyy

qdx

x

qdq

dyy

pdx

x

pdp

),( yxp

),( ydxxp

x dxx

)),(),,((),( yxqyxpRyxE

x

q

q

R

x

p

p

R

x

E

y

q

q

R

y

p

p

R

y

E

Move towards the SDD of the reflectance map on the image plane

then, what happen to (p(x,y),q(x,y)) ?

move towards the SDD of the image brightness on the gradient

space

Sp

Rx

Sq

Ry

p

R

q

R

S

),(q

R

p

R

on the image

Sx

E

Sx

q

q

R

x

p

p

RS

y

p

q

R

x

p

p

R

Sq

R

y

pS

p

R

x

py

y

px

x

pp

y

E

x

E, ),( qpon

),( qp RR

),( qp EE p

q

0.10.20.30.40.5

xEdp yEdq

x

y

0.1

0.2

0.30.4

from a known point, (you know (p,q) and E)you can determine (p,q) along a characteristic strip

x

y

Sp

Rx

Sq

Ry

character strip

reconstructed contour

q

p

Horn 75

Problem of characteristic stip method

1. Error accumulation.

2. The method starts from a singular point the start point is unreliable.

3. Determine surface orientation only along characteristic stripes.

4. Occluding contours are big evidences. We cannot use that information. (p,q) become infinite.

relaxation method with occluding contours

???

Occluding boundary

Surface orientations on occluding boundaries are known from the shape of silhouette.

These surface orientations cannot be represented by the gradient space. (p,q) becomes infinite.

We will use the stereographic plane, (f,g). On (f,g) plane, occluding boundaries lie on the unit circle.

boundary condition

Stereographic Projection

2222 11

2,

11

2

qp

qg

qp

pf

occluding boundaries lie on the unit circle

gradientspace

p

q

gaussian sphere

Relaxation method

1. Image irradiance equation on (f,g) space on the (f,g) space, we can also define a reflectance map.

2. Smoothness constraint. Neighboring points have roughly the same surface orientation.

)),(),,((),( yxgyxfRyxE

02222

y

g

x

g

y

f

x

f

Relaxation method3. Set up a minimization problem.

4. Using the calculus of variations get iterative formula.

dxdyy

g

x

g

y

f

x

f

yxgyxfRyxEE

2222

2))),(),,((),((

)),(),,((),(

)1,()1,(),1(),1(41),(1

yxgyxfRyxE

yxfyxfyxfyxfyxf

nn

nnnnn

→min

)),(),,((),(

)1,()1,(),1(),1(41),(1

yxgyxfRyxE

yxgyxgyxgyxgyxg

nn

nnnnn

...)...,1(41),(1 yxfyxf nn

...)...,1(41),(1 yxgyxg nn

n+hsolution

n+1+hsolution

brightness image

occluding boundary

needle map depth map

Ikeuchi & Horn 81

Summary

1. Gaussian sphere and reflectance map2. Get a depth map from a needle map

1. direct integration2. relaxation method

3. Get a needle map from a single image1. characteristic strip expansion method2. relaxation method regularization

4. Read Horn pp. 244-269, B&B pp. 93-101

Shape-from-shading #3

Shape-from-shading #3– Shape from shading

More advanced researches– 4-light photometric stereo– Extended light source– Photometric sampling– Shape from interreflection– Inverse polarization raytracingspecular spike interreflection

+ transmission

diffuse interreflection

diffuse

diffuse + specular

specular spike

diffuse + specular spike

4-light photometric stereo

[diffuse pixel]

4 correctanswer

smalldeviation

[specular pixel]

1 correctanswer

largedeviation

4-light photometric stereo [Coleman&Jain]

[diffuse pixel]

Averageof four

[specular pixel]

Use 3dark pixels

Deviation

Smaller than threshold Larger than threshold

4-light photometric stereo [Barsky&Petrou]

specular

diffuse

length

threshold[specular]dark 3

[non-specular]bright 3

[specular pixel] [diffuse pixel] [shadow pixel]

light

specular normal surface normal

difference

4-light photometric stereo [Solomon&Ikeuchi]

4 3

3

3

3

2 2

22

Gaussian sphere

[4 light region]

[Coleman&Jain]

[3 light region]

Use two lights

p

q

Gradient space

Two possible normals

Choose negative shading of shadow light

shadowspecular

[2 light region]Use two lights Two possible normals

diffuse

diffuse

p

q

Gradient spaceChoose negative shading of shadow light

shadowshadow

diffuse

diffuse

Planar extended light source

L

l0

X

Brightness: f

Line lamp

Specular object

Lambertian plane(Extended light source)Camera

r

Irradiance: E(f,r,X,l0,L)

Reflectance map3 linear lamps 3 reflectance maps Solve by photometric stereo

Photometric sampler

Spherical extended light sourceL()=L(;s,R,H,I,C)

C: constantI: radiant intensity of light S

: termination angle

Photometric function for Lambertian surface

)cos( nsAI Lambertian

Lambertian of extended light)cos( nkk AI

Estimate reflectance A'and surface normal n

by fitting cosine function to images

Photometric function for specular surface

)2( nsBI Specular spike

)2( nkk LBI Specular spike of extended light

Point sources are separatedby source termination angle

Only two specular peaks appear

Photometric function for specular surface)2( nkk LBI

Point sources are separatedby source termination angle

)()(

)()()(

1

1

kk

kkk LL

LLD

1-to-1 correspondence

n is calculated

)2( nk

k

L

IB

is calculated

Photometric function for hybrid surface)2()cos( nsns BAI

)2()cos( nknkk LBAI

1. Remove 2 intensities2. Calculate nl by fitting

Lambertian model fromremaining intensities

3. Calculate ns from 2 intensities

Calculate (nl,ns)for all possibilities

Choose where

Finally,

nsnl

BA

BA nsnln

Example

Interreflection

Form factor

dxdx'r

'

Interreflection kernel: 2

coscos),(),(

r

xxVxxK

Visibility function: V=1 if visible, V=0 if invisible

xdxLxxKx

xxdLm )(),()(

),(

Radiance of x Irradiance of x

Albedo Radiance of x'

Interreflection equationPKLLL s

Facet radiance vector: TmLLL ,,, 21 L Source contribution vector: Tsmsss LLL ,,, 21 L

Albedo matrix:

m

00

00

00

1 2

1

P Kernel matrix:

0

0

0

21

221

112

mm

m

m

KK

KK

KK

K

Inverse radiosity

PKLLL s sLPKIL 1

sFL s (Shading) Facet matrix: TmNNNF ,,, 21 i

ii nN

Albedo

Surface normal

Source direction vector: Tzyx sss ,,s

sFPKIL 1

sFL p

FPKIF 1pPseudo facet matrix:

(Pseudo shading) Estimate Fp by conventional photometric stereo(L: input image, s: known light source)

Estimate F iteratively from initial value Fp

pFPKIF

kP kKand is straighforwardly calculated from kF

pkkk FKPIF 1Recovery algorithm:

Example

Application to book scanning

Image scanner

Scanning a book

Observed image

Estimated shape

Restored image

Polarization

Light = wave oscillates Oscillates in certain direction polarization DOP = degree of polarization

Unpolarized(DOP 0)

Light

Perfectly polarized(DOP 1)

Polarizer

Partially polarized(DOP 0~1)

Incident Reflected

AirObject

Transmitted

Reflection and transmissionNormal

Unpolarized

Air

Object

Partially polarized

Partially polarized

Light

Dependsupon

Polarization raytracing

Calculate polarization(Mueller calculus)

Calculate reflection & transmission(Ray-tracing)

Light(4D vector)

Reflection&transmission(4x4 matrix)

Minimization

Solve by Shape-From-Shading technique

dxdyqHpHII yxRE222min

RE II Polarization raytracing equation

Input Calculated by polarization raytracing

Example

Result(10 iteration)

Glass(refractive index 1.5)

Summary

4-light photometric stereo– Photometric stereo from proper 3 lights

Extended light source– Photometric stereo from Lambertian planar light source

Photometric sampling– Shape and reflectance from spherical extended light source

Shape from interreflection– Photometric stereo for concave diffuse object

Inverse polarization raytracing– Shape of transparent object by using SFS technique

Reference Horn, B.K.P. "Robot Vision," MIT Press, 1986 Zhang, R., Tsai, P.S., Cryer, J.D., Shah, M., "Shape from Shading: A

Survey," IEEE Trans. PAMI, Vol. 21, No. 8, pp. 690-706, 1999. Horn, B.K.P “Obtaining shape from shading information,” in The Psy

chology of Computer Vision, P.H. Winston (ed.), McGraw-Hill, 1995 Ikeuchi, K. & B.K.P. Horn, “Numerical shape from shading and occlu

ding boundaries,” Artificial Intelligence, Vol. 17, 1981. Pentland, A.P., “Local shading analysis,” IEEE Trans. PAMI, Vol.6, 1

984. Klinker, G.J., S.A. Shafer & T. Kanade, “The measurement of highlig

ht in color image,” Int. J. Computer Vision, Vol.2, 1988. Nayar S.K., K. Ikeuchi, and T. Kanade "Surface reflection: physical an

d geometrical perspectives", IEEE Trans. PAMI, Vol. 13, pp.661-634, 1991.