Slide 1

Shape-from-X Class 11 Some slides from Shree Nayar. others

Slide 2

3D photography course schedule (tentative) LectureExercise Sept 26Introduction- Oct. 3Geometry & Camera modelCamera calibration Oct. 10Single View MetrologyMeasuring in images Oct. 17Feature Tracking/matching (Friedrich Fraundorfer) Correspondence computation Oct. 24Epipolar GeometryF-matrix computation Oct. 31Shape-from-Silhouettes (Li Guan) Visual-hull computation Nov. 7Stereo matchingProject proposals Nov. 14Structured light and active range sensing Papers Nov. 21Structure from motionPapers Nov. 28Multi-view geometry and self-calibration Papers Dec. 5Shape-from-XPapers Dec. 123D modeling and registrationPapers Dec. 19Appearance modeling and image-based rendering Final project presentations

Slide 3

Shape-from-X X =Shading X =Multiple Light Sources (photometric stereo) X =Texture X =Focus/Defocus X =Specularities X =Shadows X =

Slide 4

Shape from shading Shading as a cue for shape reconstruction What is the relation between intensity and shape? Reflectance Map

Slide 5

Relates image irradiance I(x,y) to surface orientation (p,q) for given source direction and surface reflectance Lambertian case: : source brightness : surface albedo (reflectance) : constant (optical system) Image irradiance: Letthen

Slide 6

Lambertian case Reflectance Map (Lambertian) cone of constant Iso-brightness contour Reflectance Map

Slide 7

Lambertian case iso-brightness contour Note: is maximum when Reflectance Map

Slide 8

Shape from a Single Image? Given a single image of an object with known surface reflectance taken under a known light source, can we recover the shape of the object? Given R(p,q) ( (p S,q S ) and surface reflectance) can we determine (p,q) uniquely for each image point? NO

Slide 9

Solution Add more constraints Shape-from-shading Take more images Photometric stereo

Slide 10

Stereographic Projection (p,q)-space (gradient space) (f,g)-space Problem (p,q) can be infinite when Redefine reflectance map as

Slide 11

Occluding Boundaries and are known The values on the occluding boundary can be used as the boundary condition for shape-from-shading

Slide 12

Image Irradiance Constraint Image irradiance should match the reflectance map Minimize (minimize errors in image irradiance in the image)

Slide 13

Smoothness Constraint Used to constrain shape-from-shading Relates orientations (f,g) of neighboring surface points : surface orientation under stereographic projection Minimize (penalize rapid changes in surface orientation f and g over the image)

Slide 14

Shape-from-Shading Find surface orientations (f,g) at all image points that minimize smoothness constraint weight image irradiance error Minimize

Slide 15

Results

Slide 16

Slide 17

Solution Add more constraints Shape-from-shading Take more images Photometric stereo

Slide 18

Photometric Stereo

Slide 19

We can write this in matrix form: Image irradiance: Lambertian case: Photometric Stereo : source brightness : surface albedo (reflectance) : constant (optical system)

Slide 20

Solving the Equations inverse

Slide 21

More than Three Light Sources Get better results by using more lights Least squares solution: Solve for as before Moore-Penrose pseudo inverse

Slide 22

Color Images The case of RGB images get three sets of equations, one per color channel: Simple solution: first solve for using one channel Then substitute known into above equations to get Or combine three channels and solve for

Slide 23

Computing light source directions Trick: place a chrome sphere in the scene the location of the highlight tells you the source direction

Slide 24

For a perfect mirror, light is reflected about N Specular Reflection - Recap We see a highlight when Then is given as follows:

Slide 25

Computing the Light Source Direction Can compute N by studying this figure Hints: use this equation: can measure c, h, and r in the image N rNrN C H c h Chrome sphere that has a highlight at position h in the image image plane sphere in 3D

Slide 26

Depth from Normals Get a similar equation for V 2 Each normal gives us two linear constraints on z compute z values by solving a matrix equation V1V1 V2V2 N

Slide 27

Limitations Big problems Doesnt work for shiny things, semi- translucent things Shadows, inter-reflections Smaller problems Camera and lights have to be distant Calibration requirements measure light source directions, intensities camera response function

Slide 28

Trick for Handling Shadows Weight each equation by the pixel brightness: Gives weighted least-squares matrix equation: Solve for as before

Slide 29

Original Images

Slide 30

Results - Shape Shallow reconstruction (effect of interreflections) Accurate reconstruction (after removing interreflections)

Slide 31

Results - Albedo No Shading Information

Slide 32

Original Images

Slide 33

Results - Shape

Slide 34

Results - Albedo

Slide 35

Results 1.Estimate light source directions 2.Compute surface normals 3.Compute albedo values 4.Estimate depth from surface normals 5.Relight the object (with original texture and uniform albedo)

Slide 36

Shape from texture Obtain normals from texture element (or statistics) deformations, Examples from Angie Loh

Slide 37

Shape from texture Obtain normals from texture element (or statistics) deformations, Examples from Angie Loh

Slide 38

Depth from focus Sweep through focus settings most sharp pixels correspond to depth (most high frequencies)

Slide 39

Depth from Defocus More complicates, but needs less images Compare relative sharpness between images

Slide 40

Shape from shadows S. Savarese, M. Andreetto, H. Rusmeier, F. Bernardini, P. Perona, 3D Reconstruction by Shadow Carving: Theory and Practical Evaluation, International Journal of Computer Vision (IJCV), vol 71, no. 3, pp. 305-336, March 2007.

Slide 41

Shape from specularities Toward a Theory of Shape from Specular Flow Y. Adato, Y. Vasilyev, O. Ben-Shahar, T. Zickler Proc. ICCV07

Slide 42

Next class: 3d modeling and registration