Tal Amir Advanced Topics in Computer Vision May 29 th, 2015 COUPLED MOTION- LIGHTING ANALYSIS 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE SURFACE RECONSTRUCTION: OVERVIEW Structure from motion: Images are taken from different locations Scene remains static Seek dense correspondence SURFACE RECONSTRUCTION: OVERVIEW Structure from motion: Images are taken from different locations Scene remains static Seeking dense correspondence Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW Structure from motion: Images are taken from different locations Scene remains static Seek dense correspondence Use it to recover depths Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW Structure from motion: Images are taken from different locations Scene remains static Seek dense correspondence Use it to recover depths Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW Very simplistic assumption Requires Lambertian reflectance Photometric reconstruction (stereo): Images of the scene are taken with different illuminations Scene and camera remain static Uses a light scattering model SURFACE RECONSTRUCTION: OVERVIEW Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), Photometric reconstruction (stereo): Few images of the scene are taken with different illuminations Scene and camera remain static Uses a light scattering model What about reconstruction with Object motion? Camera motion? SURFACE RECONSTRUCTION: OVERVIEW Coupled motion-lighting analysis Uses a light scattering model Takes object / camera motion into account Assumes small (differential) motion, which is known SURFACE RECONSTRUCTION: OVERVIEW Why assume differential motion? If the motion is large SURFACE RECONSTRUCTION: OVERVIEW Why assume differential motion? we need point correspondences. SURFACE RECONSTRUCTION: OVERVIEW Why assume differential motion? If we have a correspondence for all points, weve already solved the problem. Therefore, we assume the camera rotation and translation are small. SURFACE RECONSTRUCTION: OVERVIEW 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections Subsurface scattering There are other, more complex phenomena, such as interreflections and shadow casting. LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): Generalizes specular and Lambertian reflections. A narrower model: Symmetric BRDF. Rotation invariant. Depends only on the angles between the three vectors. LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): Generalizes specular and Lambertian reflections. A narrower model: Symmetric BRDF. Rotation invariant. Depends only on the angles between the three vectors. Does not account for subsurface scattering. Only good for opaque surfaces. LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE Basri, R., & Frolova, D. (2008): A two-frame theory of motion, lighting and shape Assumptions: Orthographic projection Known (differential) object motion Lambertian reflectance Possibly non-uniform albedo Known directional lighting SHAPE FROM OBJECT MOTION Basri, R., & Frolova, D. (2008, June). A two-frame theory of motion, lighting and shape. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on (pp. 1-7). IEEE. SHAPE FROM OBJECT MOTION We need boundary conditions. At the boundary of the silhouette, we know the direction of the surface normal. SHAPE FROM OBJECT MOTION Results SHAPE FROM OBJECT MOTION 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013): What object motion reveals about shape with unknown BRDF and lighting A general framework for surface reconstruction from camera motion. GENERALIZING TO UNKNOWN LIGHTING AND BRDF Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013, June). What object motion reveals about shape with unknown BRDF and lighting. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp ). IEEE. Tells us when we can or cannot reconstruct surface. Treats both orthographic and perspective projections. Generalizes to complex unknown illumination. Does not assume Lambertian reflectance. Limiting assumptions: Known differential motion Symmetric BRDF Object is distant from camera and light sources GENERALIZING TO UNKNOWN LIGHTING AND BRDF Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF Lighting is co-located with the camera: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF Unknown lighting: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF Synthetic images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE Chandraker, M., (2014): What camera motion reveals about shape with unknown BRDF Handles the case of camera motion, while object and lighting remain static. This task is more complex than object motion. SHAPE FROM CAMERA MOTION Chandraker, M. (2014, June). What camera motion reveals about shape with unknown BRDF. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on (pp ). IEEE. For orthographic projection: Depth cannot be recovered purely from camera motion. A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION For orthographic projection: Depth cannot be recovered purely from camera motion. A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. For perspective projection: Depth can be directly recovered from 4 images. An additional linear PDE can be recovered if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF Orthographic projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF Perspective projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF Perspective projection (real images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE 1.Surface reconstruction: Overview Structure from Motion Photometric Reconstruction Coupled Motion-Lighting Analysis 2.Light scattering models Lambertian reflectance BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE We have seen: Surface reconstruction from two frames Known directional lighting Lambertian reflectance Orthographic projection General framework for object motion Unknown illumination Symmetric BRDF Orthographic / perspective projection General framework for camera motion CONCLUSION