recovering photometric properties of architectural scenes from photographs

July 1998

Yizhou Yu Jitendra MalikYizhou Yu Jitendra Malik

Computer Science DivisionComputer Science DivisionUniversity of California at BerkeleyUniversity of California at Berkeley

Recovering Photometric Properties of Architectural Scenes from Photographs

Context

• IBMR re-renders from novel viewpoints. IBMR re-renders from novel viewpoints.

– Façade, Plenoptic modeling,Façade, Plenoptic modeling,

– Lumigraph, Light field, Lumigraph, Light field,

– Panoramic mosaicsPanoramic mosaics

• But, unlike traditional rendering, But, unlike traditional rendering, lighting cannot be changed.lighting cannot be changed.

The Problem

• Texture Maps areTexture Maps are not not Reflectance Maps ! Reflectance Maps !

• Need to factorize images into Need to factorize images into lighting and reflectance mapslighting and reflectance maps

Illumination Radiance

Reflectance

Objective

• Start from photographsStart from photographs

• Recover parametric models for lighting and Recover parametric models for lighting and reflectancereflectance

• Re-render the scene under novel lighting Re-render the scene under novel lighting conditionsconditions

Some Photographs...

Camera Radiance Response Curve

• Pixel brightness value is a Pixel brightness value is a nonlinear function of nonlinear function of radiance.radiance.

– Debevec & Malik[Siggraph’97] Debevec & Malik[Siggraph’97] give a method to recover this give a method to recover this nonlinear mapping.nonlinear mapping.

RadianceRadiance

IntensitySaturation

Previous Work

• BRDF measurement and recoveryBRDF measurement and recovery

– [Ward 92],[Dana et al. 97][Ward 92],[Dana et al. 97]

– [Sato & Ikeuchi 96], [Sato et al. 97][Sato & Ikeuchi 96], [Sato et al. 97]

• Rendering outdoor scenes under skylightRendering outdoor scenes under skylight

– [Nishita and Nakamae 86], [Tadamura et al. 93][Nishita and Nakamae 86], [Tadamura et al. 93]

Basic Approach

• Recover geometric modelRecover geometric model

• Measure and recover illuminationMeasure and recover illumination

• Recover reflectanceRecover reflectance

• Predict illumination at novel times of dayPredict illumination at novel times of day

• RenderRender

Illumination Radiance

Reflectance

Technical Challenges

• Nonlinear mapping between input radiance Nonlinear mapping between input radiance and digital output .and digital output .

• Photographs cannot easily recover full Photographs cannot easily recover full spectral BRDF.spectral BRDF.

• Re-rendering the scene at novel times of Re-rendering the scene at novel times of day requires predicting lighting conditions.day requires predicting lighting conditions.

Basic Approach

Measure and recover illuminationMeasure and recover illumination

• Recover reflectanceRecover reflectance

• Predict illumination at novel times of dayPredict illumination at novel times of day

• RenderRender

Illumination Radiance

Reflectance

Modeling the Illumination

• The sunThe sun

– Its diameter extends 31.8’ seen from the earth.Its diameter extends 31.8’ seen from the earth.

• The skyThe sky

– A hemispherical area light source.A hemispherical area light source.

• The surrounding environmentThe surrounding environment

– Modeled as a set of oriented Lambertian facets.Modeled as a set of oriented Lambertian facets.

A Sky Radiance Model----based on [Perez 93]

) cos ) exp( 1 ))( /cos exp( 1 Lvz( 2 edcba f

• Recover a set of parameters Recover a set of parameters for each color channelfor each color channel

– Take photographs for parts of the skyTake photographs for parts of the sky

– Use Levenberg-Marquardt algorithm to fit dataUse Levenberg-Marquardt algorithm to fit data

sun

zenithSky element

Lvz, a, b, c, d, e, f

A Recovered Sky Radiance Model

R,G,B channels

Coarse-grain Environment Radiance Maps• Partition the lower hemisphere Partition the lower hemisphere

into small regions into small regions

• Take photographs at several Take photographs at several times of daytimes of day

• Project pixels into regions and Project pixels into regions and obtain the average radianceobtain the average radiance

• Use photometric stereo to Use photometric stereo to recover a facet model for each regionrecover a facet model for each region

Basic Approach

• Measure and recover illuminationMeasure and recover illumination

Recover reflectanceRecover reflectance

• Predict illumination at novel times of dayPredict illumination at novel times of day

• RenderRender

Recovering Reflectance

• Parametric model [Lafortune et al.]Parametric model [Lafortune et al.]

• Triangulate the surfaces Triangulate the surfaces

• Set a grid on each triangle to Set a grid on each triangle to capture spatial variationscapture spatial variations

• Use one-bounce reflection to Use one-bounce reflection to approximate self-interreflectionsapproximate self-interreflections

nzzyyxxsd ) vu vu vu ( zyx CCC nzzyyxxsd ) vu vu vu ( zyx CCC

Pseudo-BRDF

• R, G, B color channels perform integration. R, G, B color channels perform integration. Define pseudo-BRDF :Define pseudo-BRDF :

• In general, the pseudo-BRDF varies with the In general, the pseudo-BRDF varies with the spectral distribution of the light source.spectral distribution of the light source.

• Recover two sets of surface pseudo-BRDFsRecover two sets of surface pseudo-BRDFs – One ==> spectral distribution of the sunOne ==> spectral distribution of the sun

– The other ==> the sky and environmentThe other ==> the sky and environment

dRI

dRI

ii

rriiii

(,,(

(,,,,),,(

Diffuse Term• For each side, at least two photographs for For each side, at least two photographs for

diffuse albedo recovery.diffuse albedo recovery.

– From the photograph not lit by the sunFrom the photograph not lit by the sun

– From the photograph lit by the sunFrom the photograph lit by the sun

– Solve for Solve for

)1()1(se

seEI

sunsun

sese EEI )2()2(

sunse ,

Specular Term

• Use an empirical specular reflection model Use an empirical specular reflection model proposed in [Lafortune et al. 97].proposed in [Lafortune et al. 97].

• Recover the parameters using Recover the parameters using least squares and robust statistics.least squares and robust statistics.

nzzyyxxs ) vu vu vu ( zyx CCC

Basic Approach

• Measure and recover illuminationMeasure and recover illumination

• Recover reflectanceRecover reflectance

Predict illumination at novel times of dayPredict illumination at novel times of day

• RenderRender

Simulating Novel Lighting for the Sun and Sky

• Interpolation with solar position alignment Interpolation with solar position alignment to obtain novel sky radiance distributionsto obtain novel sky radiance distributions

• Use to model solar radiance Use to model solar radiance during sunrise and sunsetduring sunrise and sunset

– This is similar to the absorption term used in This is similar to the absorption term used in scattering theory.scattering theory.

d) exp(-

A Local Facet Model for the Environment

• Recover a distinct model for each Recover a distinct model for each environment regionenvironment region

– Obtain environment radiance maps.Obtain environment radiance maps.

– Set up over-determined systems as in Set up over-determined systems as in photometric stereo and ignore inter-reflections.photometric stereo and ignore inter-reflections.

– Solve forSolve for

otherwise.

,0n if

,

),n(

envsun

skysky

envsunsunsun

skysky

env

l

E

lEEI

envsunsky n ,,

nenvlsun

Recovered Environment Radiance Models

Synthetic Real

Relative Importance of the Components• On shaded sides, the irradiance from the On shaded sides, the irradiance from the

landscape is larger than that from the sky.landscape is larger than that from the sky.

• On sunlit sides, the sun dominates the On sunlit sides, the sun dominates the illumination.illumination.

• The specular component is very small The specular component is very small compared to the diffuse component.compared to the diffuse component.

Video

Basic Approach

• Measure and recover illuminationMeasure and recover illumination

• Recover reflectanceRecover reflectance

• Predict illumination at novel times of dayPredict illumination at novel times of day

RenderRender

Comparison with Real Photographs

Synthetic Real

High Resolution Re-rendering

• Low resolution and Low resolution and High resolutionHigh resolution

• and are given.and are given.

• since the illumination has since the illumination has small variations in high frequencies. small variations in high frequencies.

•

),(~

),,(~

),,(~ yxEyxIyx),(),,(),,( yxEyxIyx

),(~

),(

),(

),(),(

yxE

yxI

yxE

yxIyx

),(~ yx ),( yxI

),(~

),( yxEyxE

High Resolution Re-rendering

Real reference image

High resolutionsynthetic image

Low resolutionsynthetic image

Video

Summary

• An approach to render real architectural An approach to render real architectural scenes under novel lighting conditionsscenes under novel lighting conditions

• The pseudo-BRDF conceptThe pseudo-BRDF concept

• Methods for modeling lighting at Methods for modeling lighting at novel times of daynovel times of day

• A simple method for high resolution A simple method for high resolution re-renderingre-rendering

Acknowledgments

• George BorshukovGeorge Borshukov

• Paul DebevecPaul Debevec

• David ForsythDavid Forsyth

• Greg Ward LarsonGreg Ward Larson

• Carlo SequinCarlo Sequin

• MURI 3DDI MURI 3DDI California MICRO Program California MICRO Program Philips CorporationPhilips Corporation