A Signal-Processing Framework A Signal-Processing Framework for Inverse Renderingfor Inverse Rendering

A Signal-Processing Framework A Signal-Processing Framework for Inverse Renderingfor Inverse Rendering

Ravi RamamoorthiRavi Ramamoorthi Pat HanrahanPat Hanrahan

Stanford UniversityStanford University

Photorealistic RenderingPhotorealistic RenderingPhotorealistic RenderingPhotorealistic Rendering

Materials/Lighting(Texture Reflectance[BRDF] Lighting)

Realistic input models required

Arnold Renderer: Marcos Fajardo

Rendering Algorithm

80’s,90’s: Physically based

Geometry

70’s, 80’s: Splines90’s: Range Data

FlowchartFlowchartFlowchartFlowchart

Photographs

Geometric model

InverseRenderingAlgorithm

Lighting

BRDF

FlowchartFlowchartFlowchartFlowchart

Photographs

Geometric model

ForwardRenderingAlgorithm

Lighting

BRDF

Rendering

FlowchartFlowchartFlowchartFlowchart

Photographs

Geometric model

ForwardRenderingAlgorithm

BRDF

Novel lighting

Rendering

AssumptionsAssumptionsAssumptionsAssumptions

• Known geometry

• Distant illumination

• Homogenous isotropic materials

• Convex curved surfaces: no shadows, interreflection

Later, practical algorithms: relax some assumptions

• Known geometry

• Distant illumination

• Homogenous isotropic materials

• Convex curved surfaces: no shadows, interreflection

Later, practical algorithms: relax some assumptions

Inverse RenderingInverse RenderingInverse RenderingInverse Rendering

Miller and Hoffman 84

Marschner and Greenberg 97

Sato et al. 97

Dana et al. 99

Debevec et al. 00

Marschner et al. 00

Sato et al. 99

Nishino et al. 01

Lighting

BRDF

Known Unknown

Unknown

Known

Textures are a third axis

Inverse Problems: Difficulties Inverse Problems: Difficulties Inverse Problems: Difficulties Inverse Problems: Difficulties

Angular width of Light Source

Su

rfac

e ro

ugh

nes

sIll-posed

(ambiguous)

ContributionsContributionsContributionsContributions

1. Formalize reflection as convolution

2. Signal-processing framework

3. Analyze well-posedness of inverse problems

4. Practical algorithms

1. Formalize reflection as convolution

2. Signal-processing framework

3. Analyze well-posedness of inverse problems

4. Practical algorithms

Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)

2

2

id

Li

( )iL Lighting

( )oB Reflected Light Field

( , )i o BRDF

( )iL ( , )i o id2

2

( , )oB

Li o

Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)

B L

, ,2l p l l pB L Fourier analysis

Li

Li

o

Frequency: product

Spatial: integral

( )iL ( , )i o id2

2

( , )oB

Spherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic Analysis

, ,2l p l l pB L

2D:

22

0 0 ,( , , , ) ( [ , ]) ( , , , )o o i i i i o o i iB L R d d

3D:

, ,lm pq l lm lq pqB L

( )iL ( , )i o id2

2

( , )oB

Insights: Signal ProcessingInsights: Signal ProcessingInsights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Insights: Signal ProcessingInsights: Signal ProcessingInsights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Filter is Delta function : Output = SignalFilter is Delta function : Output = Signal

Mirror BRDF : Image = Lighting [Miller and Hoffman 84]

Mirror BRDF : Image = Lighting [Miller and Hoffman 84]

Image courtesy Paul DebevecImage courtesy Paul Debevec

Insights: Signal ProcessingInsights: Signal ProcessingInsights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Signal is Delta function : Output = FilterSignal is Delta function : Output = Filter

Point Light Source : Images = BRDF [Marschner et al. 00]

Point Light Source : Images = BRDF [Marschner et al. 00]

Am

plit

ude

Frequency

Roughness

Phong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet Models

Mirror

Illumination estimationill-posed for rough surfaces

Inverse LightingInverse LightingInverse LightingInverse Lighting

Well-posed unless denominator vanishes• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors low pass filters: Inverse lighting ill-

posed

Well-posed unless denominator vanishes• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors low pass filters: Inverse lighting ill-

posed

,

,

1 lm pqlm

l lq pq

BL

, ,lm pq l lm lq pq

B L

B L

Given: B,ρ find LGiven: B,ρ find L

Inverse BRDFInverse BRDFInverse BRDFInverse BRDF

Well-posed unless Llm vanishes• Lighting should have sharp features (point sources, edges)• BRDF estimation ill-conditioned for soft lighting

Well-posed unless Llm vanishes• Lighting should have sharp features (point sources, edges)• BRDF estimation ill-conditioned for soft lighting

,,

1 lm pqlq pq

l lm

B

L

Directional

Source

Area source

Same BRDF

Given: B,L find ρGiven: B,L find ρ

Factoring the Light FieldFactoring the Light FieldFactoring the Light FieldFactoring the Light Field

Light Field can be factored• Up to global scale factor• Assumes reciprocity of BRDF• Can be ill-conditioned• Analytic formula in paper

Light Field can be factored• Up to global scale factor• Assumes reciprocity of BRDF• Can be ill-conditioned• Analytic formula in paper

Given: B find L and ρGiven: B find L and ρ

B L

4D 2D 3D

More knowns (4D)than unknowns (2D/3D)

More knowns (4D)than unknowns (2D/3D)

Practical IssuesPractical IssuesPractical IssuesPractical Issues

• Incomplete sparse data (few photographs)Difficult to compute frequency spectra

• Concavities: Self Shadowing and Interreflection

• Spatially varying BRDFs: Textures

• Incomplete sparse data (few photographs)Difficult to compute frequency spectra

• Concavities: Self Shadowing and Interreflection

• Spatially varying BRDFs: Textures

Practical IssuesPractical IssuesPractical IssuesPractical Issues

• Incomplete sparse data (few photographs)Difficult to compute frequency spectra

• Concavities: Self Shadowing and Interreflection

• Spatially varying BRDFs: Textures

Issues can be addressed; can derive practical algorithmsDual spatial (angular) and frequency-space representationSimple extensions for shadowing, textures

• Incomplete sparse data (few photographs)Difficult to compute frequency spectra

• Concavities: Self Shadowing and Interreflection

• Spatially varying BRDFs: Textures

Issues can be addressed; can derive practical algorithmsDual spatial (angular) and frequency-space representationSimple extensions for shadowing, textures

Algorithm ValidationAlgorithm ValidationAlgorithm ValidationAlgorithm ValidationPhotograph

Kd 0.91

Ks 0.09

μ 1.85

0.13

“True” values

Algorithm ValidationAlgorithm ValidationAlgorithm ValidationAlgorithm Validation

Renderings

Unknown lighting

Photograph

Known lighting

Kd 0.91 0.89 0.87

Ks 0.09 0.11 0.13

μ 1.85 1.78 1.48

0.13 0.12 .14

“True” values

Image RMS error 5%

Inverse BRDF: SpheresInverse BRDF: SpheresInverse BRDF: SpheresInverse BRDF: Spheres

Bronze Delrin Paint Rough Steel

Photographs

Renderings(Recovered

BRDF)

Complex GeometryComplex GeometryComplex GeometryComplex Geometry

3 photographs of a sculpture• Complex unknown illumination• Geometry known• Estimate microfacet BRDF and distant lighting

ComparisonComparisonComparisonComparison

Photograph Rendering

New View, LightingNew View, LightingNew View, LightingNew View, Lighting

Photograph Rendering

Textured ObjectsTextured ObjectsTextured ObjectsTextured Objects

Photograph Rendering

SummarySummarySummarySummary

• Reflection as convolution

• Signal-processing framework

• Formal study of inverse rendering

• Practical algorithms

• Reflection as convolution

• Signal-processing framework

• Formal study of inverse rendering

• Practical algorithms

Implications and Future WorkImplications and Future WorkImplications and Future WorkImplications and Future Work

• Frequency space analysis of reflection

• Well-posedness of inverse problems• Perception, human vision• Forward rendering [Friday]

• Complex uncontrolled illumination

• Frequency space analysis of reflection

• Well-posedness of inverse problems• Perception, human vision• Forward rendering [Friday]

• Complex uncontrolled illumination

AcknowledgementsAcknowledgementsAcknowledgementsAcknowledgements• Marc Levoy

• Szymon Rusinkiewicz

• Steve Marschner

• John Parissenti, Jean Gleason

• Scanned cat sculpture is “Serenity” by Sue Dawes

• Hodgson-Reed Stanford Graduate Fellowship

• NSF ITR grant #0085864: “Interacting with the Visual World”

Paper Website: http://graphics.stanford.edu/papers/invrend

• Marc Levoy

• Szymon Rusinkiewicz

• Steve Marschner

• John Parissenti, Jean Gleason

• Scanned cat sculpture is “Serenity” by Sue Dawes

• Hodgson-Reed Stanford Graduate Fellowship

• NSF ITR grant #0085864: “Interacting with the Visual World”

Paper Website: http://graphics.stanford.edu/papers/invrend

The EndThe EndThe EndThe End

The EndThe EndThe EndThe End

Related WorkRelated WorkRelated WorkRelated Work

• Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99

• Reflection as frequency-space operator: D’Zmura 91

• Lambertian reflection is convolution: Basri Jacobs 01

Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case

• Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99

• Reflection as frequency-space operator: D’Zmura 91

• Lambertian reflection is convolution: Basri Jacobs 01

Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case

Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)

-1-2 0 1 2

0

1

2

.

.

.

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

Dual RepresentationDual Representation Dual RepresentationDual Representation

Diffuse BRDF: Filter width small in frequency domain

Specular: Filter width small in spatial (angular) domain

Practical Representation: Dual angular, frequency-space

Diffuse BRDF: Filter width small in frequency domain

Specular: Filter width small in spatial (angular) domain

Practical Representation: Dual angular, frequency-space

B Bd

diffuse

Bs,slow

slow specular(area sources)

Bs,fast

fast specular(directional)

= + +

Frequency Angular Space

Inverse LambertianInverse LambertianInverse LambertianInverse LambertianSum l=2 Sum l=4True Lighting

Mirror

Teflon

Other PapersOther PapersOther PapersOther Papers

• Linked to from website for this paper• http://graphics.stanford.edu/papers/invrend/

• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]

• Application to other areas• Forward Rendering (Friday) [SIGGRAPH 01]• Lighting variability object recognition [CVPR 01]

• Linked to from website for this paper• http://graphics.stanford.edu/papers/invrend/

• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]

• Application to other areas• Forward Rendering (Friday) [SIGGRAPH 01]• Lighting variability object recognition [CVPR 01]