a signal-processing framework for forward and inverse rendering coms 6998-3, lecture 8

A Signal-Processing Framework A Signal-Processing Framework for Forward and Inverse Renderingfor Forward and Inverse Rendering

COMS 6998-3, Lecture 8

Illumination IllusionIllumination Illusion

People perceive materials more easily under natural illumination than simplified illumination.

Images courtesy Ron Dror and Ted Adelson

Illumination IllusionIllumination Illusion

People perceive materials more easily under natural illumination than simplified illumination.

Images courtesy Ron Dror and Ted Adelson

Material RecognitionMaterial Recognition

Photographs of 4 spheres in 3 different

lighting conditions

courtesy Dror and Adelson

Estimating BRDF and LightingEstimating BRDF and Lighting

Photographs

Geometric model

InverseRenderingAlgorithm

Lighting

BRDF

Estimating BRDF and LightingEstimating BRDF and Lighting

Photographs

Geometric model

ForwardRenderingAlgorithm

Rendering

Lighting

BRDF

Estimating BRDF and LightingEstimating BRDF and Lighting

Photographs

Geometric model

ForwardRenderingAlgorithm

Rendering

Novel lighting

BRDF

Inverse Problems: Difficulties Inverse Problems: Difficulties

Angular width of Light Source

Su

rfac

e ro

ugh

nes

sIll-posed

(ambiguous)

MotivationMotivation

Understand nature of reflection and illumination

Applications in computer graphics• Real-time forward rendering• Inverse rendering

Contributions of ThesisContributions of Thesis

1. Formalize reflection as convolution

2. Signal-processing framework

3. Practical forward and inverse algorithms

OutlineOutline

• Motivation

• Reflection as Convolution• Preliminaries, assumptions• Reflection equation, Fourier analysis (2D)• Spherical Harmonic Analysis (3D)

• Signal-Processing Framework

• Applications

• Summary and Implications

AssumptionsAssumptions• Known geometry

Laser range scanner

Structured light

AssumptionsAssumptions

• Known geometry

• Convex curved surfaces: no shadows, interreflection

Complex geometry: use surface normal

AssumptionsAssumptions

• Known geometry

• Convex curved surfaces: no shadows, interreflection

• Distant illumination

Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec

AssumptionsAssumptions

• Known geometry

• Convex curved surfaces: no shadows, interreflection

• Distant illumination

• Homogeneous isotropic materials

Isotropic Anisotropic

AssumptionsAssumptions

• Known geometry

• Convex curved surfaces: no shadows, interreflection

• Distant illumination

• Homogeneous isotropic materials

Later, practical algorithms: relax some assumptions

ReflectionReflection

2

2

id( )iL Lighting

L

i

( )oB Reflected Light Field

( , )i o BRDF

B

o

Reflection as Convolution (2D)Reflection as Convolution (2D)

2

2

id( )iL Lighting

( )oB Reflected Light Field

( , )i o BRDF

Li o B

i

B

L L

o

Reflection as Convolution (2D)Reflection as Convolution (2D)

2

2

id( )iL Lighting

( )oB Reflected Light Field

( , )i o BRDF

( )iL ( , )i o id2

2

( , )oB

Li o B

L

i

Bo

Reflection as Convolution (2D)Reflection as Convolution (2D)

( )iL ( , )i o id2

2

( , )oB

Li o B

L

i

Bo

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u

1 1( ) ( ) ( )h u g x u f x dx

u1

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u

u2

2 2( ) ( ) ( )h u g x u f x dx

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u

u3

3 3( ) ( ) ( )h u g x u f x dx

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u

( ) ( ) ( )h u g x u f x dx

h f g g f Fourier analysis

h f g

Reflection as Convolution (2D)Reflection as Convolution (2D)

B L

Frequency: product

Spatial: integral

( )iL ( , )i o id2

2

( , )oB

, ,2l p l l pB L Fourier analysis

R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208

L

i

Bo

Li o B

Related WorkRelated Work

• Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99

• Reflection as frequency-space operator: D’Zmura 91

• Lambertian reflection is convolution: Basri Jacobs 01

Our Contributions• Explicitly derive frequency-space convolution formula• Formal quantitative analysis in general 3D case

Spherical HarmonicsSpherical Harmonics

-1-2 0 1 2

0

1

2

.

.

.

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Spherical Harmonic AnalysisSpherical Harmonic Analysis

, ,2l p l l pB L

2D:

22

0 0 ,( , , , ) ( [ , ]) ( , , , )o o i i i i o o i iB L R d d

3D:

( )iL ( , )i o id2

2

( , )oB

, ,lm pq l lm lq pqB L

OutlineOutline

• Motivation

• Reflection as Convolution

• Signal-Processing Framework• Insights, examples• Well-posedness of inverse problems

• Applications

• Summary and Implications

Insights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Insights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Filter is Delta function : Output = Signal

Mirror BRDF : Image = Lighting [Miller and Hoffman 84]

Image courtesy Paul Debevec

Insights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution

Signal is Delta function : Output = Filter

Point Light Source : Images = BRDF [Marschner et al. 00]

Am

plit

ude

Frequency

Roughness

Phong, Microfacet ModelsPhong, Microfacet Models

Mirror

Illumination estimationill-posed for rough surfaces

Analytic formulae in R. Ramamoorthi and P. Hanrahan“A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128

LambertianLambertian

Incident radiance (mirror sphere)

Irradiance (Lambertian)

N

21

2

2

(1)!2

(2)(1)2!

l

lll

lleven

ll

l

2 / 3

/ 4

0

l0 1 2

R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459

R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390

Inverse LightingInverse Lighting

Well-posed unless denominator vanishes• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors low pass filters: Inverse lighting ill-

posed

,

,

1 lm pqlm

l lq pq

BL

, ,lm pq l lm lq pq

B L

B L

Given: B,ρ find L

Inverse BRDFInverse BRDF

Well-posed unless Llm vanishes• Lighting should have sharp features (point sources, edges)• BRDF estimation ill-conditioned for soft lighting

,,

1 lm pqlq pq

l lm

B

L

Directional

Source

Area source

Same BRDF

Given: B,L find ρ

Factoring the Light FieldFactoring the Light Field

Light Field can be factored• Up to global scale factor• Assumes reciprocity of BRDF• Can be ill-conditioned• Analytic formula derived

Given: B find L and ρ

B L

4D 2D 3D

More knowns (4D)than unknowns (2D/3D)

OutlineOutline

• Motivation

• Reflection as Convolution

• Signal-Processing Framework

• Applications• Forward rendering (convolution)• Inverse rendering (deconvolution)

• Summary and Implications

Computing IrradianceComputing Irradiance• Classically, hemispherical integral for each pixel

• Lambertian surface is like low pass filter

• Frequency-space analysis

IncidentRadiance

Irradiance

9 Parameter Approximation9 Parameter Approximation

Exact imageOrder 01 term

(constant)

RMS error = 25 %

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

9 Parameter Approximation9 Parameter Approximation

Exact imageOrder 14 terms(linear)

RMS Error = 8%

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

9 Parameter Approximation9 Parameter Approximation

Exact imageOrder 29 terms

(quadratic)

RMS Error = 1%

For any illumination, average error < 2% [Basri Jacobs 01]

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

ComparisonComparison

Incident illumination

300x300

Irradiance mapTexture: 256x256

HemisphericalIntegration 2Hrs

Irradiance mapTexture: 256x256

Spherical HarmonicCoefficients 1sec

Time 300 300 256 256 Time 9 256 256

Inverse RenderingInverse Rendering

Miller and Hoffman 84

Marschner and Greenberg 97

Sato et al. 97

Dana et al. 99

Debevec et al. 00

Marschner et al. 00

Sato et al. 99

Lighting

BRDF

Known Unknown

Unknown

Known

Textures are a third axis

ContributionsContributions

• Complex illumination

• Factorization of BRDF, lighting (find both)

• New representations and algorithms

• Formal study of inverse problems (well-posed?)

ComplicationsComplications

• Incomplete sparse data (few photographs)

• Concavities: Self Shadowing

• Spatially varying BRDFs

ComplicationsComplications

Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra

Solution:• Use parametric BRDF model• Dual angular and frequency space representation

Algorithm ValidationAlgorithm ValidationPhotograph

Kd 0.91

Ks 0.09

μ 1.85

0.13

“True” values

Algorithm ValidationAlgorithm Validation

Renderings

Unknown lighting

Photograph

Known lighting

Kd 0.91 0.89 0.87

Ks 0.09 0.11 0.13

μ 1.85 1.78 1.48

0.13 0.12 0.14

“True” values

Image RMS error 5%

Inverse BRDF: SpheresInverse BRDF: Spheres

Bronze Delrin Paint Rough Steel

Photographs

Renderings(Recovered

BRDF)

ComplicationsComplications

Challenge: Complex geometry with concavities Self shadowing

Solution:• Use associativity of convolution• Blur lighting, treat specular BRDF term as mirror• Single ray for shadowing, easy in ray tracer

Complex GeometryComplex Geometry

3 photographs of a sculpture• Complex unknown illumination• Geometry known• Estimate microfacet BRDF and distant lighting

ComparisonComparison

Photograph Rendering

New View, LightingNew View, Lighting

Photograph Rendering

ComplicationsComplications

Challenge: Spatially varying BRDFs

Solution:• Use textures to modulate BRDF parameters

Textured ObjectsTextured Objects

Photograph Rendering

SummarySummary

• Reflection as convolution

• Frequency-space analysis gives many insights

• Practical forward and inverse algorithms

• Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision