computational fundamentals of reflection coms 6998-3, lecture 7 0 01 2

Computational Fundamentals of Computational Fundamentals of ReflectionReflection

COMS 6998-3, Lecture 7

lA

2 / 3

/ 4

0l0 1 2

MotivationMotivation

Understand intrinsic computational structure of reflection and illumination

Necessary for many applications in computer graphics (cannot solve by brute force!!)• Real-time forward rendering• IBR sampling rates, dimensionality explosion• Inverse rendering and inverse problems in general• Computer vision: complex lighting, materials

Real-Time Rendering DemoReal-Time Rendering Demo

Motivation: Interactive rendering with complex natural illumination and realistic, measured BRDFs

QuestionsQuestions

• Images are view-dependent (4D quantity)

• Can we find low-dimensional structure to capture view-dependence?

Space of images as lighting variesSpace of images as lighting varies

Illuminate subject from many incident directions

Example ImagesExample Images

Images from Debevec et al. 00

Principal Component AnalysisPrincipal Component Analysis• Try to approximate with low dimensional subspace

• Linear combination of few principal components

= .5 + .5 + …

= .7 + .3 + …

Principal component imagesPrincipal component images

Lighting VariabilityLighting Variability

TheoryInfinite number of light directions, one coefficient/directionSpace of images infinite dimensional [Belhumeur 98]

Empirical [Hallinan 94, Epstein 95]Diffuse objects: 5D subspace suffices

No satisfactory theoretical explanation of observations

Complex Light TransportComplex Light Transport

• Shadows high frequency

• Analysis possible?

• Low-dim. structure?

• Real-time complex lights?

Agrawala, Ramamoorthi, Heirich, Moll SIGGRAPH 00

ChallengesChallenges

• Illumination complexity

• Material (BRDF)/view complexity

• Transport complexity (shadows, interreflection)

Fundamental questions• Theoretical analysis of intrinsic complexity• Sampling rates and resolutions• Efficient practical algorithms

OutlineOutline

• Lighting variability in appearance [PAMI Oct, 2002]

• View variability real-time rendering [SIGGRAPH 02]

• Visibility/shadows [In Progress]

Lighting variability analysisLighting variability analysis

1. Frequency space analytic PCA construction

2. Mathematical derivation of principal components

3. Explain empirical results quantitatively• Dimensionality of approximating subspace• Forms of principal components• Relative importance of principal components

AssumptionsAssumptions

• Single view of single object

• Lambertian

• Distant illumination

• Discount texture

• Discount concavities: interreflection, cast shadows

Consider attached shadows (backfacing normals)

DefinitionsDefinitions

max( ,0) max(cos ,0)E L N Irradiance(image)

Radiance(light)

Normal Lambertian = half-cosine

Previous Theoretical WorkPrevious Theoretical Work

• Discount attached shadows [Shashua 97, …]

Resulting 3D subspace does not fully explain data

• Analytic PCA (without shadows) [Zhao & Yang 99]

cosE L N

max( ,0) max(cos ,0)E L N Irradiance(image)

Radiance(light)

Normal Lambertian = half-cosine

Spherical HarmonicsSpherical Harmonics

-1-2 0 1 2

0

1

2

.

.

.

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Spherical Harmonic ExpansionSpherical Harmonic Expansion

Expand lighting (L), irradiance (E) in basis functions

0

( , ) ( , )l

lm lml m l

L L Y

0

( , ) ( , )l

lm lml m l

E E Y

= .67 + .36 + …

Lambertian BRDF Expansion Lambertian BRDF Expansion

10

0

max(cos ,0) ( )l l ll

AY

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Lambertian coefficients

0.89 1.02 0.50z 23 1z 1

Analytic Irradiance FormulaAnalytic Irradiance Formula

Lambertian surface acts like low-pass filter

lm l lmE A LlA

2 / 3

/ 4

0

2 1

2

2

( 1) !2

( 2)( 1) 2 !

l

l l l

lA l even

l l

l0 1 2

Basri & Jacobs 01Ramamoorthi & Hanrahan 01

9 Parameter Approximation9 Parameter Approximation

Exact imageOrder 29 terms

RMS Error = 1%

For any illumination, average error < 2% [Basri Jacobs 01]

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Open QuestionsOpen Questions

• Relationship between spherical harmonics, PCA

• 9D approximation > 5D empirical subspace

Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere

Intuition: Backwards Half-CosineIntuition: Backwards Half-Cosine

00 10 20max( cos ,0) 0.89 1.02 0.50Y Y Y

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Front 0

1 parameter irrelevant8 or fewer params

enough

Back cos

Intuition: dimensionality reductionIntuition: dimensionality reduction

Start with 9D space, remove dimensions• Mean (constant term) subtracted • Backwards half-cosine• x, xz very similar • y, yz very similar

Left with 5D subspace

11 21( , )Y Y

1 1 2 1( , )Y Y

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Results: Image of a SphereResults: Image of a Sphere

• Principal components (eigenvectors) mix (linear combinations of) spherical harmonics

• Results agree with experiment [Epstein 95]• We predict: 3 eigenvectors = 91% variance, 5 give 96%• Empirical : 3 eigenvectors = 94% variance, 5 give 98%

% VAF(eigenvalue)

43% 24% 24% 2% 2%

Results: Human FaceResults: Human Face

• Numerically compute orthogonality matrix

• Specific distribution of surface normals important• Symmetries in sphere broken (faces are elongated)• Principal components somewhat different from sphere

% VAF 42% 33% 16% 4% 2%

Results: Human FaceResults: Human Face

• Prediction: Principal components have specific forms

• Empirical : [Hallinan 94]Frontal lighting, side, above/below, extreme side, corner

Frontal% VAF 42% 33% 16% 4% 2%

Side Above/Below Extreme side Corner

Results: Human FaceResults: Human Face

• Prediction: Space is close to 5D3 principal components = 91% variance, 5 components =

97%

• Empirical : [Epstein 95]3 principal components = 90% variance, 5 components =

94%

Frontal% VAF 42% 33% 16% 4% 2%

Side Above/Below Extreme side Corner

Results: Human FaceResults: Human Face

• Prediction: groups of principal components• Group 1: first two (frontal and side)• Group 2: next three [with above/below always 3rd]

• Empirical: [Hallinan 94]• Two groups [first two (frontal,side) and next three]• Within group, %VAF close, may exchange places

Frontal% VAF 42% 33% 16% 4% 2%

Side Above/Below Extreme side Corner

Summary: Lighting AnalysisSummary: Lighting Analysis

• Analytic PCA construction with attached shadowsSpherical harmonic analysis: Orthogonality matrix

• Mathematically derive principal components

• Qualitative, quantitative agreement with experiment

• Extend 9D Lambertian model to single view case

Implications Lighting AnalysisImplications Lighting Analysis

• Attached shadows nearly free: 5D subspace enough

• Mathematical derivation of principal components• Basis functions for subspace methods for recognition,…• Graphics applications: Image-Based, inverse rendering

• Complex illumination in computer vision

OutlineOutline

• Lighting variability in appearance [PAMI Oct, 2002]

• View variability real-time rendering [SIGGRAPH 02]

• Visibility/shadows [In Progress]

Reflection EquationReflection Equation

( )L R N l

2D Environment Map

Reflection EquationReflection Equation

( )L R N l

2D Environment Map

NL

,l V

BRDF

Reflection EquationReflection Equation

4D Orientation Light Field

2D Environment Map

Previous Work: Blinn & Newell 76, Miller & Hoffman 84,

Greene 86, Kautz & McCool 99, Cabral et al. 99, …

( , ) ( ) ,B N V L R N l l V dl

BRDF

NL

GoalsGoals

• Efficiently precompute and represent OLF

• Real-time rendering with OLF

QuestionsQuestions

• Parameterization and structure of OLF

• Structure leads to representation

• Computation and rendering of OLF

OLF ParameterizationOLF Parameterization

N LN

V

( , )B N V

OLF ParameterizationOLF Parameterization

N

V

( , )B N V

N

V

( , )B R V

RReparameterize

by reflection vector

OLF ParameterizationOLF Parameterization

• Captures structure of BRDF (and hence OLF) better

• Reflective BRDFs become low-dimensional

N

V

( , )B N V

N

V

( , )B R V

RReparameterize

by reflection vector

OLF StructureOLF Structure

( , )B R V

( )VB R

( )RB V

2D view array of reflection maps

2D image arrayof view maps

OLF Structure: PhongOLF Structure: Phong

( , ) ( )B R V B R

( )VB R

( )RB V

2D view array of reflection maps

2D image arrayof view maps

Environment Map Phong Reflection Map(blurred environment map)

Same reflection map for all views

OLF Structure: PhongOLF Structure: Phong

( , ) ( )B R V B R

( )VB R

( )RB V

Same reflection map for all views

Viewx

Vie

wy

OLF Structure: PhongOLF Structure: Phong

( , ) ( )B R V B R

( )VB R

( )RB V

Same reflection map for all views View maps constant for each R

Viewx

Vie

wy

OLF Structure: PhongOLF Structure: Phong

( , ) ( )B R V B R

( )VB R

( )RB V

Same reflection map for all views View maps constant for each R

Viewx

Vie

wy

Reflectionx

Ref

lect

ion

y

OLF Structure: LafortuneOLF Structure: Lafortune

( )VB R

Vie

wy

• Single 2D reflection map no longer sufficient

• But variation with viewing direction is slow

Viewx

OLF Structure: LafortuneOLF Structure: Lafortune

( )VB R

( )RB V

View maps vary slowly

Reflectionx

Ref

lect

ion

y

Vie

wy

Viewx

A Simple FactorizationA Simple Factorization

( )VB R

( )RB V

Viewx

Vie

wy

Ref

lect

ion

yReflectionx

( , ) ( ) * ( )B R V f R g V

*

Spherical Harmonic Reflection MapSpherical Harmonic Reflection Map

• View-dependent reflection (cube)map

• Encode view maps with low-order spherical harmonics

( )RB V

PrefilteringPrefiltering

• Directly compute SHRM from Lighting, BRDF

• Convolution easier to compute in frequency domain

,,() iijijiiji LLBLBR

Input Lighting and BRDF

Spherical Harmonic coeffs.

Convolution SHRM

PrefilteringPrefiltering

• 3 to 4 orders of magnitude faster (< 1 s compared to minutes or hours)

• Detailed analysis, algorithms, experiments in paper

,,() iijijiiji LLBLBR

Input Lighting and BRDF

Spherical Harmonic coeffs.

Convolution SHRM

Number of terms: CURETNumber of terms: CURET

Analysis for all 61 samples [full bar chart in paper]• For essentially all materials, 9-16 terms in SHRM suffice

DemoDemo

Summary view variabilitySummary view variability

• Theoretical, empirical analysis of sampling rates and resolutions• Frequency space analysis directly on lighting, BRDF• Low order expansion suffices for essentially all BRDFs

• Spherical Harmonic Reflection Maps• Hybrid angular-frequency space• Compact, efficient, accurate• Easy to analyze errors, determine number of terms

• Fast computation using convolution

ImplicationsImplications

• Frequency space methods for rendering • Global illumination• Fast computation of surface light fields

• Compression for optimal factored representations• PCA on SHRMs

• Theoretical analysis of sampling rates, resolutions• General framework for sampling in image-based rendering

OutlineOutline

• Lighting variability in appearance [PAMI Oct, 2002]

• View variability real-time rendering [SIGGRAPH 02]

• Visibility/shadows [In Progress]

Visibility complexity (high freq)Visibility complexity (high freq)

But Sparse (< 4%)But Sparse (< 4%)

Questions on VisibilityQuestions on Visibility

• Theory• Locally low-dimensional subspaces?• Intrinsic complexity of binary function?

• Practical• Real-time rendering with complex soft shadows, changing

illumination for lighting design, simulation• Efficient encoding/decoding (wavelets, PCA, dictionaries,

hierarchical?)

• In progress….

Overall SummaryOverall Summary

Many applications in graphics cannot be solved by brute force• Real-time rendering• IBR sampling rates, dimensionality explosion• Inverse rendering, inverse problems• Computer vision: complex lighting, materials

Need fundamental understanding of nature of reflection/lighting• Illumination complexity• Material (view) complexity• Transport complexity

Overall SummaryOverall Summary

Theoretical analysis tools• Signal processing, sampling theory• Low-dimensional subspaces• Information theory, information-based complexity?

Practical algorithms• Real-time rendering with complex lights,view, transport?• Lighting, Material design?• Exploit theoretical analysis (sampling rates,

forward/inverse duality, angular/frequency/sparsity duality, subspace results, differential analysis, perception)