Rendering Synthetic Rendering Synthetic Objects into Real Objects into Real Scenes:Scenes:Bridging Traditional and Image-Bridging Traditional and Image-based Graphics with Global based Graphics with Global Illumination and High Dynamic Illumination and High Dynamic Range PhotographyRange PhotographyPaul Debevec, University of California at Berkley

MotivationMotivation

Visual EffectsVisual Effects– Adding synthetic actors/propsAdding synthetic actors/props

Architectural simulationArchitectural simulation– Adding buildings to landscapesAdding buildings to landscapes

Related WorkRelated Work

Previous methods used to insert Previous methods used to insert synthetic objects into real scenessynthetic objects into real scenes– DrawingDrawing– Environment mappingEnvironment mapping– Approximate geometric model and Approximate geometric model and

lightinglighting Global illumination algorithmsGlobal illumination algorithms Recovery of HDR radiance maps Recovery of HDR radiance maps

from photographsfrom photographs

Related WorkRelated Work

High Dynamic Range refers to the High Dynamic Range refers to the range of brightness values present in a range of brightness values present in a scenescene

Eyes, photos, displays, all capture a Eyes, photos, displays, all capture a limited, non-linear rangelimited, non-linear range

Everyday brightness range (about 1012:1)

1000:1 100:1

Source: Microsoft Developers Network

Related WorkRelated Work

Why is this a Why is this a problem?problem?

Limited brightness Limited brightness range results in lost range results in lost scene detail from scene detail from saturation and/or saturation and/or underexposureunderexposure

OK for traditional OK for traditional image useimage use

Photos from OpenEXR by LucasFilm, LTD. http://www.openexr.com/samples.html

Related WorkRelated Work

What about image-based lighting?What about image-based lighting?– Photo pixel values are not Photo pixel values are not

representative of relative Radiance in representative of relative Radiance in the scenethe scene

Images by Paul Debevec, 98

Related WorkRelated Work

Solution: Take multiple photos with Solution: Take multiple photos with varying shutter speeds and ‘combine’ varying shutter speeds and ‘combine’ somehow to create accurate relative somehow to create accurate relative radiance mapsradiance maps

Images by Paul Debevec, 98

Related WorkRelated Work

Is HDR really important?Is HDR really important?

MethodMethod

Divide the scene into three partsDivide the scene into three parts

MethodMethod

Distant SceneDistant Scene– Light-based modelLight-based model– Required to provide correct incident Required to provide correct incident

illumination to local/synthetic illumination to local/synthetic objects from desired viewpointobjects from desired viewpoint

– Assumed that no light from the Assumed that no light from the model will affect the distant scenemodel will affect the distant scene

– Any level of geometric detail wantedAny level of geometric detail wanted

MethodMethod

Local SceneLocal Scene– Material-based model (has BRDF)Material-based model (has BRDF)– Must approximate the real scene local Must approximate the real scene local

to the position of the synthetic to the position of the synthetic objectsobjects

– ‘‘Local’ means photometrically localLocal’ means photometrically local– ‘‘Approximate’ means geometrically Approximate’ means geometrically

AND photometrically representative AND photometrically representative of the real sceneof the real scene

MethodMethod

Synthetic ObjectsSynthetic Objects– Complete materials-based models of Complete materials-based models of

elements that are to be inserted into elements that are to be inserted into the scenethe scene

– Geometric and lighting informationGeometric and lighting information– Any shape and material supported Any shape and material supported

by the global illumination method by the global illumination method that you plan to usethat you plan to use

MethodMethod

Compositing using Compositing using a light probea light probe– One method to One method to

create the distant create the distant scene from a real scene from a real locationlocation

– Uses a metal Uses a metal sphere and a sphere and a camera to gather camera to gather light from a pointlight from a point

MethodMethod

Compositing using Compositing using a light probea light probe– This light is This light is

mapped onto the mapped onto the distant scene distant scene geometry and geometry and used to calculate used to calculate the lightingthe lighting

– Lighting is Lighting is calculated with calculated with global illuminationglobal illumination

MethodMethod

ResultsResults

MethodMethod

Problem with the current Problem with the current definition of the local scenedefinition of the local scene– We said that it must approximate We said that it must approximate

the material of the real scenethe material of the real scene– It can be really hard to get the It can be really hard to get the

BRDF/material of a real object!BRDF/material of a real object!– Can we modify the local scene so it Can we modify the local scene so it

doesn’t require this?doesn’t require this?

MethodMethod

Local Scene 2 (Differential Local Scene 2 (Differential Rendering)Rendering)– Approximate the material of the Approximate the material of the

scene to within reason (not exact)scene to within reason (not exact)– Render with and without your Render with and without your

synthetic objects and calculate the synthetic objects and calculate the differencedifference

– Add that difference back into the Add that difference back into the original imageoriginal image

MethodMethod

Difference between local scene render with Difference between local scene render with and without synthetic objectsand without synthetic objects

-

=

ResultsResults

ConclusionConclusion

This paper presented a framework This paper presented a framework for rendering synthetic objects into for rendering synthetic objects into real world environments with real real world environments with real lightinglighting– High Dynamic Range radiance mapsHigh Dynamic Range radiance maps– Global IlluminationGlobal Illumination– Distant and local scenes, synthetic Distant and local scenes, synthetic

objectsobjects– Differential renderingDifferential rendering

Future WorkFuture Work

Method to generate boundary Method to generate boundary between distant and local scenebetween distant and local scene

More efficient global illuminationMore efficient global illumination

CommentsComments

ProsPros– Open frameworkOpen framework– Good resultsGood results

ConsCons– Hard to decide what local scene Hard to decide what local scene

should be!should be!– Local scene can be very complicatedLocal scene can be very complicated