Lightfields, Lumigraphs, Lightfields, Lumigraphs, and Other Image-Based and Other Image-Based

MethodsMethods

Image-Based Modeling and Image-Based Modeling and RenderingRendering

• For many applications, re-rendering For many applications, re-rendering is goalis goal

• Traditional vision / graphics pipelines:Traditional vision / graphics pipelines:

• Image-based pipeline:Image-based pipeline:

WorldWorldWorldWorld NewNewImagesImages

NewNewImagesImages

Light sourcesLight sourcesLight sourcesLight sourcesReflectanceReflectanceReflectanceReflectanceGeometryGeometryGeometryGeometry

VisionVision GraphicsGraphics

WorldWorldWorldWorld NewNewImagesImages

NewNewImagesImages

CapturedCapturedImagesImages

CapturedCapturedImagesImages

Plenoptic FunctionPlenoptic Function

• LL((x,y,z,x,y,z,,,,t,,t,))

• Captures all light flow in a sceneCaptures all light flow in a scene

• Enough information to construct any Enough information to construct any image of the scene at any timeimage of the scene at any time

• Practical approximations:Practical approximations:– Static scenes: ignore dependence on Static scenes: ignore dependence on tt– Represent color as RGB: eliminate Represent color as RGB: eliminate – 7D 7D 3 3 5D 5D

Plenoptic Function – Special Plenoptic Function – Special CasesCases

• Sample at one (Sample at one (xx,,yy,,zz):):– LL((,,) is just an (omnidirectional) image) is just an (omnidirectional) image

• Full 5D L(Full 5D L(xx,,yy,,zz,,,,):):– Omnidirectional image at each point in Omnidirectional image at each point in

spacespace– Enough information to reconstruct any Enough information to reconstruct any

viewview

Free SpaceFree Space

• Consider a region of space without Consider a region of space without occlusionocclusion

• Light travels in straight lines Light travels in straight lines some some pixels in different images are the pixels in different images are the same ray of lightsame ray of light

“Rebinning” pixels

General caseGeneral case Free spaceFree space

Light FieldLight Field

• In unoccluded space, can reduce In unoccluded space, can reduce plenoptic function to 4Dplenoptic function to 4D

• 2D array of 2D images2D array of 2D images

• Still contains enough information to Still contains enough information to reconstruct new viewsreconstruct new views

Image-Based Modeling and Image-Based Modeling and RenderingRendering

• Generate new views of a scene Generate new views of a scene directly from existing viewsdirectly from existing views

• ““Pure” IBR (such as lightfields): no Pure” IBR (such as lightfields): no geometric model of scenegeometric model of scene

• Other IBR techniques try to obtain Other IBR techniques try to obtain higher quality with less storage by higher quality with less storage by building a modelbuilding a model

LightfieldsLightfields

• Advantages:Advantages:– Simpler computation vs. traditional CGSimpler computation vs. traditional CG– Cost independent of scene complexityCost independent of scene complexity– Cost independent of material Cost independent of material

properties and other optical effectsproperties and other optical effects– Avoid hard vision problemsAvoid hard vision problems

• Disadvantages:Disadvantages:– Static geometryStatic geometry– Fixed lightingFixed lighting– High storage costHigh storage cost

Using LightfieldsUsing Lightfields

• Obtain 2D slices of 4D data setObtain 2D slices of 4D data set

• Arbitrary views: take other 2D slicesArbitrary views: take other 2D slices

• Challenges:Challenges:– CaptureCapture– ParameterizationParameterization– CompressionCompression– RenderingRendering

Capturing LightfieldsCapturing Lightfields

• Need a 2D set of (2D) imagesNeed a 2D set of (2D) images

• Choices:Choices:– Camera motion: human vs. computerCamera motion: human vs. computer– Constraints on camera motionConstraints on camera motion– Coverage and sampling uniformityCoverage and sampling uniformity– AliasingAliasing

• Point / anglePoint / angle

• Two points on a sphereTwo points on a sphere

• Points on two planesPoints on two planes

• Original images and camera positionsOriginal images and camera positions

Lightfield ParameterizationLightfield Parameterization

CompressionCompression

• Compress individual images (JPEG, Compress individual images (JPEG, etc.)etc.)

• Adapt video compression to 2D Adapt video compression to 2D arraysarrays

• Decomposition into basis functionsDecomposition into basis functions

• Vector quantizationVector quantization

RenderingRendering

• How to select rays?How to select rays?

• InterpolationInterpolation

• Taking advantage of hardwareTaking advantage of hardware– Graphics hardwareGraphics hardware– Compression hardwareCompression hardware

ImplementationsImplementations

• Lightfields, Levoy and Hanrahan Lightfields, Levoy and Hanrahan ((SIGGRAPH 96SIGGRAPH 96))

• Lumigraphs, Gortler et al. (Lumigraphs, Gortler et al. (SIGGRAPH 96SIGGRAPH 96))

• Unstructured lumigraphs, Buehler et Unstructured lumigraphs, Buehler et al. (al. (SIGGRAPH 01SIGGRAPH 01))

Light Field RenderingLight Field Rendering

• Capture:Capture:– Computer-controlled camera rigComputer-controlled camera rig– Move camera to grid of locations on a Move camera to grid of locations on a

planeplane

Light Field RenderingLight Field Rendering

• Parameterization:Parameterization:– Two planes, evenly sampled: “light Two planes, evenly sampled: “light

slab”slab”– In general, planes in arbitrary In general, planes in arbitrary

orientationsorientations– In practice, one plane = camera In practice, one plane = camera

locationslocations• Minimizes resamplingMinimizes resampling

Light Field CoverageLight Field Coverage

Multi-Slab Light FieldsMulti-Slab Light Fields

RenderingRendering

• For each desired ray:For each desired ray:– Compute intersection with (u,v) andCompute intersection with (u,v) and

(s,t) planes(s,t) planes– Take closest rayTake closest ray

• Variants: interpolationVariants: interpolation– Bilinear in (u,v) onlyBilinear in (u,v) only– Bilinear in (s,t) onlyBilinear in (s,t) only– Quadrilinear in (u,v,s,t)Quadrilinear in (u,v,s,t)

Light Field CompressionLight Field Compression

• Based on vector quantizationBased on vector quantization

• Preprocessing: build a representative Preprocessing: build a representative codebook of 4D tilescodebook of 4D tiles

• Each tile in lightfield represented by Each tile in lightfield represented by indexindex

• Example: 2x2x2x2 tiles, 16 bit index Example: 2x2x2x2 tiles, 16 bit index = 24:1 compression= 24:1 compression

The LumigraphThe Lumigraph

• Capture: move camera by handCapture: move camera by hand

• Camera intrinsics assumed calibratedCamera intrinsics assumed calibrated

• Camera pose recovered from Camera pose recovered from markersmarkers

Lumigraph PostprocessingLumigraph Postprocessing

• Obtain rough geometric modelObtain rough geometric model– Chroma keying (blue screen) to extract Chroma keying (blue screen) to extract

silhouettessilhouettes– Octree-based space carvingOctree-based space carving

• Resample images to two-plane Resample images to two-plane parameterizationparameterization

Lumigraph RenderingLumigraph Rendering

• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality

Lumigraph RenderingLumigraph Rendering

• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality

Lumigraph RenderingLumigraph Rendering

Without usinggeometry

Using approximategeometry

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

• Further enhancement of lumigraphs:Further enhancement of lumigraphs:do not use two-plane do not use two-plane parameterizationparameterization

• Store original pictures: no resamplingStore original pictures: no resampling

• Hand-held camera, moved around an Hand-held camera, moved around an environmentenvironment

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

• To reconstruct views, assign penalty To reconstruct views, assign penalty to each original rayto each original ray– Distance to desired ray, usingDistance to desired ray, using

approximate geometryapproximate geometry– ResolutionResolution– Feather near edges of imageFeather near edges of image

• Construct “camera blending field”Construct “camera blending field”

• Render using texture mappingRender using texture mapping

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

Blending field Rendering

Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices

• Spherical motionSpherical motionof camera aroundof camera aroundan objectan object

• Samples space ofSamples space ofdirections uniformlydirections uniformly

• Second arm toSecond arm tomove light source –move light source –measure reflectancemeasure reflectance

Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices

• Acquire an entireAcquire an entirelight field at oncelight field at once

• Video ratesVideo rates

• Integrated MPEG2Integrated MPEG2compression forcompression foreach cameraeach camera

(Bennett Wilburn, Michal Smulski, Mark Horowitz)(Bennett Wilburn, Michal Smulski, Mark Horowitz)