lightfields, lumigraphs, and image-based rendering

27
Lightfields, Lightfields, Lumigraphs, and Image- Lumigraphs, and Image- based Rendering based Rendering

Upload: edmund-pierce

Post on 17-Dec-2015

226 views

Category:

Documents


0 download

TRANSCRIPT

Lightfields, Lumigraphs, Lightfields, Lumigraphs, and Image-based and Image-based

RenderingRendering

2D to 4D2D to 4D

• More than can be captured in one More than can be captured in one imageimage

• All light passing through a region All light passing through a region of spaceof space

• General (5D) vs. unoccluded space General (5D) vs. unoccluded space (4D)(4D)

“Rebinning” pixels

Image-Based Modeling and Image-Based Modeling and RenderingRendering

• Generate new views of a scene Generate new views of a scene directly from existing viewsdirectly from existing views

• ““Pure” IBR (such as lightfields): no Pure” IBR (such as lightfields): no model of scenemodel of scene

• Other IBR techniques try to obtain Other IBR techniques try to obtain higher quality with less storage by higher quality with less storage by building a modelbuilding a model

LightfieldsLightfields

• Advantages:Advantages:– Simpler computation vs. traditional CGSimpler computation vs. traditional CG

– Cost independent of scene complexityCost independent of scene complexity

– Cost independent of material Cost independent of material properties and other optical effectsproperties and other optical effects

• Disadvantages:Disadvantages:– Static geometryStatic geometry

– Fixed lightingFixed lighting

– High storage costHigh storage cost

Using LightfieldsUsing Lightfields

• Obtain 2D slices of 4D data setObtain 2D slices of 4D data set

• Arbitrary views: take other 2D Arbitrary views: take other 2D slicesslices

• Challenges:Challenges:– CaptureCapture

– ParameterizationParameterization

– CompressionCompression

– RenderingRendering

Capturing LightfieldsCapturing Lightfields

• Need a 2D set of (2D) imagesNeed a 2D set of (2D) images

• Choices:Choices:– Camera motion: human vs. computerCamera motion: human vs. computer

– Constraints on camera motionConstraints on camera motion

– Coverage and sampling uniformityCoverage and sampling uniformity

– AliasingAliasing

• Point / anglePoint / angle

• Two points on a sphereTwo points on a sphere

• Points on two planesPoints on two planes

• Original images and camera Original images and camera positionspositions

Lightfield ParameterizationLightfield Parameterization

CompressionCompression

• Compress individual images (JPEG, Compress individual images (JPEG, etc.)etc.)

• Adapt video compression to 2D Adapt video compression to 2D arraysarrays

• Decomposition into basis functionsDecomposition into basis functions

• Vector quantizationVector quantization

RenderingRendering

• How to select rays?How to select rays?

• InterpolationInterpolation

• Taking advantage of hardwareTaking advantage of hardware– Graphics hardwareGraphics hardware

– Compression hardwareCompression hardware

ImplementationsImplementations

• Lightfields, SIGGRAPH 96,Lightfields, SIGGRAPH 96,Levoy and HanrahanLevoy and Hanrahan

• Lumigraphs, SIGGRAPH 96,Lumigraphs, SIGGRAPH 96,Gortler et al.Gortler et al.

• Unstructured lumigraphs, Unstructured lumigraphs, SIGGRAPH 01, Buehler et al.SIGGRAPH 01, Buehler et al.

Light Field RenderingLight Field Rendering

• Capture:Capture:– Computer-controlled camera rigComputer-controlled camera rig

– Move camera to grid of locations on a Move camera to grid of locations on a planeplane

Light Field RenderingLight Field Rendering

• Parameterization:Parameterization:– Two planes, evenly sampled: “light Two planes, evenly sampled: “light

slab”slab”

– In general, planes in arbitrary In general, planes in arbitrary orientationsorientations

– In practice, one plane = camera In practice, one plane = camera locationslocations• Minimizes resamplingMinimizes resampling

Light Field CoverageLight Field Coverage

Multi-Slab Light FieldsMulti-Slab Light Fields

RenderingRendering

• For each desired ray:For each desired ray:– Compute intersection with (u,v) andCompute intersection with (u,v) and

(s,t) planes(s,t) planes

– Take closest rayTake closest ray

• Variants: interpolationVariants: interpolation– Bilinear in (u,v) onlyBilinear in (u,v) only

– Bilinear in (s,t) onlyBilinear in (s,t) only

– Quadrilinear in (u,v,s,t)Quadrilinear in (u,v,s,t)

Light Field CompressionLight Field Compression

• Based on vector quantizationBased on vector quantization

• Preprocessing: build a Preprocessing: build a representative codebook of 4D representative codebook of 4D tilestiles

• Each tile in lightfield represented Each tile in lightfield represented by indexby index

• Example: 2x2x2x2 tiles, 16 bit Example: 2x2x2x2 tiles, 16 bit index = 24:1 compressionindex = 24:1 compression

Lightfield Compression byLightfield Compression byVector QuantizationVector Quantization

• Advantages:Advantages:– Fast at runtimeFast at runtime

– Reasonable compressionReasonable compression

• Disadvantages:Disadvantages:– Preprocessing slowPreprocessing slow

– Manually-selected codebook sizeManually-selected codebook size

– Does not take advantage of structureDoes not take advantage of structure

The LumigraphThe Lumigraph

• Capture: move camera by handCapture: move camera by hand

• Camera intrinsics assumed Camera intrinsics assumed calibratedcalibrated

• Camera pose recovered from Camera pose recovered from markersmarkers

Lumigraph PostprocessingLumigraph Postprocessing

• Obtain rough geometric modelObtain rough geometric model– Chroma keying (blue screen) to Chroma keying (blue screen) to

extract silhouettesextract silhouettes

– Octree-based space carvingOctree-based space carving

• Resample images to two-plane Resample images to two-plane parameterizationparameterization

Lumigraph RenderingLumigraph Rendering

• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality

Lumigraph RenderingLumigraph Rendering

• Use rough depth information to Use rough depth information to improve rendering qualityimprove rendering quality

Lumigraph RenderingLumigraph Rendering

Without usinggeometry

Using approximategeometry

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

• Further enhancement of Further enhancement of lumigraphs:lumigraphs:do not use two-plane do not use two-plane parameterizationparameterization

• Store original pictures: no Store original pictures: no resamplingresampling

• Hand-held camera, moved around Hand-held camera, moved around an environmentan environment

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

• To reconstruct views, assign To reconstruct views, assign penalty to each original raypenalty to each original ray– Distance to desired ray, usingDistance to desired ray, using

approximate geometryapproximate geometry

– ResolutionResolution

– Feather near edges of imageFeather near edges of image

• Construct “camera blending field”Construct “camera blending field”

• Render using texture mappingRender using texture mapping

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

Blending field Rendering

Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices

• Spherical motionSpherical motionof camera aroundof camera aroundan objectan object

• Samples space ofSamples space ofdirections uniformlydirections uniformly

• Second arm toSecond arm tomove light source –move light source –measure BRDFmeasure BRDF

Other Lightfield Acquisition Other Lightfield Acquisition DevicesDevices

• Acquire an entireAcquire an entirelight field at oncelight field at once

• Video ratesVideo rates

• Integrated MPEG2Integrated MPEG2compression forcompression foreach cameraeach camera

(Bennett Wilburn, Michal Smulski, Mark Horowitz)