Image-Based Rendering Image-Based Rendering to Accelerate to Accelerate Interactive Interactive
WalkthroughsWalkthroughs
Lucent TechnologiesBell Labs Innovations
Daniel G. AliagaDaniel G. Aliaga
Visual Communications Visual Communications ResearchResearch
Lucent Technologies Bell LabsLucent Technologies Bell Labs
E2
3D Models3D Models
1.7Mtris
0.9Mtris
2.0Mtris
1.0Mtris
E3
Why Use Images?Why Use Images?
• Independent of scene complexityIndependent of scene complexity
640x480 pixels640x480 pixels 640x480 pixels640x480 pixels
E4
Flight SimulatorsFlight Simulators
• Mid-1980’sMid-1980’s– E&S CT-6 one of first to use real-time E&S CT-6 one of first to use real-time
photo texturesphoto textures
• Hand-selected objects:Hand-selected objects:– Terrain, trees, airplanes, buildings, Terrain, trees, airplanes, buildings,
etc.etc.
• 30-60 Hz30-60 Hz– High visual fidelityHigh visual fidelity
E5
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E6
Replacing Geometry with Replacing Geometry with ImagesImages
• AlgorithmAlgorithm– Select subset of modelSelect subset of model– Create image of the subsetCreate image of the subset– Cull subset and replace with imageCull subset and replace with image
• Why?Why?– Image displayed in (approx.) constant Image displayed in (approx.) constant
timetime– Image reused for several framesImage reused for several frames
E7
Simple ExampleSimple Example
E8
Simple ExampleSimple Example
E9
Simple ExampleSimple Example
E10
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E11
Geometric Geometric DiscontinuityDiscontinuity
• If we move from the center-of-projection, If we move from the center-of-projection, discontinuities appear at the borderdiscontinuities appear at the border
E12
Temporal DiscontinuityTemporal Discontinuity
• While moving, if we switch between While moving, if we switch between geometry and image, a sudden geometry and image, a sudden poppop occurs occurs
E13
ApproachesApproaches
• Geometric and Temporal Geometric and Temporal ContinuityContinuity– Error metricsError metrics– Geometry warpingGeometry warping– Image warpingImage warping– Lightfield/LumigraphLightfield/Lumigraph
E14
Error MetricsError Metrics
• Use an Use an error metricerror metric to control to control amount of discontinuityamount of discontinuity
c1
c2
[Maciel95][Shade96][Schaufler96]
E15
Error MetricError Metric
• Relies on “angular-deviation” Relies on “angular-deviation” measuring the visual quality of measuring the visual quality of using the (same) imageusing the (same) image
E16
Video Segment IVideo Segment I
• Pre-rendered ImpostorsPre-rendered Impostors– Maciel95Maciel95
• Dynamic Image-CachingDynamic Image-Caching– Shade96, Schaufler96Shade96, Schaufler96
E17
Geometry WarpingGeometry Warping
[Aliaga96]
E18
Geometry WarpingGeometry Warping
E19
Geometry WarpingGeometry Warping
E20
Geometry WarpingGeometry Warping
E21
Geometry WarpingGeometry Warping
E22
Geometry WarpingGeometry Warping
E23
Geometry WarpingGeometry Warping
E24
Geometry WarpingGeometry Warping
E25
Geometry WarpingGeometry Warping
Surrounding geometry warpedSurrounding geometry warped(incorrect perspective)(incorrect perspective)
Correct perspectiveCorrect perspective
E26
Geometry WarpingGeometry Warping
• Surrounding geometry warped to Surrounding geometry warped to match imagematch image
Center-of-projection
Viewpoint
SurroundingGeometry
E27
Video Segment IIVideo Segment II
• Geometry WarpingGeometry Warping– Aliaga96Aliaga96
E28
Image WarpingImage Warping
• Change the image itselfChange the image itself– Re-project the image to the current Re-project the image to the current
viewpointviewpoint• [Chen93][McMillan95][Max95][Shade98][Chen93][McMillan95][Max95][Shade98]
– Display image as a (simplified, Display image as a (simplified, textured) meshtextured) mesh• [Darsa97][Sillion97][Darsa97][Sillion97]
E29
Image WarpingImage Warping
• A raster scan of each sheet produces a A raster scan of each sheet produces a back-to-front ordering of warped pixelsback-to-front ordering of warped pixels
eye
Reference COP
E30
Image WarpingImage Warping
• McMillan and Bishop’s Warping McMillan and Bishop’s Warping EquationEquation
~Texture mappingMove pixels based on distance to eye
xx22 = = (x(x11) P) P22-1-1
(c(c11 - c - c22) + P) + P22-1-1
PP1 1 xx11
x1c1
c2x2
d
E31
Example...Example...
• Image outlined in yellowImage outlined in yellow• Viewed from image’s center-of-projectionViewed from image’s center-of-projection
E32
3D Image Warp3D Image Warp
• Single sample per pixelSingle sample per pixel
E33
Layered Depth Image Layered Depth Image WarpWarp
• Multiple samples per pixel Multiple samples per pixel – Previous occlusions are filled-inPrevious occlusions are filled-in
[Popescu98]
E34
MeshesMeshes
• (Simplified) Textured Depth Mesh(Simplified) Textured Depth Mesh– Per-pixel depth creates mesh that Per-pixel depth creates mesh that
approximates 3D parallax effectsapproximates 3D parallax effects– Image is texture-mapped onto meshImage is texture-mapped onto mesh
E35
Video Segment IIIVideo Segment III
• 3D Image Warping3D Image Warping– McMillan95McMillan95
• Textured MeshesTextured Meshes– Darsa97, Sillion97Darsa97, Sillion97
E36
Lightfield/LumigraphLightfield/Lumigraph
• Flow of light at all positions and Flow of light at all positions and directionsdirections– [Levoy96][Gortler96][Levoy96][Gortler96]
• Large number of images are used as Large number of images are used as 2D slices of a 4D light function2D slices of a 4D light function
(u,v)
(s,t)
E37
Video Segment IVVideo Segment IV
• Light fieldLight field– Levoy96Levoy96
• LumigraphLumigraph– Gortler96Gortler96
E38
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E39
Automatic Image-Automatic Image-PlacementPlacement
• As a preprocessAs a preprocess– Select geometry to replace with an Select geometry to replace with an
image in order to limit the number of image in order to limit the number of primitives to render for any frameprimitives to render for any frame
• At run timeAt run time– Display selected geometry as a Display selected geometry as a
(depth) image(depth) image– Render remaining geometry normallyRender remaining geometry normally
E40
Automatic Image-Automatic Image-PlacementPlacement
PreprocessPreprocess
Run timeRun time
ModelAutomatic
Image-Placement
Create Images
Cull GeometryReplaced with
Images
RenderGeometry and
ImagesDisplayModel
E41
Example RenderingExample Rendering
Final Scene
Geometry
Image
+ =
E42
Viewpoint
- Overview- Image Placement- Displaying Images- ConclusionsKey ObservationKey Observation
• Example modelExample model• Too much Too much
geometry in geometry in view frustumview frustum
E43
Viewpoint
Key ObservationKey Observation
• Geometry is Geometry is replaced by replaced by image to limit image to limit the number of the number of primitives to primitives to renderrender
E44
Eye
Grid viewpoint
Key ObservationKey Observation
• Less geometry Less geometry is in the view is in the view frustum from frustum from the eye than the eye than the one from the one from the grid the grid viewpoint viewpoint
E45
Recursive Subdivision Recursive Subdivision AlgorithmAlgorithm
2k 2k+1
2k+22k+1
Even to odd
Odd to even
E46
Example GridExample Grid
3D grid of 1557 viewpoints3D grid of 1557 viewpoints
-1
-0.50
0.51
-1
-0.5
0
0.5
1-1
-0.5
0
0.5
1
Wireframe renderingWireframe rendering
E47
Sample PathSample Path
0
100000
200000
300000
400000
500000
600000
700000
0 50 100 150 200 250 300Frame
View Frustum CullingImages Present
Pol
ygon
s
Power Plant, 2M tris, SGI Onyx2 w/ IR
E48
Images Per TriangleImages Per Triangle
0
0.001
0.002
0.003
0.004
0 10 20 30 40 50
Maximum Rendered Triangles (% of Model)
Imag
es P
er M
odel
Tri
angl
e
Power PlantTorpedo RoomBrooks HousePipesBest CaseWorst Case
E49
Preprocessing Preprocessing SummarySummary
Model No. of Images Prep. Time(hours)
EstimatedSpace (MB)
Power Plant(2M)
239-5815 1.2-21.7 156-3802
Torpedo Room(850k)
181-2333 1.1-11.8 72- 933
Brooks House(1.7M)
561-2492 11.4-28.4 388-1725
Pipes(1M)
282- 893 2.4- 4.6 175- 554
E50
Video Segment VVideo Segment V
• Automatically Bounding Geometric Automatically Bounding Geometric Complexity by Using ImagesComplexity by Using Images– Aliaga99Aliaga99
E51
Cells and PortalsCells and Portals
[Airey90, Teller91, Luebke95][Airey90, Teller91, Luebke95]
E52
Portal ImagesPortal Images
[Aliaga97][Aliaga97]
E53
Creating Portal ImagesCreating Portal Images
portaleye
Ideal portal Ideal portal image would be image would be one sampled one sampled from the current from the current eye positioneye position
E54
Creating Portal ImagesCreating Portal Images
Reference COPs
Display one of a large number of pre-computed images (~120) portal
E55
Creating Portal ImagesCreating Portal Images
portal
Reference COPs
or…
Warp one of a much smaller number of reference images
E56
Brooks HouseBrooks House
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0 50 100 150 200
Frame
Ren
der
ing
Tim
e (s
ec.)
View Frustum Culling
Portal Culling
Static Portal Images
Onyx IR
E57
Brooks HouseBrooks House
0
0.1
0.2
0.3
0.4
0.5
0 100 200 300 400 500
Frame
Re
nd
erin
g T
ime
(sec
.)
Portal Culling
Warped Portal Images
Onyx IR
E58
Video Segment VIVideo Segment VI
• Architectural Walkthroughs using Architectural Walkthroughs using Portal ImagesPortal Images– Aliaga97, Rafferty98Aliaga97, Rafferty98
E59
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E60
Image QualityImage Quality
• What about measuring quality? What about measuring quality? – Need a perceptual quality metric!Need a perceptual quality metric!
• We knowWe know– Texture-mapping: bad perspective, Texture-mapping: bad perspective,
small distortions believable (geometry small distortions believable (geometry warping)warping)
– IBR: correct perspective, disocclusionsIBR: correct perspective, disocclusions– Meshes: stretching of skinsMeshes: stretching of skins
E61
LimitationsLimitations
• Diffuse illuminationDiffuse illumination– Deferred shading?Deferred shading?
• Static modelsStatic models– Incremental updating?Incremental updating?
• Cannot sample all visible surfacesCannot sample all visible surfaces– Smarter reconstruction/resampling?Smarter reconstruction/resampling?
• Can only sample surfaces at a fixed Can only sample surfaces at a fixed resolutionsresolutions– Multi-resolution reference images?Multi-resolution reference images?
E62
AcknowledgmentsAcknowledgments
• Authors of the Video SegmentsAuthors of the Video Segments• ModelsModels
– Discreet Logic, UNC Walkthrough Discreet Logic, UNC Walkthrough GroupGroup
• UNC-Chapel HillUNC-Chapel Hill– Walkthrough, PixelFlow, ImageFlowWalkthrough, PixelFlow, ImageFlow
• NSF, NIH, DARPANSF, NIH, DARPA• Lucent Technologies Bell LabsLucent Technologies Bell Labs