unstructured lumigraph rendering chris buehler michael bosse leonard mcmillan leonard mcmillan...
Post on 19-Dec-2015
230 views
TRANSCRIPT
![Page 1: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/1.jpg)
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
Chris BuehlerChris BuehlerMichael Bosse Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS
Steven J. Gortler Steven J. Gortler Harvard University
Michael F. Cohen Michael F. Cohen Microsoft Research
![Page 2: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/2.jpg)
The Image-Based Rendering ProblemThe Image-Based Rendering Problem
• Synthesize novel views from reference Synthesize novel views from reference imagesimages
•Static scenes, fixed lighting
•Flexible geometry and camera configurations
![Page 3: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/3.jpg)
The ULR Algorithm The ULR Algorithm
• Designed to work over a range of Designed to work over a range of image and geometry configurationsimage and geometry configurations
• Designed to satisfy desirable Designed to satisfy desirable propertiesproperties
Geometric Fidelity
# o
f Im
ag
es
VDTMVDTM
LFLFULRULR
![Page 4: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/4.jpg)
Desired Camera
“Light Field Rendering,” SIGGRAPH ‘96
u0
s0
u
s
Desired color interpolated from “nearest cameras”
![Page 5: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/5.jpg)
Desired Camera
“Light Field Rendering,” SIGGRAPH ‘96
u
s
Desired Property #1: Epipole consistency
![Page 6: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/6.jpg)
Desired Camera
“The Scene”
“The Lumigraph,” SIGGRAPH ‘96
u
PotentialArtifact
![Page 7: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/7.jpg)
“The Scene”
“The Lumigraph,” SIGGRAPH ‘96
Desired Property #2: Use of geometric proxy
Desired Camera
![Page 8: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/8.jpg)
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Desired Camera
![Page 9: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/9.jpg)
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Rebinning
Note: all images are resampled.
Desired Camera
Desired Property #3: Unstructured input images
![Page 10: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/10.jpg)
“The Lumigraph,” SIGGRAPH ‘96
“The Scene”
Desired Property #4: Real-time implementation
Desired Camera
![Page 11: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/11.jpg)
Desired Camera
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
“The Scene”
Occluded
Out of view
![Page 12: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/12.jpg)
Desired Camera
“The Scene”
Desired Property #5: Continuous reconstruction
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
![Page 13: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/13.jpg)
Desired Camera
“The Scene”
θ1
θ2
θ3
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
![Page 14: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/14.jpg)
Desired Camera
“The Scene”
θ1
θ2
θ3
Desired Property #6: Angles measured w.r.t. proxy
View-Dependent Texture Mapping, SIGGRAPH ’96, EGRW ‘98
![Page 15: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/15.jpg)
“The Scene”
Desired Camera
![Page 16: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/16.jpg)
Desired Camera
“The Scene”
Desired Property #7: Resolution sensitivity
![Page 17: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/17.jpg)
Previous WorkPrevious Work
Light fields and LumigraphsLight fields and Lumigraphs•Levoy and Hanrahan, Gortler et al., Isaksen et al.
View-dependent Texture MappingView-dependent Texture Mapping•Debevec et al., Wood et al.
Plenoptic Modeling w/Hand-held Plenoptic Modeling w/Hand-held CamerasCameras•Heigl et al.
Many others…Many others…
![Page 18: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/18.jpg)
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
1.1. Epipole consistencyEpipole consistency
2.2. Use of geometric proxyUse of geometric proxy
3.3. Unstructured inputUnstructured input
4.4. Real-time implementationReal-time implementation
5.5. Continuous reconstructionContinuous reconstruction
6.6. Angles measured w.r.t. proxyAngles measured w.r.t. proxy
7.7. Resolution sensitivityResolution sensitivity
![Page 19: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/19.jpg)
Desired Camera
Blending FieldsBlending Fields
colordesired = Σ wi colorii
![Page 20: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/20.jpg)
Desired Camera
Blending FieldsBlending Fields
colordesired = Σ w(ci) colorii
![Page 21: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/21.jpg)
• Explicitly construct blending fieldExplicitly construct blending field•Computed using penalties
•Sample and interpolate over desired image
• Render with hardwareRender with hardware•Projective texture mapping and alpha
blending
Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering
![Page 22: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/22.jpg)
Angle PenaltyAngle Penalty
Geometric Proxy
θ1
θ2 θ3
θ4
θ5
θ6
penaltyang(Ci) = θi
C1
C2C3
C4
C5
C6
Cdesired
![Page 23: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/23.jpg)
Resolution PenaltyResolution Penalty
Cdesired
Geometric Proxy
penaltyres(Ci) = max(0,dist(Ci) – dist(Cdesired ))
distdesired
pena
lty
res
Ci
![Page 24: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/24.jpg)
Field-Of-View PenaltyField-Of-View Penalty
angle
pena
lty
FO
V
![Page 25: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/25.jpg)
Total PenaltyTotal Penalty
penalty(Ci) =α penaltyang(i) + β penaltyres(i) + γ penaltyfov(i)
![Page 26: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/26.jpg)
K-Nearest Continuous BlendingK-Nearest Continuous Blending
• Only use cameras with K smallest Only use cameras with K smallest penaltiespenalties
• CC0 0 Continuity: contribution drops to Continuity: contribution drops to zero as camera leaves zero as camera leaves K-nearest K-nearest setset
• w(Ci) = 1- penalty(Ci)/penalty(Ck+1st closest )
• Partition of Unity: normalize Partition of Unity: normalize
• w(Ci) = w(Ci)/Σw(Cj)
~
jj
![Page 27: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/27.jpg)
Blending Field VisualizationBlending Field Visualization
![Page 28: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/28.jpg)
Sampling Blending FieldsSampling Blending Fields
Just epipole samplingEpipole and grid sampling
![Page 29: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/29.jpg)
Select blending field sample locationsfor each sample location j do
for each camera i doCompute penalty(i) for sample location j
end forFind K smallest penaltiesCompute blending weights for sample location j
end forTriangulate sample locations
Hardware Assisted AlgorithmHardware Assisted Algorithm
Clear frame bufferfor each camera i do
Set current texture and projection matrixCopy blending weights to vertices’ alpha channelDraw triangles with non-zero alphas
end for
Sample Blending Field
Render with Graphics Hardware
![Page 30: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/30.jpg)
Blending over one triangleBlending over one triangle
Just epipole samplingEpipole and grid sampling
![Page 31: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/31.jpg)
Select blending field sample locationsfor each sample location j do
for each camera i doCompute penalty(i) for sample location j
end forFind K smallest penaltiesCompute blending weights for sample location j
end forTriangulate sample locations
Hardware Assisted AlgorithmHardware Assisted Algorithm
Clear frame bufferfor each camera i do
Set current texture and projection matrixCopy blending weights to vertices’ alpha channelDraw triangles with non-zero alphas
end for
Sample Blending Field
Render with Graphics Hardware
![Page 32: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/32.jpg)
DemoDemo
![Page 33: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/33.jpg)
Future WorkFuture Work
• Optimal sampling of the camera Optimal sampling of the camera blending fieldblending field
• More complete treatment of More complete treatment of resolution effects in IBRresolution effects in IBR
• View-dependent geometry proxiesView-dependent geometry proxies• Investigation of geometry vs. Investigation of geometry vs.
images tradeoffimages tradeoff
![Page 34: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/34.jpg)
ConclusionsConclusions
Unstructured Lumigraph Unstructured Lumigraph RenderingRendering •unifies view-dependent texture mapping and
lumigraph rendering methods
•allows rendering from unorganized images
• sampled camera blending field
![Page 35: Unstructured Lumigraph Rendering Chris Buehler Michael Bosse Leonard McMillan Leonard McMillan MIT-LCS Steven J. Gortler Steven J. Gortler Harvard University](https://reader035.vdocuments.us/reader035/viewer/2022062714/56649d2b5503460f94a0146d/html5/thumbnails/35.jpg)
AcknowledgementsAcknowledgementsThanks to the members of theThanks to the members of the
MIT Computer Graphics GroupMIT Computer Graphics Groupandand
Microsoft ResearchMicrosoft ResearchGraphics and Computer Vision GroupsGraphics and Computer Vision Groups
• DARPA ITO Grant F30602-971-0283
• NSF CAREER Awards 9875859 & 9703399
• Microsoft Research Graduate Fellowship Program
• Donations from Intel Corporation, Nvidia, andMicrosoft Corporation