transport and rendering challenges of multi-stream, 3d tele-immersion data herman towles,
DESCRIPTION
Transport and Rendering Challenges of Multi-Stream, 3D Tele-Immersion Data Herman Towles, Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Larsen and Nathan Beddes Department of Computer Science University of North Carolina at Chapel Hill October 28, 2003. ‘’ Portal’ to Another Place. - PowerPoint PPT PresentationTRANSCRIPT
1
Transport and Rendering Challenges Transport and Rendering Challenges of Multi-Stream, 3D Tele-of Multi-Stream, 3D Tele-
Immersion DataImmersion Data
Herman Towles,Herman Towles,Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott
Larsen and Nathan BeddesLarsen and Nathan Beddes
Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill
October 28, 2003October 28, 2003
Transport and Rendering Challenges Transport and Rendering Challenges of Multi-Stream, 3D Tele-of Multi-Stream, 3D Tele-
Immersion DataImmersion Data
Herman Towles,Herman Towles,Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott
Larsen and Nathan BeddesLarsen and Nathan Beddes
Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill
October 28, 2003October 28, 2003
2
‘’‘’Portal’ to Another Place Portal’ to Another Place
UNC - January 2000
Static, High-Resolution 3D Scene
Head-Tracked, Passive Stereo Projector Display
3
Static Scene Acquisition Static Scene Acquisition
• DeltaSphere™ DeltaSphere™ Scanning Laser Scanning Laser RangefinderRangefinder– Time of FlightTime of Flight
– Resolution: ~3K x 2K for 150Resolution: ~3K x 2K for 150oo azimuth and 90azimuth and 90oo elevation elevation
– Digital Camera mounted at Digital Camera mounted at identical COP for color imageidentical COP for color image
• OutputOutput– Range ImageRange Image
– Color Camera ImageColor Camera Image
• DeltaSphere™ DeltaSphere™ Scanning Laser Scanning Laser RangefinderRangefinder– Time of FlightTime of Flight
– Resolution: ~3K x 2K for 150Resolution: ~3K x 2K for 150oo azimuth and 90azimuth and 90oo elevation elevation
– Digital Camera mounted at Digital Camera mounted at identical COP for color imageidentical COP for color image
• OutputOutput– Range ImageRange Image
– Color Camera ImageColor Camera Image
Images courtesy of 3rd Tech, Inc.
4
NTII CollaboratorsNTII Collaborators
• UNC-CH – Computer GraphicsUNC-CH – Computer Graphics– Henry Fuchs et al.Henry Fuchs et al.
• UPenn – Computer VisionUPenn – Computer Vision– Kostas Daniilidas et al.Kostas Daniilidas et al.
• Brown University – Collaborative Brown University – Collaborative GraphicsGraphics
– Andy van Dam et al.Andy van Dam et al.
• Advanced Network & ServicesAdvanced Network & Services– Jaron Lanier & Amela SadagicJaron Lanier & Amela Sadagic
Goal: Develop a Live 3D Goal: Develop a Live 3D ‘Portal’‘Portal’
• UNC-CH – Computer GraphicsUNC-CH – Computer Graphics– Henry Fuchs et al.Henry Fuchs et al.
• UPenn – Computer VisionUPenn – Computer Vision– Kostas Daniilidas et al.Kostas Daniilidas et al.
• Brown University – Collaborative Brown University – Collaborative GraphicsGraphics
– Andy van Dam et al.Andy van Dam et al.
• Advanced Network & ServicesAdvanced Network & Services– Jaron Lanier & Amela SadagicJaron Lanier & Amela Sadagic
Goal: Develop a Live 3D Goal: Develop a Live 3D ‘Portal’‘Portal’
5
3D Tele-Immersion – Phase 3D Tele-Immersion – Phase 1 1
• 3D Real-Time 3D Real-Time Reconstruction Reconstruction
– Correlation-based Stereo AlgorithmCorrelation-based Stereo Algorithm
– 1-3 fps at 320 x 240 (quad-PC/550)1-3 fps at 320 x 240 (quad-PC/550)
– 1 cubic meter working volume 1 cubic meter working volume
– Up to 5 streams of depth imagesUp to 5 streams of depth images
• 3D Tracked Stereo Display3D Tracked Stereo Display– Life-size with gaze awarenessLife-size with gaze awareness
– 3-PC Rendering Cluster3-PC Rendering Cluster
– Composite SceneComposite Scene
» Live 3D Remote CollaboratorLive 3D Remote Collaborator
» Static, office backgroundStatic, office background
» Collaborative GraphicsCollaborative Graphics
• 3D Real-Time 3D Real-Time Reconstruction Reconstruction
– Correlation-based Stereo AlgorithmCorrelation-based Stereo Algorithm
– 1-3 fps at 320 x 240 (quad-PC/550)1-3 fps at 320 x 240 (quad-PC/550)
– 1 cubic meter working volume 1 cubic meter working volume
– Up to 5 streams of depth imagesUp to 5 streams of depth images
• 3D Tracked Stereo Display3D Tracked Stereo Display– Life-size with gaze awarenessLife-size with gaze awareness
– 3-PC Rendering Cluster3-PC Rendering Cluster
– Composite SceneComposite Scene
» Live 3D Remote CollaboratorLive 3D Remote Collaborator
» Static, office backgroundStatic, office background
» Collaborative GraphicsCollaborative Graphics
6
Standard Videoconferencing Standard Videoconferencing ModelModel
Camera MonitorNetwork
• 1-Camera to 1-Monitor Paradigm 1-Camera to 1-Monitor Paradigm
• Single Stream of 2D Image DataSingle Stream of 2D Image Data
View-Dependent Data !View-Dependent Data !
• 1-Camera to 1-Monitor Paradigm 1-Camera to 1-Monitor Paradigm
• Single Stream of 2D Image DataSingle Stream of 2D Image Data
View-Dependent Data !View-Dependent Data !
7
3D Tele-Conferencing 3D Tele-Conferencing ModelModel
3D Camera 3D RenderingNetwork
• N-Camera to 1-Monitor N-Camera to 1-Monitor
• Multiple Streams of 3D Image DataMultiple Streams of 3D Image Data– Depth Image (16-bit 1/z) and RGB Image (24-bit)Depth Image (16-bit 1/z) and RGB Image (24-bit)
View-Independent Data w/ View-View-Independent Data w/ View-Dependent Rendering !Dependent Rendering !
• N-Camera to 1-Monitor N-Camera to 1-Monitor
• Multiple Streams of 3D Image DataMultiple Streams of 3D Image Data– Depth Image (16-bit 1/z) and RGB Image (24-bit)Depth Image (16-bit 1/z) and RGB Image (24-bit)
View-Independent Data w/ View-View-Independent Data w/ View-Dependent Rendering !Dependent Rendering !
3D Camera
3D Camera
User Tracker
8
3D Tele-Immersion- Phase 23D Tele-Immersion- Phase 2
• Higher QualityHigher Quality– Improve Resolution, Frame Improve Resolution, Frame
Rate and DisplayRate and Display
• Larger Larger Reconstruction Reconstruction VolumeVolume– 30-60 color cameras 30-60 color cameras
– Acquire full environmentAcquire full environment
• Higher QualityHigher Quality– Improve Resolution, Frame Improve Resolution, Frame
Rate and DisplayRate and Display
• Larger Larger Reconstruction Reconstruction VolumeVolume– 30-60 color cameras 30-60 color cameras
– Acquire full environmentAcquire full environment Terascale Computing System (Lemieux)at Pittsburgh Supercomputing Center (PSC)
Acquisition Reconstruction
3D Camera
3D Camera 3D RenderingNetworkReconstructionAcquisition Network
9
2000-2002 Results 2000-2002 Results
• Show videoShow video• Show videoShow video
10
Outline Outline
• IntroductionIntroduction
• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions
• Rendering Challenges & SolutionsRendering Challenges & Solutions
• Conclusions/FuturesConclusions/Futures
• IntroductionIntroduction
• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions
• Rendering Challenges & SolutionsRendering Challenges & Solutions
• Conclusions/FuturesConclusions/Futures
11
Data Transport ChallengesData Transport Challenges
• Reliable TransportReliable Transport– Low LatencyLow Latency
– Synchronized Arrival of Multi-StreamsSynchronized Arrival of Multi-Streams
• Large Bandwidth RequirementsLarge Bandwidth Requirements
• Reliable TransportReliable Transport– Low LatencyLow Latency
– Synchronized Arrival of Multi-StreamsSynchronized Arrival of Multi-Streams
• Large Bandwidth RequirementsLarge Bandwidth Requirements
No. Streams No. Streams & Resolution& Resolution
2D 2D Bandwidth Bandwidth
2Hz@24bpp, 2Hz@24bpp, MbpsMbps
3D 3D Bandwidth Bandwidth
2Hz@40bpp, 2Hz@40bpp, Mbps Mbps
5@320x2405@320x240 18.418.4 30.830.8
5@640x4805@640x480 73.873.8 122.8122.8
15@640x48015@640x480 221.4221.4 368.4368.4
12
Compression OpportunitiesCompression Opportunities
• Color Image CompressionColor Image Compression– Spatial Compression (JPEG)Spatial Compression (JPEG)
– Temporal Compression (MPEG)Temporal Compression (MPEG)
• Depth ImageDepth Image– 16-bit 1/z image16-bit 1/z image
• Elimination of Redundant DataElimination of Redundant Data – Overlapping FOVs from Multiple CamerasOverlapping FOVs from Multiple Cameras
– Reduces BW and Rendering Requirements !!Reduces BW and Rendering Requirements !!
• Color Image CompressionColor Image Compression– Spatial Compression (JPEG)Spatial Compression (JPEG)
– Temporal Compression (MPEG)Temporal Compression (MPEG)
• Depth ImageDepth Image– 16-bit 1/z image16-bit 1/z image
• Elimination of Redundant DataElimination of Redundant Data – Overlapping FOVs from Multiple CamerasOverlapping FOVs from Multiple Cameras
– Reduces BW and Rendering Requirements !!Reduces BW and Rendering Requirements !!
13
Overlapping Depth ImagesOverlapping Depth Images
14
Redundant Data Redundant Data EliminationElimination
• Algorithm: Compare Every Depth Algorithm: Compare Every Depth StreamStream
• ProsPros– Best compressionBest compression
– Eliminates all redundant DataEliminates all redundant Data
• ConsCons– Jeopardizes Real-Time PerformanceJeopardizes Real-Time Performance
– Not scalable with number of 3D camerasNot scalable with number of 3D cameras
– Maximizes network BW requirements between Maximizes network BW requirements between ‘3D Cameras’ ‘3D Cameras’
• Algorithm: Compare Every Depth Algorithm: Compare Every Depth StreamStream
• ProsPros– Best compressionBest compression
– Eliminates all redundant DataEliminates all redundant Data
• ConsCons– Jeopardizes Real-Time PerformanceJeopardizes Real-Time Performance
– Not scalable with number of 3D camerasNot scalable with number of 3D cameras
– Maximizes network BW requirements between Maximizes network BW requirements between ‘3D Cameras’ ‘3D Cameras’
15
Group-Based, Differential Group-Based, Differential Stream Compression Stream Compression
• Two Stream Differential Two Stream Differential ComparisonComparison– Maintains real-time performanceMaintains real-time performance
- Scales with number of camerasScales with number of cameras
- Static, Disjoint Stream GroupsStatic, Disjoint Stream Groups- Geometric coherency metricGeometric coherency metric
• One Reference Stream per GroupOne Reference Stream per Group
• Main Reference StreamMain Reference Stream– Stream most closely aligned with remote Stream most closely aligned with remote
viewpointviewpoint
– UncompressedUncompressed
• Two Stream Differential Two Stream Differential ComparisonComparison– Maintains real-time performanceMaintains real-time performance
- Scales with number of camerasScales with number of cameras
- Static, Disjoint Stream GroupsStatic, Disjoint Stream Groups- Geometric coherency metricGeometric coherency metric
• One Reference Stream per GroupOne Reference Stream per Group
• Main Reference StreamMain Reference Stream– Stream most closely aligned with remote Stream most closely aligned with remote
viewpointviewpoint
– UncompressedUncompressed
16
Grouping & Differential Grouping & Differential EncodingEncoding
Group1
Group2
Group3
6
8 9
2 3
7
1
4 5
Group2 5
17
Algorithm PerformanceAlgorithm Performance
• Using Synthetic Camera ArraysUsing Synthetic Camera Arrays– 13 and 22 3D-camera arrays13 and 22 3D-camera arrays
– Real-world 3D dataReal-world 3D data
• 5X Compression Ratio5X Compression Ratio
• 50-60% of Best Possible Point 50-60% of Best Possible Point ReductionReduction
• 10z Performance at 640x48010z Performance at 640x480
• Using Synthetic Camera ArraysUsing Synthetic Camera Arrays– 13 and 22 3D-camera arrays13 and 22 3D-camera arrays
– Real-world 3D dataReal-world 3D data
• 5X Compression Ratio5X Compression Ratio
• 50-60% of Best Possible Point 50-60% of Best Possible Point ReductionReduction
• 10z Performance at 640x48010z Performance at 640x480
18
Multi-Stream TransportMulti-Stream Transport
• Reliable is Good, so TCP/IPReliable is Good, so TCP/IP
WRONG!WRONG!• LatencyLatency
• Unsynchronized Arrival of Unsynchronized Arrival of StreamsStreams– Synchronized at CamerasSynchronized at Cameras
– Must be re-synchronize at rendererMust be re-synchronize at renderer
• TCP Flows Competing for BWTCP Flows Competing for BW
• Reliable is Good, so TCP/IPReliable is Good, so TCP/IP
WRONG!WRONG!• LatencyLatency
• Unsynchronized Arrival of Unsynchronized Arrival of StreamsStreams– Synchronized at CamerasSynchronized at Cameras
– Must be re-synchronize at rendererMust be re-synchronize at renderer
• TCP Flows Competing for BWTCP Flows Competing for BW
19
Frame Arrival Time Frame Arrival Time Variation Variation
20
Throughput Variation Throughput Variation
21
CP-RUDP CP-RUDP
• Reliable UDP (RUDP)Reliable UDP (RUDP)
• Coordination Protocol (CP)Coordination Protocol (CP)
• CharacteristicsCharacteristics– A TCP-friendly UDP protocolA TCP-friendly UDP protocol
– Aggregately acts like N TCP flowsAggregately acts like N TCP flows
– Providing BW coordination and synchronization of multiple Providing BW coordination and synchronization of multiple flowsflows
– Prevents send rates from exceeding network capacityPrevents send rates from exceeding network capacity
Flows that are ahead are given less BWFlows that are ahead are given less BW
Flows that are behind are given more Flows that are behind are given more BWBW
• Reliable UDP (RUDP)Reliable UDP (RUDP)
• Coordination Protocol (CP)Coordination Protocol (CP)
• CharacteristicsCharacteristics– A TCP-friendly UDP protocolA TCP-friendly UDP protocol
– Aggregately acts like N TCP flowsAggregately acts like N TCP flows
– Providing BW coordination and synchronization of multiple Providing BW coordination and synchronization of multiple flowsflows
– Prevents send rates from exceeding network capacityPrevents send rates from exceeding network capacity
Flows that are ahead are given less BWFlows that are ahead are given less BW
Flows that are behind are given more Flows that are behind are given more BWBW
22
Future CP-RUDP Testbed Future CP-RUDP Testbed
• Future Network Router FunctionalityFuture Network Router Functionality
• Emulated today on fast FreeBSD PCEmulated today on fast FreeBSD PC
• Future Network Router FunctionalityFuture Network Router Functionality
• Emulated today on fast FreeBSD PCEmulated today on fast FreeBSD PC
3D Camera Internet2
3D Camera
3D Camera
3D Rendering
User Tracker
CP-RUDPSoftware
Router
CP-RUDPSoftware
Router
23
Outline Outline
• IntroductionIntroduction
• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions
• Rendering Challenges & SolutionsRendering Challenges & Solutions
• Conclusions/FuturesConclusions/Futures
• IntroductionIntroduction
• Data Transport Challenges & Data Transport Challenges & SolutionsSolutions
• Rendering Challenges & SolutionsRendering Challenges & Solutions
• Conclusions/FuturesConclusions/Futures
24
Rendering ChallengesRendering Challenges
• High Performance, Interactive High Performance, Interactive ArchitectureArchitecture
• Quality Surface RepresentationQuality Surface Representation
• Scalable PerformanceScalable Performance
• High Performance, Interactive High Performance, Interactive ArchitectureArchitecture
• Quality Surface RepresentationQuality Surface Representation
• Scalable PerformanceScalable Performance
25
High Performance, Interactive High Performance, Interactive ArchitectureArchitecture
• Frame Re-synchronizationFrame Re-synchronization– ‘‘N’ 3D-camera StreamsN’ 3D-camera Streams
– Timestamps on each data frameTimestamps on each data frame
– Buffering -> More Latency Buffering -> More Latency BAD!BAD!
• Multi-threaded Design for Multi-threaded Design for InteractivityInteractivity– 30Hz Refresh Rate Desirable30Hz Refresh Rate Desirable
– <10 Hz Data Update Rate<10 Hz Data Update Rate
• 3-PC Rendering Cluster (Linux)3-PC Rendering Cluster (Linux)– Left & right eye rendering nodesLeft & right eye rendering nodes
– 33rdrd Node – Network Aggregation Point Node – Network Aggregation Point
• Frame Re-synchronizationFrame Re-synchronization– ‘‘N’ 3D-camera StreamsN’ 3D-camera Streams
– Timestamps on each data frameTimestamps on each data frame
– Buffering -> More Latency Buffering -> More Latency BAD!BAD!
• Multi-threaded Design for Multi-threaded Design for InteractivityInteractivity– 30Hz Refresh Rate Desirable30Hz Refresh Rate Desirable
– <10 Hz Data Update Rate<10 Hz Data Update Rate
• 3-PC Rendering Cluster (Linux)3-PC Rendering Cluster (Linux)– Left & right eye rendering nodesLeft & right eye rendering nodes
– 33rdrd Node – Network Aggregation Point Node – Network Aggregation Point
26
Rendering Architecture Rendering Architecture
27
Quality Surface Quality Surface RepresentationRepresentation
• Depth Images (1/z + RGB per pixel)Depth Images (1/z + RGB per pixel)
• Real-Time TriangulationReal-Time Triangulation– Performance LimitingPerformance Limiting
– Artifacts due to noisy outliers. Artifacts due to noisy outliers. Most Disturbing!Most Disturbing!
• Real-Time Massive Point Cloud Real-Time Massive Point Cloud RenderingRendering– Display List vs. VARDisplay List vs. VAR
– GL_POINT vs. GL_POINT_SPRINTGL_POINT vs. GL_POINT_SPRINT
– Screen-Oriented vs. Object-Oriented SplatsScreen-Oriented vs. Object-Oriented Splats
• Depth Images (1/z + RGB per pixel)Depth Images (1/z + RGB per pixel)
• Real-Time TriangulationReal-Time Triangulation– Performance LimitingPerformance Limiting
– Artifacts due to noisy outliers. Artifacts due to noisy outliers. Most Disturbing!Most Disturbing!
• Real-Time Massive Point Cloud Real-Time Massive Point Cloud RenderingRendering– Display List vs. VARDisplay List vs. VAR
– GL_POINT vs. GL_POINT_SPRINTGL_POINT vs. GL_POINT_SPRINT
– Screen-Oriented vs. Object-Oriented SplatsScreen-Oriented vs. Object-Oriented Splats
28
Fixed-Size SplatsFixed-Size Splats
29
Screen vs. Object Space Screen vs. Object Space SplatsSplats
30
Scalable PerformanceScalable Performance
• 2x Gain w/3-PC Rendering Cluster2x Gain w/3-PC Rendering Cluster
• Ride the Wave (Moore’s Law & Ride the Wave (Moore’s Law & Beyond)Beyond)
Not Enough!Not Enough!
• Divide and ConquerDivide and Conquer– Eliminate Data Replication RequirementEliminate Data Replication Requirement
– Match # Primitives to PerformanceMatch # Primitives to Performance
• Depth Compositing Depth Compositing – UNC PixelFlow, Stanford/Intel Lightning-2 or HP SepiaUNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia
– DVI output – RGB and ZDVI output – RGB and Z
• 2x Gain w/3-PC Rendering Cluster2x Gain w/3-PC Rendering Cluster
• Ride the Wave (Moore’s Law & Ride the Wave (Moore’s Law & Beyond)Beyond)
Not Enough!Not Enough!
• Divide and ConquerDivide and Conquer– Eliminate Data Replication RequirementEliminate Data Replication Requirement
– Match # Primitives to PerformanceMatch # Primitives to Performance
• Depth Compositing Depth Compositing – UNC PixelFlow, Stanford/Intel Lightning-2 or HP SepiaUNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia
– DVI output – RGB and ZDVI output – RGB and Z
31
Conclusions/FuturesConclusions/Futures
• Much work remains !Much work remains !– Performance and FidelityPerformance and Fidelity
• Investigating Temporal Compression Investigating Temporal Compression of 3D Streamsof 3D Streams
• Testing CP-RUDP Network ProtocolTesting CP-RUDP Network Protocol
• New Voxel-Based SolutionNew Voxel-Based Solution– Elminates Redundant DataElminates Redundant Data
– Normals and Confidence Value per VoxelNormals and Confidence Value per Voxel
– GPU Programs for Object-oriented Quad Splats with GPU Programs for Object-oriented Quad Splats with Projective TextureProjective Texture
• Much work remains !Much work remains !– Performance and FidelityPerformance and Fidelity
• Investigating Temporal Compression Investigating Temporal Compression of 3D Streamsof 3D Streams
• Testing CP-RUDP Network ProtocolTesting CP-RUDP Network Protocol
• New Voxel-Based SolutionNew Voxel-Based Solution– Elminates Redundant DataElminates Redundant Data
– Normals and Confidence Value per VoxelNormals and Confidence Value per Voxel
– GPU Programs for Object-oriented Quad Splats with GPU Programs for Object-oriented Quad Splats with Projective TextureProjective Texture
32
Research Sponsors Research Sponsors
• National Science FoundationNational Science Foundation– Grant IIS-0121293 Junku Yuh, Program DirectorGrant IIS-0121293 Junku Yuh, Program Director
• DARPADARPA– 3-D Tele-Immersion over NGI3-D Tele-Immersion over NGI
• Advanced Network & Services, Advanced Network & Services, Armonk, NYArmonk, NY– National Tele-Immersion Initiative (NTII), 1998-2000National Tele-Immersion Initiative (NTII), 1998-2000
• National Science FoundationNational Science Foundation– Grant IIS-0121293 Junku Yuh, Program DirectorGrant IIS-0121293 Junku Yuh, Program Director
• DARPADARPA– 3-D Tele-Immersion over NGI3-D Tele-Immersion over NGI
• Advanced Network & Services, Advanced Network & Services, Armonk, NYArmonk, NY– National Tele-Immersion Initiative (NTII), 1998-2000National Tele-Immersion Initiative (NTII), 1998-2000
33
Thank You Thank You
UNC - January 2000
34
Thank You Thank You
UNC - January 2000
35
Positive Technology TrendsPositive Technology Trends
• Low-cost image sensorsLow-cost image sensors
• Vigorous research into new light field and Vigorous research into new light field and image-based scene reconstruction image-based scene reconstruction algorithmsalgorithms
• Incredible performance in commodity PCs Incredible performance in commodity PCs and 3D graphicsand 3D graphics
• Ease of Developing Custom HardwareEase of Developing Custom Hardware
• New Displays – HiRes LCD and Plasma New Displays – HiRes LCD and Plasma panels, Tiled-Projectors, OLEDs, Auto-panels, Tiled-Projectors, OLEDs, Auto-stereoscopicstereoscopic
• Increasing network bandwidth to the Increasing network bandwidth to the workplace and homeworkplace and home
• Low-cost image sensorsLow-cost image sensors
• Vigorous research into new light field and Vigorous research into new light field and image-based scene reconstruction image-based scene reconstruction algorithmsalgorithms
• Incredible performance in commodity PCs Incredible performance in commodity PCs and 3D graphicsand 3D graphics
• Ease of Developing Custom HardwareEase of Developing Custom Hardware
• New Displays – HiRes LCD and Plasma New Displays – HiRes LCD and Plasma panels, Tiled-Projectors, OLEDs, Auto-panels, Tiled-Projectors, OLEDs, Auto-stereoscopicstereoscopic
• Increasing network bandwidth to the Increasing network bandwidth to the workplace and homeworkplace and home
36
Renderer Block Diagram Renderer Block Diagram
37
• Information Society Technologies (IST) Programme of the Information Society Technologies (IST) Programme of the European CommissionEuropean Commission
http://bs.hhi.de/projects/VIRTUE.htmhttp://bs.hhi.de/projects/VIRTUE.htm
• Information Society Technologies (IST) Programme of the Information Society Technologies (IST) Programme of the European CommissionEuropean Commission
http://bs.hhi.de/projects/VIRTUE.htmhttp://bs.hhi.de/projects/VIRTUE.htm
VIRTUE: VIRtual Team User VIRTUE: VIRtual Team User EnvironmentEnvironment
38
VIRTUE PartnersVIRTUE Partners
• Heinrich-Hertz Institut (now Heinrich-Hertz Institut (now Fraunhofer member)Fraunhofer member)– Disparity Estimator, Real-time ComponentsDisparity Estimator, Real-time Components
• University of DelftUniversity of Delft– 3D Algorithms, Image-Based Rendering3D Algorithms, Image-Based Rendering
• TNO Human FactorsTNO Human Factors• Sony U.K. Sony U.K.
– System design, Compositor, Rendering, MPEG-2System design, Compositor, Rendering, MPEG-2
• Harriot Watt UniversityHarriot Watt University– 3D Video Processing3D Video Processing
• British Telecom British Telecom – Project Prime, Video-based Head TrackingProject Prime, Video-based Head Tracking
• Heinrich-Hertz Institut (now Heinrich-Hertz Institut (now Fraunhofer member)Fraunhofer member)– Disparity Estimator, Real-time ComponentsDisparity Estimator, Real-time Components
• University of DelftUniversity of Delft– 3D Algorithms, Image-Based Rendering3D Algorithms, Image-Based Rendering
• TNO Human FactorsTNO Human Factors• Sony U.K. Sony U.K.
– System design, Compositor, Rendering, MPEG-2System design, Compositor, Rendering, MPEG-2
• Harriot Watt UniversityHarriot Watt University– 3D Video Processing3D Video Processing
• British Telecom British Telecom – Project Prime, Video-based Head TrackingProject Prime, Video-based Head Tracking
39
VIRTUE TechnologyVIRTUE Technology
• Realistic Wide View Synthesis for Realistic Wide View Synthesis for Dynamic ScenesDynamic Scenes
• Object-based SegmentationObject-based Segmentation
• Disparity Engine runs 4x4 grid @ Disparity Engine runs 4x4 grid @ videoratevideorate
• Facial feature (eye) trackingFacial feature (eye) tracking
• Generating View-dependent Novel ViewGenerating View-dependent Novel View
• 2D Display (with motion parallax) 2D Display (with motion parallax) rendered with virtual backgroundrendered with virtual background
Demonstrated at IBC – Fall 2002Demonstrated at IBC – Fall 2002
• Realistic Wide View Synthesis for Realistic Wide View Synthesis for Dynamic ScenesDynamic Scenes
• Object-based SegmentationObject-based Segmentation
• Disparity Engine runs 4x4 grid @ Disparity Engine runs 4x4 grid @ videoratevideorate
• Facial feature (eye) trackingFacial feature (eye) tracking
• Generating View-dependent Novel ViewGenerating View-dependent Novel View
• 2D Display (with motion parallax) 2D Display (with motion parallax) rendered with virtual backgroundrendered with virtual background
Demonstrated at IBC – Fall 2002Demonstrated at IBC – Fall 2002
40
Image-based Visual Image-based Visual HullHull
Image-based Visual Image-based Visual HullHull
HP Labs ‘Coliseum’ System HP Labs ‘Coliseum’ System
41
HP Labs ‘Coliseum’ SystemHP Labs ‘Coliseum’ System
• Personal Immersive Conferencing Personal Immersive Conferencing
• Based on ‘Visual Hull’ research of Based on ‘Visual Hull’ research of Matusik, McMillan et al. at MITMatusik, McMillan et al. at MIT– Space carving algorithmSpace carving algorithm
– Foreground/Background Foreground/Background
• 5 camera (1394) array, Now runs on 1 5 camera (1394) array, Now runs on 1 PCPC– 224 x 224 @ 10 Hz224 x 224 @ 10 Hz
• 300 x 300 display window300 x 300 display window– 3D Volume Composited with 3D VRML scene3D Volume Composited with 3D VRML scene
Demonstrated at ITP – Dec 2002Demonstrated at ITP – Dec 2002
• Personal Immersive Conferencing Personal Immersive Conferencing
• Based on ‘Visual Hull’ research of Based on ‘Visual Hull’ research of Matusik, McMillan et al. at MITMatusik, McMillan et al. at MIT– Space carving algorithmSpace carving algorithm
– Foreground/Background Foreground/Background
• 5 camera (1394) array, Now runs on 1 5 camera (1394) array, Now runs on 1 PCPC– 224 x 224 @ 10 Hz224 x 224 @ 10 Hz
• 300 x 300 display window300 x 300 display window– 3D Volume Composited with 3D VRML scene3D Volume Composited with 3D VRML scene
Demonstrated at ITP – Dec 2002Demonstrated at ITP – Dec 2002
42
CreditsCredits
• Colleagues at Colleagues at – UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar,
Xenofon ZabulisXenofon Zabulis
– PSC: John Urbanic, Kathy BenningerPSC: John Urbanic, Kathy Benninger
– Advanced Networks & Services: Jaron Lanier, Amela Advanced Networks & Services: Jaron Lanier, Amela SadagicSadagic
– UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-PatelUNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel
• UNC RAsUNC RAs– Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta
Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and Gabe SuOtt, and Gabe Su
• Colleagues at Colleagues at – UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar,
Xenofon ZabulisXenofon Zabulis
– PSC: John Urbanic, Kathy BenningerPSC: John Urbanic, Kathy Benninger
– Advanced Networks & Services: Jaron Lanier, Amela Advanced Networks & Services: Jaron Lanier, Amela SadagicSadagic
– UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-PatelUNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel
• UNC RAsUNC RAs– Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta
Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Sinha, Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and Gabe SuOtt, and Gabe Su
43
Our Vision: ‘Xtreme Tele-Our Vision: ‘Xtreme Tele-PresencePresence
‘Office of the Future’
Andrei State 1998
44
Our Vision: ‘Xtreme Our Vision: ‘Xtreme ChallengesChallenges
‘Office of the Future’
Andrei State 1998
Distributed Graphics• Collaborative Datasets• User Interfaces
3D Scene Acquisition• Cameras• 3D Reconstruction
Presentation• 3D Stereo Display• View Dependent• Front Projection• Spatialized Audio
Networking• QoS • BW-Latency-Jitter
45
Immersive Electronic BooksImmersive Electronic Books
• New Paradigm for Surgical TrainingNew Paradigm for Surgical Training
• Allow Surgeons to Witness & ExploreAllow Surgeons to Witness & Explore– Replay a surgical procedure from any novel viewpoint in Replay a surgical procedure from any novel viewpoint in
3D3D
– Augmented with annotations & relevant medical Augmented with annotations & relevant medical metadatametadata
• New Paradigm for Surgical TrainingNew Paradigm for Surgical Training
• Allow Surgeons to Witness & ExploreAllow Surgeons to Witness & Explore– Replay a surgical procedure from any novel viewpoint in Replay a surgical procedure from any novel viewpoint in
3D3D
– Augmented with annotations & relevant medical Augmented with annotations & relevant medical metadatametadata
46
Remote Medical Remote Medical ConsultationConsultation