improving the speed of virtual rear projection: a gpu-centric architecture
DESCRIPTION
Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture. Matthew Flagg, Jay Summet, James M. Rehg GVU Center College of Computing Georgia Institute of Technology. Ubiquitous Interactive Displays. Every flat surface can be an interactive display. VRP: Shadow Elimination. - PowerPoint PPT PresentationTRANSCRIPT
Improving the Speed ofVirtual Rear Projection:
A GPU-Centric Architecture
Matthew Flagg, Jay Summet,James M. Rehg
GVU CenterCollege of Computing
Georgia Institute of Technology
Matthew Flagg © 2005 22
Ubiquitous Interactive Displays
Every flat surface can be an interactive display
Matthew Flagg © 2005 33
VRP: Shadow Elimination
Single Projector Case
Matthew Flagg © 2005 44
Half power shadows
Shadow Elimination
Double Projector Case
Passive VRP
Matthew Flagg © 2005 55
Shadow Elimination
Boosting projector outputs
Proportional feedback law
Matthew Flagg © 2005 66
Occluder Light Suppression
Detecting occluded pixels
Matthew Flagg © 2005 77
Detecting occluded pixels
Occluder Light Suppression
Matthew Flagg © 2005 88
Detecting occluded pixels
Occluder Light Suppression
Matthew Flagg © 2005 99
Detecting occluded pixels
Occluder Light Suppression
Nonlinear feedback law
Matthew Flagg © 2005 1010
Virtual Rear Projection
Show ICCV’03 demo video
Matthew Flagg © 2005 1111
2 Challenges for VRP
High image qualitySeams between display regions projected by
different projectorsPhotometric Uniformity
Fast CompensationAvoid perception of shadows caused by
system lagImage processing required to ensure high
image quality
Matthew Flagg © 2005 1212
Camera view of screen must be unobstructedRequires reference image capture before
occlusionCannot be co-located with projector
Shadows still perceptibleShadow detection image processing
performed on CPU
Limitations With Previous Work
Matthew Flagg © 2005 1313
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
Matthew Flagg © 2005 1414
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0IR backlit camera image
Matthew Flagg © 2005 1515
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
Turn off occluded pixels
Matthew Flagg © 2005 1616
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Occluder Light Suppression
Matthew Flagg © 2005 1717
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Shadow Eliminator Blinding Light Suppressor
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
Turn on occluded pixels with second projector
Matthew Flagg © 2005 1818
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Shadow Elimination
Matthew Flagg © 2005 1919
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Shadow Elimination and Occluder Light Suppression
Matthew Flagg © 2005 2020
Fast Compensation: GPU-Centric Approach
Matthew Flagg © 2005 2121
Fast Compensation: GPU-Centric Approach
1. Warping, background subtraction
2. Median filtering anddilation for inter-frame tolerance
3. Gaussian blur for blending
4. Compositing and warping
Matthew Flagg © 2005 2222
Pixel Shader Pipeline
camera
texture
background
texture
render
texture 1
render
texture 2
back
buffer
display
image
(A) (B) (C)
Matthew Flagg © 2005 2323
Addressing Image Quality
LAM for left projector
LAM for right projector
Luminance Attenuation Maps (LAMs) Simple feedback-based
approach to accommodate non-linearities of projector-camera
Seam Blending
seam – no blending
seam – with blending
Matthew Flagg © 2005 2424
Virtual Rear Projection Results
Play Video
Matthew Flagg © 2005 2525
Virtual Rear Projection Results
System Component LatencyCamera Capture to PC-Memory 9.09ms
PC-Memory to GPU-Memory 1.70ms
Pixel Shaders 2.14ms
Projector 40.27ms
Total Latency 53.20ms
Image processing speed increased from 15Hz to 110Hz (camera capture rate), placing limit on the projector (85Hz refresh rate)
Projector latency accounts for 76% of total system latency! With occluder movement tolerance of 5cm, shadows are imperceptible up to 94 cm/sec (fast walking)
Matthew Flagg © 2005 2626
Conclusion Presented new approach to VRP
Occluder Detection in IR spectrum All processing moved to GPU
2 System Challenges Met Display Image Quality Shadow Perception Avoidance
Shadows eliminated fast enough to accommodate walking
Matthew Flagg © 2005 2727
Future Work Explore hardware solutions
Recent results show an LCD projector having ½ the latency of a DLP and LCOS
User Study VRP currently used in Collaborative Design Lab in School of
Aerospace Engineering Replicate laboratory evaluation of passive VRP with new active
VRP system Improve Image Quality
Better seam blending