real-time video effects using a depth sensor robbie fraser and ridho jeftha
TRANSCRIPT
REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR
Robbie Fraser and Ridho Jeftha
The ProblemVideo effects:
• Difficult
• Expensive
• Time consuming
The SolutionApplication:
• Automated
• Real time
AutomationDepth sensor
• Capture colour with camera (RGBD)
• Capture depth with sensor (RGBD)
Research QuestionCan image based effects be applied to video
footage in real-time?
AimDevelop a system to:
• Capture RGBD from a camera and depth sensor
• Render a variety of different video effects
Test the system:
• Benchmark performance
• Evaluate the viability of real-time video effects
System Design
Mapping SystemPrepare the data
• Depth map
• Point cloud
• Normal map
Refine the data
• Real time input
• Reflectance map
Video Capture
Normal Map GenerationDepth Map to Normal Map
Normal Map GenerationDepth Map to Normal Map
Reflectance Map
• Estimate initial lighting conditions
• Use normals
• Estimate reflectance at each point
Map Refinement
Map Refinement
Effects SystemFour effects:
• Fog Synthesis
• Scene Relighting
• Depth of Field
• Cartoon Shading
Fog Synthesis
Scene Relighting
Depth of Field
Cartoon Shading
User Interface
• Interface with cameras
• View effects
• Edit parameters
PerformanceUtilise graphics processors
• Fragment shaders
• CUDA
Testing
• Benchmarko Different effectso Different parameterso Different resolutions
• Performance profiling
Success CriterionPerformance
• Entire process at 25fps
Work Allocation
Thank YouQuestions?