High-quality Scanning usingTime-Of-Flight Depth Superresolution
17nd March 2008
Sebastian [email protected]
Prepared for: Final Presentation CS223B, Winter 2008
Sebastian Schuon [email protected] Project_26.ptt 2
Problem StatementTime-of-Flight (TOF) Cameras Have a Low Resolution
ZCam by 3DV Systems (Israel)
Measurement principle: time of flight
Depth recording: 320x240, 8bit
► Less noisy depth images with higher resolution desired
Native resolution Superresolution (4x) Geometry renderingfrom superresolution
Sebastian Schuon [email protected] Project_26.ptt 3
ApproachCombine Several Images to Increase Resolution
Use multiple, here N=15, recordings from different viewpoints by translating the camera
Estimating the high resolution image resembles to an optimization problem
Optimization is multi-objective: similarity and smoothness is enforced
►
Sebastian Schuon [email protected] Project_26.ptt 4
ResultsSubtle Details Become Visible and Noise is Reduced
De
pth
ma
p3
D g
eo
me
try
ren
de
ring
Native resolution Superresolution (4x)
Sebastian Schuon [email protected] Project_26.ptt 6
Hidden SlideDistribution of Work
Project solely undertaken by Sebastian Schuon (me)
Supervision / Collaboration:Christian TheobaltJames Davis (UCSC)
Additional imagery for paper provided byHylke Buisman
Sebastian Schuon [email protected] Project_26.ptt 7
SuperresolutionGoals
Enhance resolution
Reduce noise
Recording Resolution
(320 x 240)
(Contrast enhanced)
(Contrast enhanced)
Super Resolution(4x upsample)
Sebastian Schuon [email protected] Project_26.ptt 8
Depth CameraTheory
ZCam by 3DV Systems (Israel)
RGB and Depth camera in one housing (“RGBD”)
RGB: 640x320 @ 30fps, Depth: 320x240, 160x120 @ 30fps
Measurement principle: time of flight
Depth image: distance between camera and object (not Z-coordinate)
Unprojection to 3D coordinates necessary
Specification of tracking window required
Sebastian Schuon [email protected] Project_26.ptt 9
Depth CameraResults
Recording scene with different layers of depth
Black, shiny surfaces tend to be problematic
One needs to know where to record (tracking window)
Unprojection of depth images leads to 3D representation
Sebastian Schuon [email protected] Project_26.ptt 10
Depth CameraNoise Characteristic
Outer regions tend to be a lot more noisy
Noise can be approximated with Gaussian
Hypothesis: Noise increases quadratic with distance
Hypothesis : Noise is correlated with color of object recorded
Variance Plot Pixel Distribution over Time at Center
Sebastian Schuon [email protected] Project_26.ptt 11
Depth CameraSystematic Bias
RBG Processing disabled
Variance Plot (RGB disabled)
RBG Processing enabled
Variance Plot (RGB enabled)
Sebastian Schuon [email protected] Project_26.ptt 12
SuperresolutionTheory
Based on Shift-And-Add family of algorithms
Quite well studied for grayscale and color images, overview in [Farsiu04]
We used Bilateral Shift-And-Add [Farsiu03]
Formulation as inverse problem
Our approach: rotating camera
Sebastian Schuon [email protected] Project_26.ptt 13
SuperresolutionResults Simple Approach
Depth Image - Superresolution(Contrast enhanced)
Depth Image - Recording Resolution(Contrast enhanced)
3D Rendering - Recording Resolution
3D Rendering - Superresolution
Sebastian Schuon [email protected] Project_26.ptt 14
Conclusion
Findings
Camera Interface and Software still beta / undocumented
Interesting effects can happen, that are not expected
Superresolution on depth images is feasible
Further steps
Reimplementation of known algorithms
Ideas for improvement:– Depth camera specific noise characteristic– Incorporating confidence map