Reconstruction with Depth and Color cameras for 3D

Autostereoscopic Consumer Displays

SAIT – INRIA collaborationPeriod: 15 July 2012 / 15 January 2013

Date: 3-4 December 2012

INRIA team

• Georgios Evangelidis, postdoc, 100%• Michel Amat, development engineer, 100%• Soraya Arias, senior development engineer, 20% • Jan Cech, posdoc, 20%• Radu Horaud, 10%

Past achievements

• A method and software for aligning TOF data with a stereoscopic camera pair

• Extension to the calibration of several TOF-stereo units

• 3D texture-based rendering of the TOF data using the color-image information

Publications

• One CVPR 2011 paper• A tutorial at ICIP 2011• One Springer Briefs book

just published• The two teams published

several other papers

Current achievements

• Finalization of the calibration & rectification methods/software

• TOF to stereo-pair mapping with filtering• TOF + texture in live mode• Disparity map initialization• Stereo correspondence based on seed-growing• Final high-resolution depth map with gap filling• A paper submitted to CVPR’13

Improved Calibration• New calibration board– mat sticker glued to a

rigid plane– plane attached to a tripod

• Refined Calibration algorithm– TOF-Stereo Calibration error: <1.5 pixel

• Improved Rectification– Rectification error:

<0.25 pixel

Given a calibrated TOF-Stereo system

• Each TOF point PT defines a correspondence between PL and PR

Correspondences (samples) obtained by using the calibration parameters

• each correspondence comes from a TOF point• different color -> different depth

Correspondences (samples) obtained by using the calibration parameters

• each correspondence comes from a TOF point• different color -> different depth

TOF-to-Left Mapping

• We use the left image as reference

TOF-to-Left Mapping is not perfect

Resolution mismatch

Left-to-Tof Occlusions

TOF-to-Left Mapping is not perfect

Left-to-Tof Occlusions: the depth decreases from left to right

Tof-to-Left Occlusions

TOF-to-Left Mapping is not perfect

Tof-to-Left Occlusions: the depth increases from left to right

Point Cloud filtering• We reject points in left-to-tof occluded area• We keep the minimum-depth points in case of

overlap (due to Tof-to-left occlusions)

Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point

cloud

Disparity Map: Initialization• Run Delauney-Triangulation on low-resolution point

cloud…• …and initialize the stereo disparity map It looks good,

but it’s noisy and non-accurate!

Seed-Growing Idea• Start from points with known disparities

(seeds) and propagate the disparity to neighboring points (video?)

• Main issues:– What are our seeds?– What is the visiting order of seeds?– How do I propagate the message?– How the stereo and depth data are fused within

this framework?

Depth-Color Fusion • Built on the seed-growing idea– A:Depth data, S: Stereo data, dN : neighbor of d– For each pixel (node), find its disparity value that

maximizes the posterior probability (MAP)

SS

AA

d N

dInput data

Pixel with unknown disparity

Range-search constraintPenalize the choice

wrt to depth information

Penalize the choice wrt to color information

Pixel with known disparityA represents the initial estimation of d (obtained by the previous interpolation)S represents the color matching cost that corresponds to d

Depth-Color Fusion• Bayes rule translates each posterior into a

likelihood

• If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme

SS

AA

d N

d

Input data

Pixel with unknown disparity

Pixel with known disparity

We are currently working on these

terms!

Because of the uniform distribution

Because of theBayes rule

Seed-Growing Idea (revisited)• For each pixel, an energy function is defined and we

look for its minimizer (disparity)• Main issues:– What are our seeds?

• the points from Tof-to-Left mapping after refinement – What is the visiting order of seeds?

• First visit reliable seeds (points with low energy value)– How do we propagate the message?

• Given the disparity of a seed, bound the disparity-range for its neighbor

– How the stereo and depth data are fused within this framework?• Described above

Examples

White areas: unreliable matches

Black areas: Occlusions

Examples (with gap filling)

Examples (with gap filling)

Examples (with gap filling)

Examples (with gap filling)

Paper Submission

• Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR 2013.

• An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.

Work during the remaining month

• Improve the accuracy of the matching by better exploiting the color/texture information

• Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second

• An updated version will be available at the end of the period (~15 January 2013)

Prospects for the next collaboration(1 February 2013 – 31 January 2014)

• Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas

• Depth disambiguation using TOF-TOF and TOF-stereo • Combine depth disambiguation with the seed-growing

algorithm• Perform full 3D realistic rendering with four TOF-

stereo units• Perform continuous 3D reconstruction with a moving

TOF-stereo unit