dynamic refraction stereo
DESCRIPTION
Dynamic Refraction Stereo. 1. Dynamic Refraction Stereo Problem. Goal: Reconstruct instantaneous height map & normal map of time-varying refractive surface projecting to 2 cameras Calibrated & synchronized cameras - PowerPoint PPT PresentationTRANSCRIPT
Dynamic Refraction Stereo
7. Contributions• Refractive disparity optimization gives stable reconstructions regardless of surface shape• Require no geometric assumptions other than single light refraction• Produces a full resolution height map and a separate, full-resolution normal map• Highly-detailed reconstructions for a variety of complex, deforming liquid surfaces
Kiriakos N. Kutulakoskyros @ cs.toronto.edu
Nigel J. W. Morrisnmorris @ dgp.toronto.edu
8. Experimental results
Experiments with dynamic water surfaces
t=0.6s
Hei
ght m
apT
ilt m
ap
Instantaneous 3D reconstructionst=0.92s t=1.32s
Simulations vs. Ground-truth Experiments Refractive index estimation
Observed results (solid red) vs simulated results (dotted blue) for 0.08 pixel localization error for set of planar refractive surfaces at various heights
Total reconstruction error as function of refractive index (liquid was water)
Hei
ght m
apT
ilt m
apH
eigh
t map
Tilt
map
t=0.92s
t=0.3s
t=0.91s
4. Refractive Stereo Algorithm
2. Related WorkSingle-view methods (require extra assumptions/optics):
• Shape from Distortion (Murase, PAMI 1992) assumes constant mean distance
• Shape from Refraction (Jähne et al., JOSA 1994)
uses collimating lens & lighting gradient
Multi-view methods:
• Sanderson et al. (PAMI 1998), Bonfort & Sturm (ICCV 2003)
static mirror scenes (known refractive index), optimization degrades for shallow liquid heights
• Multi-media photogrammetry (Flach & Maas, IAPRS 2000)
known parametric shape model
3. The Refraction Stereo Constraint
Key insight: For a generic refractive surface, knowledge of pixel q & pattern point C(q) defines a constraint curve in the 3D space of possible normals & refractive index values
• Height hypothesis in Camera 1
correspondence in Camera 2
• q & C(q) in Camera 1
1st constraint curve
• q’ & C(q’) in Camera 2
2nd constraint curve
• Generically, curves do not intersect for incorrect height hypotheses
can resolve height, normal, refractive index
Goal: Reconstruct instantaneous height map & normal map of
time-varying refractive surface projecting to 2 cameras
• Calibrated & synchronized cameras
• Arbitrary, time-varying surface shape (i.e. no prior shape models)
• Refractive index unknown
• Calibrated pattern at known 3D position, visible through liquid
• Known 1-1 mapping from pixels q to points C(q) on pattern
1. Dynamic Refraction Stereo Problem
p
n
Cam1 (time t)
Cam2(time t)
q
C(q)
• Discretize interval of possible refractive index values.For each value, do Steps 1-4:
Step 1: Initialize correspondence function C(.) for time 0
Step 2: For each time t and each pixel q in Camera 1
2a: (Refractive disparity optimization) 1D optimization along ray of q, searching for height hypothesis consistent with both viewpoints
2b: (Bundle adjustment) 5D optimization of p and n using reprojection error
Step 3: Fuse depth & normal map to obtain 3D surface
Step 4: Update correspondence function for next frame
• Choose refractive index value minimizing total reconstruction error (across all frames and pixels)
5. 1D Optimization: Refractive Disparity
• Large liquid heights: both criteria are 0 when p on ‘true’ surface
• As height → 0
estimation of n1, n2 unstable ||d1||2 + ||d2||2 remains stable (reduces to standard stereo)
minimize angle(n1,n2)
p
n1 n2
minimize ||d1||2 + ||d2||2
Bonfort & Sturm Refractive Disparity
n1n2n2n1
d1 d2
n1n2
p
nq
C(q)
q’
C(q’)
6. Computing Correspondence C(q)
Iteratively compute flow between un-refracted & refractedviews using Lucas-Kanade (Baker & Matthews, IJCV 2004)
Refracted view (time t)Un-refracted view (reference)
t=0.3s t=0.35s t=0.4s
t=0.75s t=0.83s t=0.91s