3D Registration 3D Fusion

3D photography course schedule

Topic

Feb 21 Introduction

Feb 28 Lecture: Geometry, Camera Model, Calibration

Mar 7 Lecture: Features & Correspondences

Mar 14 Project Proposals

Mar 21 Lecture: Epipolar Geometry

Mar 28 Depth Estimation + 2 papers

Apr 4 Single View Geometry + 2 papers

Apr 11 Active Ranging and Structured Light + 2 papers

Apr 18 Project Updates

Apr. 25 --- Easter ---

May 2 SLAM + 2 papers

May 9 3D & Registration + 2 papers

May 16 SfM/Self Calibration + 2 papers

May 23 Shape from Silhouettes + 2 papers

May 30 Final Projects (if not demo day)

3D modeling

• Aligning range images

• Pairwise

• Globally

• Surface reconstruction

• Single range image

• Merged

(some slides from S. Rusinkiewicz, J. Ponce,…)

Aligning 3D Data

• If correct correspondences are known, it is possible to find correct relative rotation/translation

q = a +

q is a quaternion,

a R is its real part, and

R3 is its imaginary part.

• Sum of quaternions:

• Multiplication by a scalar:

• Quaternion product:

• Conjugate: )

Operations on quaternions:

• Norm:

Note:

Intermezzo: quaternions

Let R denote the rotation of angle about the unit vector u.

Define

Then for any vector ,

Reciprocally, if q = a + ( b, c, d )T is a unit quaternion,

the corresponding rotation matrix is:

Intermezzo: quaternions and rotations

Problem: Find the rotation matrix R and the vector t that

minimize

i=1

E = | xi’ – R xi – t |2 . n

At a minimum: 0 = E/ t = –2 ( xi’ – R xi – t ) . n

i=1

Or.. t = x’ – R x.

If yi = xi –x and yi’ = xi’ –x’, the error is (at a minimum):

i=1

E = | yi’ – R yi |2

n

i=1

= | yi’ q – q yi |2

n

i=1 = | yi’ – q yi q|2|q|2

n

Or E =

Linear least squares !!

Estimate rigid transformation

Aligning 3D Data

• How to find corresponding points?

• Previous systems based on user input, feature matching, surface signatures, etc.

Spin Images

• Johnson and Hebert

• “Signature” that captures local shape

• Similar shapes similar spin images

Computing Spin Images

• Start with a point on a 3D model

• Find (averaged) surface normal at that point

• Define coordinate system centered at this point, oriented according to surface normal and two (arbitrary) tangents

• Express other points (within some distance) in terms of the new coordinates

Computing Spin Images

• Compute histogram of locations of other points, in new coordinate system, ignoring rotation around normal:

n̂ p n̂ p

n̂ p n̂ p

“radial dist.”

“elevation”

Computing Spin Images

“radial dist.”

“elevation”

Spin Image Parameters

• Size of neighborhood

• Determines whether local or global shape is captured

• Big neighborhood: more discriminatory power

• Small neighborhood: resistance to clutter

• Size of bins in histogram:

• Big bins: less sensitive to noise

• Small bins: captures more detail

Alignment with Spin Image

• Compute Spin Image for each point / subset of points in both sets

• Find similar spin images => potential correspondences

• Compute alignment from correspondences

Same problems as with image matching:

- Robustness of descriptor vs. discriminative power

- Mismatches => robust estimation required

Solving 3D puzzles with VIPs

SIFT features

• Extracted from 2D images

• Variation due to viewpoint

VIP features

• Extracted from 3D model

• Viewpoint invariant

15

(Wu et al., CVPR08)

Presented afterwards !

Aligning 3D Data

Alternative: assume closest points correspond

to each other, compute the best transform…

Aligning 3D Data

… and iterate to find alignment

Iterated Closest Points (ICP) [Besl & McKay 92]

Converges if starting position “close enough“

ICP Variant – Point-to-Plane Error Metric

• Using point-to-plane distance instead of point-to-point lets flat regions slide along each other more easily [Chen & Medioni 91]

Finding Corresponding Points

• Finding closest point is most expensive stage of ICP

• Brute force search – O(n)

• Spatial data structure (e.g., k-d tree) – O(log n)

• Voxel grid – O(1), but large constant, slow preprocessing

Finding Corresponding Points

• For range images, simply project point [Blais 95]

• Constant-time, fast

• Does not require precomputing a spatial data structure

Efficient ICP

• “Efficient Variants of the ICP algorithm” [Rusinkiewicz & Levoy, 3DIM 2001]

=> Presented afterwards

3D Global Registration

3D Global registration The problem:

• Given: n scans around an object

• Goal: align them all

• First attempt: ICP each scan to one other

• Want method for distributing accumulated error among all scans

3D Global registration

Approach #1: Avoid the Problem

• In some cases have 1 scan that covers large part of surface (e.g., cylindrical scan)

• Align all other scans to this “anchor”

• Disadvantage: not always practical to obtain anchor scan

Approach #2: The Greedy Solution

• Align each new scan to all previous scans

• Disadvantages:

• Order dependent

• Doesn’t spread out error

Approach #3: Brute-Force Solution

• While not converged:

• For each scan:

• For each point:

– For every other scan

» Find closest point

• Minimize error w.r.t. transforms of all scans

• Disadvantage:

• Solve (np)(np) matrix equation, where n is number of scans and p is number of points per scan

Approach #3a: Slightly Less Brute-Force

• While not converged:

• For each scan:

• For each point:

– For every other scan

» Find closest point

• Minimize error w.r.t. transform of this scan

• Faster than previous method (matrices are pp)

Graph Methods

• Many globalreg algorithms create a graph of pairwise alignments between scans

Scan 1 Scan 1 Scan 5 Scan 5

Scan 4 Scan 4

Scan 3 Scan 3

Scan 2 Scan 2 Scan 6 Scan 6

Pulli’s Algorithm

• Perform pairwise ICPs, record sample (e.g. 200) of corresponding points

• For each scan, starting w. most connected

• Align scan to existing set

• While (change in error) > threshold

• Align each scan to others

Sharp et al. Algorithm

• Perform pairwise ICPs, record only optimal rotation/translation for each

• Decompose alignment graph into cycles

• While (change in error) > tolerance

• For each cycle:

• Spread out error equally among all scans in the cycle

• For each scan belonging to more than 1 cycle:

• Assign average transform to scan

Lu and Milios Algorithm

• Perform pairwise ICPs, record optimal rotation/translation and covariance for each

• Least squares simultaneous minimization of all errors (covariance-weighted)

• Requires linearization of rotations

Open Questions in Global Registration

• Best way to do correctly-weighted globalreg without linearizing rotations?

• How to prevent bias (if many scans in one area, few scans in another)?

• Robust outlier detection

• One bad ICP can throw off the entire model

Surface Reconstruction

Problems With Reconstruction from General Point Clouds

Surface Reconstruction from Range Images

• Often an easier problem than reconstruction from arbitrary point clouds

• Implicit information about adjacency, connectivity

• Roughly uniform spacing

Surface Reconstruction From Range Images

• First, construct surface from each range image

• Then, merge resulting surfaces

• Obtain average surface in overlapping regions

• Control point density

Range Image Tesselation

• Given a range image, connect up the neighbors

3D surface model

Depth image Triangle mesh Texture image

Textured 3D

Wireframe model

Range Image Tesselation

• Caveat #1: can’t be too aggressive

• Introduce distance threshold for tesselation

• Caveat #2: Which way to triangulate?

• Possible heuristics:

• Shorter diagonal

• Dihedral angle closer to 180

• Maximize smallest angle in both triangles

• Always the same way (best triangle strips)

Range Image Tesselation

Scan Merging Using Zippering

• Turk & Levoy, 1994

• Erode geometry in overlapping areas

• Stitch scans together along seam

• Re-introduce all data

• Weighted average

Zippering

Point Weighting

• Higher weights to points facing the camera

• Favor higher sampling rates

Point Weighting

• Lower weights (tapering to 0) near boundaries

• Smooth blends between views

Volumetric Reconstruction

• Implicit function defined volumetrically

• Usually stored sampled on a 3D grid

• Can be compressed (e.g., using RLE)

• Another possibility: hierarchical data structures

• Can extract isosurface (i.e., subset of space where implicit function = some constant)

Volumetric Reconstruction Overview

• Generate signed distance function (or something close to it) for each scan

• Compute average (possibly weighted)

• Extract isosurface

Volumetric integration (Curless and Levoy, Siggraph´96)

sensor

range surfaces

volume

distance

depth

weight (~accuracy)

signed distance

to surface

surface1

surface2

combined estimate

• use voxel space

• new surface as zero-crossing

(find using marching cubes)

• least-squares estimate

(zero derivative=minimum)

Volumetric Reconstruction Benefits

• Always generates a manifold surface

• Can control sampling density

• Averaging of signed distance functions corresponds to averaging the surfaces

Volumetric Reconstruction Drawbacks

• Represent a 3D entity rather than 2D

• Running time

• Storage

• Resampling step – bandlimits the function

• Generates consistent topology, but not always the topology you wanted

• Problems with very thin surfaces

From volume to mesh: Marching Cubes

“Marching Squares” in 2D

“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

From volume to mesh: Marching Cubes

“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

Multiple depth images Volumetric integration

Volumetric 3D integration

Presentations

• Efficient Variants of the ICP algorithm

• Model Matching with Viewpoint Invariant Patches