example based 3d shape completion mark pauly 1,2, niloy j. mitra 1, joachim giesen 2, markus gross...

Example Based 3D Shape Completion

Mark Pauly1,2, Niloy J. Mitra1, Joachim Giesen2, Markus Gross2, Leonidas J. Guibas1

1 Stanford University 2 ETH, Zurich

Example Based 3D Shape Completion

Shape Completion

• Incomplete raw scans

• Imperfect range scanned data• Complex objects with occluded

regions

• Misalignment of multiple-views depth image scans

• Specular highlights

Ill-posed problem

Use prior knowledge !!

Example Based 3D Shape Completion

Diffusion (Davis, Marschner, Garr, Levoy, 2002)

Example Based 3D Shape Completion

Context Based Solution (Sharf, Alexa, Cohen-or, 2004)

Example Based 3D Shape Completion

Template based Solution (Allen, Curless, Popovic, 2003; Kraevoy and Sheffer, 2005)

Example Based 3D Shape Completion

Our Solution

• Use 3D model database to provide geometric priors for shape completion

• Apply non-rigid transforms on the models• More deformation less likely completion

• Consistently combine geometric information from multiple context models

• Final result comes with confidence values

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Data Classification

Local analysis

• quality of fit

• uniformity of sample distribution

Scored Point Cloud

• confidence value assigned to each point

High

Low

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Database Retrieval

1.93 1.71 1.46 1.27 1.0

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Non-rigid Alignment

Similar to the approaches proposed by:• Allen, Curless and Popovic, 2003.

• Sumner and Popovic, 2004.

Example Based 3D Shape Completion

Non-rigid Alignment

Deformation Model

• Piecewise linear.Each vertex of the mesh assigned an independent displacement vector.

Optimize for smallest Shape Matching Penalty

• Distortion Measure

• Geometric Error

Feature Correspondence

Derived in the continuous setting to allow consistent comparison between different context models.

Example Based 3D Shape Completion

Warped Models

High

Low

Context Model Warped Model Matching Penalty

Example Based 3D Shape Completion

Non-rigid Transform

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Initial Segmentation

Input Data Warped Context Model

Example Based 3D Shape Completion

Patch Growing

Initial Segmentation Final Segmentation

Example Based 3D Shape Completion

Video

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Blending Final Segments

Example Based 3D Shape Completion

Result

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Shape Completion Pipeline

Example Based 3D Shape Completion

Segmentation

Giraffe Example

Context Models

Deformed Models

Example Based 3D Shape Completion

Final Model

Giraffe Example

Context Models

Deformed Models

Example Based 3D Shape Completion

Evaluation

Input Data Context Model Final Model Evaluation

Example Based 3D Shape Completion

Enriching the Database

Example Based 3D Shape Completion

Additional Constraints

Physical Model Acquired Data Context Model

No Constraints Symmetry Constraints

Example Based 3D Shape Completion

Galleria dell’ Accademia

Example Based 3D Shape Completion

Galleria dell’ Accademia

Example Based 3D Shape Completion

Galleria dell’ Accademia

Example Based 3D Shape Completion

Galleria dell’ Accademia

Example Based 3D Shape Completion

Future Directions

• Improve the retrieval stage.

• Automatic feature point detection.

• Use of more semantic information.

• Apply learning techniques to shape completion.

• Completion of additional attributes like surface texture, color.

Example Based 3D Shape Completion

Acknowledgements

• NSF grants CARGO-0138456, ITR-0205671, FRG-0454543, ARO grant DAAD19-03-1-033.

• Stanford Graduate fellowship.

• Mario Botsch, David Koller, Doo Young Kwon, Marc Levoy, Filip Sadlo, Vin da Silva, and Bob Sumner.

Example Based 3D Shape Completion

Questions

Example Based 3D Shape Completion

Example Based 3D Shape Completion

Correspondence

High

Input Data Warped Model Low

Invalid

Valid

Example Based 3D Shape Completion

Video

Example Based 3D Shape Completion

• Example-based image completion[Drori et al. 2003; Jia and tang 2003; Sun et al. 2003]

• Texture synthesis[Efros and Leung 1999; Efros and Freeman 2001;

Wei and Levoy 2000; Ying et al. 2001]

• Smooth surface completion

[Curless and Levoy 1996; Davis et al. 2002;

Ilic and Fua 2003; Verdera et al. 2003; Liepa 2003]

• Model-based surface reconstruction [Savchenko et al. 2002]

• Curve analogies [Hertzmann et al. 2002]

Drori et al. 03

Liepa 03

Wei and Levoy 00

Hertzmann et al. 02

Related Works