The space of human body shapes:reconstruction and parameterization

from range scans

Brett AllenBrian CurlessZoran Popović

University of Washington

Motivation Traditional high level characterization

and sparse anthropometric measurements do not capture the detailed shape variations needed for realism

PS: Sheldon, W.H. ,Stevens, S.S.,Tucker, W.B. 1940 The varieties of Human Physique ,New York

MotivationTraditional anthropometry has focused on

sets ofone-dimensional measurements.

MotivationFull body shape capture promises to advance

the state of the art.

?

CAESARCivilian American & European Surface Anthropometry Resource

• thousands of subjects in the U.S. and Europe• traditional anthropometry• demographic survey• laser range scans

We’ll use 250 of these scans (125 male, 125 female).

Related work• Praun, E.,Sweldens, W,,Schroder, P. 2001 Consistent Mesh Parameterization (In proceeding of ACM SIGGRAPH 2001)

• Lee, A.W.F.,Moreton, H.,Hoppe, H. 2000 Displaced Subdivision Surfaces (In proceeding of ACM SIGGRAPH 2000)

• Blanz, V.,Vetter, T. 1999 A Morphable Model For The Synthesis Of 3D Faces (In proceeding of ACM SIGGRAPH 1999)

Scan detail

~250,000 trianglesincomplete coverage

surface color74 markers

Scan detail

Caused by occlusions and grazing angle views

After template-based parameterization and hole filling

The Correspondence Problem

Consistent Mesh Parameterization

Emil PraunWim SweldensPeter Schroder

PrincetonBell Lab

Purpose and Problem Specification

Algorithm Input Set of meshes S Feature points F defined

on each mesh M

Algorithm Goal Determine common

base domain B and connectivity L0

Remesh each M with base domain B

Create fair patch boundaries equivalent to L0

Algorithms

Topologically Equivalent Nets

Definition: A patch is a region of semi-regular connectivity in which triangles correspond to a single triangleDefinition: A net is the outline of patch boundariesWe want a net that matches the connectivity L0

Two patch boundaries may only intersect at a feature vertex

Each feature vertex has a consistent cyclical ordering of edges

Patch boundaries may not intersect

Naïve algorithm (shortest path) does not achieve this

Restricted Brush Fire Algorithm

Standard Brush Fire Starts fire at vertex Fire expands uniformly until it hits other vertex That is the shortest path

Restricted Brush Fire Uses existing paths as firewalls Will only connect to vertex if approached from correct

ordering

Must avoid blocking off vertices Avoid completing any cycle until spanning

tree of L0 traced

Fair Boundary CurvesRather than simply accept topologically equivalent net, we would like certain properties

Equal distribution of surface area between patches Smooth Patch Boundaries Fair patch boundaries (should not swirl)

First two can be handled using relaxation Iterative technique that involves progressively

improving curve

Third requires optimization of complicated expression

Intractable, so try heuristics

Swirl Detection

If the shortest path from {c} to {a,b} falls on the wrong side (left) ,the triangle {a,b,c} is considered flipped and may lead to swirls .

On the right the trace reaches {a,b}on the correct side and the path is accepted.

Implementation

Example

Example

Matching algorithm

scantemplate

Find the shape that:

1. Matches the template markers to the scanned markers

2. Moves template vertices to scanned vertices

3. Minimizes the deformation

Match Algorithm

1. Applied a set of affine transformation Ti to the vertices of the template surface T and result in a new surface T’

2. Minimizing three error term(data error, smoothness error, marker error)

Matching Algorithm

Objective Function

Objective Function has three weighted terms Data error Smoothness error Marker error

Will use different weights in each phase of process Multistep / Multi-resolution fitting process

Objective Function – Data Error

Data Error term requires current match to be close to target Uses distance from each transformed vertex to the

target surface Weighted by confidence measure (from scanning) Hole regions have weight = 0 Sums total error

Distance function Uses transformed template vertex Takes minimum distance to “compatible” vertices in

target

Objective Function – Smoothness Error

Measures smoothness of deformation applied to template Problem is under-constrained using data error E_s measures change in T_I between adjacent vertices Encourages similarly-shaped features to be mapped to

each other

Uses Frobenius norm (vector L2 norm)

Objective Function – Smoothness Error

補充 ; Frobenius norm

|| A ||f =

Smoothness Error

By smoothness, we are not referring to smoothness of the deformed surface itself, but rather smoothness of the actual deformation applied to the template surface. In particular, we require afine transformations applied within a region of the surface to be as similar as possible.

Objective Function – Marker Error

Data and Smoothness Error can hit local minima Example: left arm transformed to right arm

Solution: Use pre-labeled markers on the test subjects Viewed as white dots in the range image Correspondences set up beforehand (as in Consistent

Mesh Parameterization) 74 markers per subject (not all are used, however)

Measure distance from template marker to target marker K_I are the indices of the markers in template, m_I are

target markers

Algorithm ProcedureMinimize error function using L-BFGS-B algorithm

Quasi-Newton method with limited memory usage

Make four passes over data (2 low res, 2 high res) Fit markers (low res, = 0, = 1, = 10) Refit using data term (low res, = 1, = 1, = 10) Repeat in high resolution (hi res, = 1, = 1, = 10) Refit using predominantly data term (hi res, = 10, = 1, = 1)

Hole Filling

PCA (Principal component analysis)

The vectors with low variance can be discarded ,and then the full data set does not to be retained in order to closely approximate the original example

Feature analysis

Provide a direct way to explore the range of bodies with intuitive controls,such as height,weight,age and sex

Feature analysis

Markerless Matching

Using PCA, we can remove the marker requirement A set of training data is fit using markers Other data can be registered using unmarked range

scans Uses PCA weights to search PCA space, not

transformation space

Markless-only

We can use PCA to remove the range imagesOnly use markers; can be captured with much cheaper equipmentAllows us to determine approximate shape of object

Application

1.Transfer of Texture

Application

1.Morphing between model

Application

1.Instrumentation transfer

Statistical analysis

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

average male mean + PCA component #1

mean + PCA component #2

Statistical analysis

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

average male

Statistical analysis

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

x0

y0

z0

x1

y1

z1

x2

average male mean + PCA component #3

PCA reconstruction

Fitting to attributes

Fitting to points Using the distribution of the PCA weights as a prior, we

can find the most likely person that fits a set of point constraints.

PCA variance

user constraint

optimized reconstruction

Summary

Contributions: - an algorithm for creating a consistent

mesh representation from range scan data.

- several ways to explore the variation in human body shape, and to synthesize and edit body models

Future work

- analyze shape variation between poses

Future work

- combine with anatomical models and physical simulation

Aubel 2003

+