BY

TH

OM

AS

FU

NK

HO

US

ER

,M

ICH

AE

LK

AZ

HD

AN

,P

AT

RIC

KM

IN,

AN

DP

HIL

IPS

HIL

AN

E

58 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

The number of 3D geometric modelsavailable in online repositories is growing dramatically. Examples include:the Protein Data Bank [1], which storesthe 3D atomic coordinates for 29,000protein molecules; the National DesignRepository [9], which stores 3D com-puter-aided design (CAD) models fortens of thousands of mechanical parts;and the Princeton Shape Database [5],which stores polygonal surface modelsfor 36,000 everyday objects crawledfrom the Web. Since graphics hardwareis getting faster and 3D scanning hardware cheaper, there is every reason to believe that demand for andsupply of 3D models will continue toincrease into the future, leading to an

Just like Google searches online text documents, emerging systems now search repositories of 3D surface models with queries describing geometric properties.

Shape-Based Retr ieval andAnalysis of 3D M odels

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 59

r ieval andM odels

WA F F L E WA R P (P LY W O O D, S T E E L P L AT E,F I B E R G L A S S) P R O D U C E D F O R A C L A S S

C A L L E D TH I C K SK I N: CAD/CAM TR A N S L AT I O N S I N T H E DE PA RT M E N T

O F AR C H I T E C T U R E AT T H E UN I V E R S I T Y

O F CA L I F O R N I A, BE R K E L E Y, 2003. PR O F. LI S A IWA M O T O A N D S T U D E N T S

MI C H A E L EG G E R S, EN R I Q U E SA N C H E Z,PA M E L A TR E E T I P U T, A N D DA N N Y LE E.

PR O T O T Y P E P R O D U C E D O N A

3D P R I N T E R.

60 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

online environment in which 3D models are asplentiful as images, videos, and audio files today.

Large digital repositories of 3Dmodels help create demand forsearch engines that are able toretrieve the data of interest andfor data mining algorithms todiscover relationships amongthem. Text annotation is almost

always helpful, but content-based methods are oftenrequired to discover novel geometric relationships inthe data. For example, a mechanical engineer mightuse a search engine to find a particular CAD model ina parts catalog based on its 3D shape characteristics; a

doctor might use an automatic classification system toaid diagnosis of a disease from the shapes of afflictedorgans; and a paleontologist might use shape analysisto link similar 3D models scanned from animal skele-tons of different species.

Such problems are common to other types of mul-timedia data, including sound, images, and video.However, databases of 3D models have several newand interesting characteristics that significantly affectshape-based retrieval and analysis algorithms. Unlikeother multimedia data types, most 3D models do notdepend on the configuration of sensors, emitters, orsurrounding objects. As a result, they do not containprojections, reflections, shadows, or occlusions,greatly simplifying identification of matches amongobjects of the same type. For example, it is plausibleto expect that the 3D model of a horse containsexactly four legs of roughly equal size. In contrast, any2D image of the same horse may contain fewer thanfour legs (if some of them are occluded by, say, tallgrass) or “extra legs” appear as the result of shadowson a barn or reflections in a nearby pond, or some ofthe legs appear smaller than others due to perspectivedistortions.

These problems are vexing when analyzing imagesbut absent from most 3D models. In other respects,representing and processing 3D models is more com-

plicated than for other types of multimedia. Themain difficulty is that 3D surfaces rarely have simpleparameterizations. Since 3D surfaces can have arbi-trary topologies, many useful methods for analyzingother media (such as Fourier analysis) have no obvi-ous analogues for 3D surface models. Moreover, thedimensionality is higher, making searches for poseregistration, feature correspondences, and modelparameters more difficult, while the likelihood ofmodel degeneracies is greater. As such, a specializedset of shape analysis methods is required for 3D data.

Here, we explore applications and technologies forshape-based retrieval and analysis of 3D models. Wesurvey motivating applications (see the sidebar“Shape Analysis Applications”), review shape repre-

sentations, present a casestudy for a shape-basedsearch engine for retrieval of3D models crawled from theWeb, and provide referencesto related projects and Websites. Our aim is to introduce

the problems in 3D shape analysis and provide a roadmap for possible solution methods.

SHAPE ANALYSIS METHODS

Methods for computer-aided shape analysis andretrieval are being pursued in several fields, includ-ing computer vision, computational geometry, andcomputer graphics. Most of this work has focusedon specific data types (such as CAD and rangescans). General-purpose methods have also beendescribed. Surveys can be found in [7, 12].

The primary challenge in building a shape-basedretrieval and analysis system is to find a computa-tional representation of shape (shape descriptors) forwhich an index can be built, similarity queriesanswered efficiently, and/or interesting features com-puted robustly. Shape-based descriptors should gener-ally be concise; efficient to compute, compare, andsearch; insensitive to noise, blur, cracks, and smallextra features; independent of 3D object representa-

Figure 1. Harmonic shapedescriptor. A robust, concise,rotation-invariant shape representation can be constructed from the amplitudes of spherical harmonic coefficients withinevery frequency (order) andspherical shell of a grid.

tion, tessellation, topology, or genus; invariant totransformations (such as scales, translations, rotations,mirrors, or articulations); and representative of keyshape features.

Most of all, they should enable an efficient algo-rithm to compute the shape properties of interest to aparticular application domain. Unfortunately, it is

unlikely that any single shape descriptor will satisfy allthese properties, as there are usually trade-offsbetween computational expense and discriminationpower. The spectrum of shape descriptors ranges fromthose that are simple to compute (but perhaps notvery discriminating) to those that require expensivecomputations (but provide sophisticated shape analy-

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 61

3D shape analysis is important in many applicationdomains, including molecular biology, chemistry, forensics,mechanical CAD, military target recognition, medicine, paleon-tology, computer graphics, computer vision, entertainment, andart. The related challenges (see the figure) are similar for eachand can be characterized by the following basic problem types:

Registration. Given two 3D models, alignthem optimally; for example, an archeolo-gist might want to align 3D scans of exca-vated pots in order to visualize theirsimilarities and differences;

Matching. Given two 3D models, com-pute the geometric similarity between them;for example, a doctor might want to com-pare a pair of MRI scans to characterize thedifferences between tumors scanned in dif-ferent patients or the same tumor scannedin a single patient at different times;

Retrieval. Given a database of 3D mod-els and a geometric query, find the modelsthat best match the query; for example, akitchen designer might search a catalog forfurniture that best matches a set of geo-metric constraints imposed by an oddlyshaped room;

Recognition. Given a database of 3D models and a querymodel, either find the query model in the database or deter-mine it is not there; for example, a criminal investigator mightsearch a database of 3D face images scanned from knowncriminals to identify the ones that best match surveillancecamera images;

Verification. Given a 3D model and a specification, deter-mine whether they match to within some tolerance; for exam-ple, a machinist might want to detect defects in manufacturedparts passing a laser scanner on a conveyer belt;

Clustering. Given a database of 3D models, automaticallypartition them into a set of classes; for example, a paleontolo-gist might derive insight into the process of evolution by clus-tering 3D models of fossils scanned from different species;

Feature detection. Given a 3D model, find geometric fea-tures of interest on its surface; for example, an art historianmight want to detect marks from a particular type of chisel on

the scanned surface of a famous statue; Classification. Given a set of model class specifications

and a query model, determine the class to which the querymodel belongs; for example, a molecular biologist mightclassify new protein structures based on geometric compar-isons to others with known function;

Segmentation. Partition a given 3D model into its salientparts; for example, an animator might want assistance inbuilding an articulated “skeleton” model to be used for ani-mating a human character model acquired through a full-body laser range scanner;

Semantic labeling. Infer semantic meaning regarding thepurpose and function of a given 3D model; for example, amesh-processing system might want to recognize importantsemantic features of a 3D model in order to avoid destroyingthem when the model is simplified or compressed; and

Synthesis. Automatically synthesize new examples typical of a given model class specification; for example, byexamining 3D surfaces of several office chairs, a computerprogram might be able to develop a parameterized modelfor the geometric arrangements of their parts, then helpautomate construction of new chairs by assembling partsautomatically. c

Funkhauser figure (6/05)

Query3D Model

Databaseof

3D Models

AnnotatedModel

ShapeIndex

ShapeDescriptor

Features,Skeletons

SalientParts

OK or Not

MatchingObject

SimilarObjects

MatchingClass

ClassSpecification

NovelObjects

AlignedObjects

SemanticLabeling

ObjectSegmentation

ObjectVerification

ObjectRecognition

ObjectRetrieval

ObjectClassification

ObjectSynthesis

ObjectRegistration

ShapeAnalysis

IndexConstruction

Clustering and Learning

ShapeAnalysis

SHAPE ANALYSIS APPLICATIONS

sis). They are sometimes integratedinto a retrieval system organized as apipeline of conservative filters,where the early ones quickly cullirrelevant 3D models from consid-eration, and the later ones providemore discriminating shape analysisfor smaller subsets of the remainingcandidates.

At the low end of the spectrum, shapes can be rep-resented by their statistical properties. The simplestapproach describes a shape as a feature vector wherethe axes of a multidimensional feature space encodeglobal geometric properties (such as genus, algebraicmoments, circularity, eccentricity, and compactness)[3]. Other methods utilize histograms of geometricstatistics (such as the distances between points and theangles between lines and planes). Yet others charac-terize statistics of frequency decompositions forshapes for, say, storing the amplitudes of sphericalharmonic coefficients to provide invariance to rota-tions (see Figure 1). These statistical methods are gen-erally robust, concise, quick to compute, indexable,and effective for characterizing large-scale shape fea-tures. But they also often provide invariance to a lim-ited range of transformations (such rigid bodytransformations), are not invertible, and fail to cap-ture subtle semantic details that might be valuable for discriminating among slightly different classes ofshapes (such as doors with keyholes vs. doors withoutkeyholes).

At the high end of the spectrum, shape can be rep-resented by model-based structures that capture thegeometric relationships among the key parts of anobject [11]. For example, hierarchical structures canbe used as templates for which their fit to a shape pro-vides useful information for recognizing or classifying

articulated shapes. Alternatively, a shape canbe decomposed into its parts automaticallythrough algorithms that decompose its sur-face along concave seams, compute anapproximation to its medial axis, or approxi-mate its volume with a set of simple primi-tives (such as covering it with ellipsoids).These methods are generally good at captur-ing the topology of an object but are almostalways time-consuming to compute, overlysensitive to small features, and difficult tomatch and/or index for retrieval applications.

In the middle of the spectrum are aplethora of approaches, including view-basedrepresentations that describe the shape of a3D object by “how it looks” in a set of 2Dprojections from different views [2] and

point-based methods that describe the local shapearound a multitude of sample points [4]. Eachmethod involves its own benefits and costs associatedwith different application domains. Comparing theseshape representations is difficult [10], especially sincethe quality of the results is subjective. However,lower-level statistical shape descriptors are generallymore successful in applications involving recognition,matching, and retrieval, while higher-level model-based shape representations are better suited for seg-mentation, semantic labeling, and synthesisapplications.

RETRIEVING 3D MODELS

Perhaps the most important application of 3D shapeanalysis is the retrieval of 3D models available onthe Web. Several such search engines have recentlybeen introduced for indexing 3D data for computergraphics, molecular biology, and mechanical CAD.One of the earliest was the Princeton 3D ModelSearch Engine, which went online in 2001 andtoday indexes 36,000 computer graphics modelscrawled from the Web. It was developed to help peo-ple looking for 3D polygonal surface models (suchas cars and humans)—in other words, a Google for3D models.

The Princeton search engine is available as a Webpage (shape.cs.princeton.edu) in which the user entersqueries, and the system retrieves the best-matching 3Dmodels from a Web-crawled repository, presentingthem as a ranked set of thumbnail images (see Figure2). If the user clicks on any returned thumbnail image,the system produces a pop-up Web page with detailedinformation and close-up images of the associated 3Dmodel, along with a link for downloading the 3Dmodel file from its original Web address.

Since different types of objects are best described

62 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

Figure 2. Princeton3D Model SearchEngine. The user

draws a 3D sketch(left), and the sys-

tem returns 3Dmodels with similar

shape from itsrepository.

through different query types, the system supportsmany different query interfaces. Perhaps the simplestis a text box (see Figure 2 upper left) in which the userenters keywords (such as “car’’), and the systemretrieves 3D models with associated text containingthem (such as matching words in the file name, thereferring Web page, and the scene graph node names).Alternatively, the user can sketch 2D outlines of thedesired object, possibly as seen from multiple view-points; the system then returns 3D models whose pro-jections match the sketches. Finally, the user canspecify a query with a 3D shape, either by uploadingan existing 3D model, sketching a coarse 3D modelfrom scratch, as in Figure 2, or clicking on the “FindSimilar Shapes’’ link under any previously retrievedthumbnail image. The system then matches the shapeof the 3D query to the models in the repository andreturns the best matches.

Each of these query types requires its own indexingand retrieval strategy. Here, we consider the 3D shapequeries (see [5] for details on the others) where theproblem is: Given a repository of 3D models, build anindex that can be searched efficiently to find the clos-est matches to an arbitrary 3D query model. Theproblem is difficult because computer graphics mod-els found on the Web can appear in any coordinatesystem, with arbitrary degeneracies (disconnected,

overlapping, and intersecting polygons)and widely varying shapes within thesame class of objects. Yet matches mustbe found within a second or so to providean interactive response to every query.Thus, robustness, transformation invari-ances, and speed are the primary con-cerns when choosing a shaperepresentation for this applicationdomain.

The search engine uses a statisticalshape descriptor representing the shapeof each 3D model by a 2D feature vectordescribing “how much shape” resideswithin each spherical harmonic fre-quency and each distance from the mod-el’s center of mass, as in Figure 1 [6]. Thisrepresentation is insensitive to meshdegeneracies, since the 3D function fromwhich spherical harmonic coefficients arecomputed depends only on distancefrom the surface, not on the tessellationof the mesh. The representation is alsoinvariant to the model’s orientation; thus,comparison of the descriptors can be per-formed without aligning the models apriori or searching over all possible rota-

tions. Third, it provides the important theoreticalguarantee that the Euclidean distance between twodescriptors provides a lower bound on the Euclideandistance between the corresponding 3D models. Theguarantee means there are no false negatives when thedescriptor is used for narrowing down a set of candi-date matches to be examined later in greater detail.Finally, it enables efficient indexing based on nearest-neighbor search structures. Empirically, for queries tothe search engine, the system takes less than one sec-ond to find the 16 best matches in a database with36,000 3D structures. It retrieves better matches thanmany larger and computationally more expensivedescriptors.

The search engine has deployed these query inter-faces and retrieval methods [8], so far receiving770,000 queries from 160,000 different hosts and130 different countries and been slashdotted twice.Thus, it is now possible to characterize how the sys-tem is used in vivo. As you might expect, the major-ity of queries (65%) include only text keywords; textis the most familiar type of query for most users andthe quickest to enter. However, one-third of thequeries have incorporated some sort of shape, mostoften as “find similar shape’’ (13% of all queries) and2D sketches (16% of all queries).

While it is difficult to say which query types users

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 63

SSHHAAPPEE--BBAASSEEDD RREETTRRIIEEVVAALL RREESSOOUURRCCEESS

For more on shape-based retrieval and analysis, explore the following projects and resources available on the Web:

ARIZONA STATE UNIVERSITY. 3D Knowledge, Acquisition, Representation, and Analysis, 3dk.asu.edu/

CARNEGIE MELLON UNIVERSITY. 3D Model Retrieval Project, amp.ece.cmu.edu/projects/3DModelRetrieval/

DREXEL UNIVERSITY. National Design Repository, www.designrepository.org/

HERIOT-WATT UNIVERSITY. ShapeSifter, www.shapesearch.net/ IBM. 3D Geometry Search Technology Project, www.trl.ibm.com/

projects/3dweb/SimSearch_e.htm INFORMATICS AND TELEMATICS INSTITUTE. 3D Search, 3d-search.iti.gr/ NATIONAL INSTITUTE OF MULTIMEDIA EDUCATION, JAPAN. Ogden,

www.nime.ac.pj/~motofumi/Ogden/ NATIONAL RESEARCH COUNCIL CANADA. Nefertiti, www.cleopatra.nrc.ca/ PRINCETON UNIVERSITY. 3D Model Search Engine, shape.cs.princeton.edu/ PURDUE UNIVERSITY. 3DESS, tools.ecn.purdue.edu/~cise/dess.htmlTAIWAN UNIVERSITY. 3D Model Retrieval System, 3d.csie.ntu.edu.tw/ UNIVERSITY OF KONSTANZ. 3D Model Similarity Search Project, dbvis.inf.

uni-konstanz.de/research/projects/SimSearch3D/UTRECHT UNIVERSITY. 3D Shape Recognition Project, www.cs.uu.nl/

centers/give/multimedia/3Drecog/index.html

find most effective, we have found that text and shapequeries augment each other in controlled user studies[5]. People tend to be most effective when startingout with a text query (to retrieve an example from adesired class of objects), then using “find similarshape’’ to find other models of the same type thatwere not well-annotated. Overall, users seem to findthe search engine a useful tool for quickly identifyingand retrieving 3D models on the Web.

CONCLUSIONShape-based retrieval and analysis tools are provinguseful in a number of application domains, mostnotably computer graphics, mechanical computer-aided design, and molecular biology. Looking for-ward, these tools will be increasingly important, as3D data acquisition hardware becomes more of acommodity, and more people begin to make and use3D models in their everyday lives. It will then beeasier to find and retrieve an existing model made bysomeone else than it will be to make a new one fromscratch yourself. New methods will soon be

deployed to perform robust feature detection, partialshape matching, part decomposition, and eventuallycomplete semantic labeling. 3D models will thenprovide not only raw 3D geometry and surfaceattributes but smarts regarding their composition,how they move, and how they are used—all keys tomaking 3D data a more useful part of the informa-tion revolution.

References1. Berman, H., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T., Weissig,

H., Shindyalov, I., and Bourne, P. The Protein Data Bank. NucleicAcids Research 28 (2000), 235–242.

2. Chen, D.-Y., Ouhyoung, M., Tian, X.-P., and Shen, Y.-T. On visualsimilarity-based 3D model retrieval. Comput. Graphics Forum 22, 3(Sept. 2003), 223–232.

3. Corney, J., Rea, H., Clark, D., Pritchard, J., Breaks, M., and MacLeod,R. Coarse filters for shape matching. IEEE Comput. Graphics Applic.22, 3 (May/June 2003), 65–74.

4. Frome, A., Huber, D., Kolluri, R., Bulow, T., and Malik, J. Recogniz-ing objects in range data using regional point descriptors. In Proceed-ings of the European Conference on Computer Vision (Prague, May11–14). Springer-Verlag, 2004, 224–237.

5. Funkhouser, T., Min, P., Kazhdan, M., Chen, J., Halderman, A.,Dobkin, D., and Jacobs, D. A search engine for 3D models. Transact.Graphics 22, 1 (Jan. 2003), 83–105.

6. Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. Rotation invariantspherical harmonic representation of 3D shape descriptors. In Proceed-ings of the Symposium on Geometry Processing (Aachen, Germany, June2003), 156–164.

7. Loncaric, S. A survey of shape analysis techniques. Pattern Recognition31, 8 (Aug. 1998), 983–1001.

8. Min, P., Halderman, J., Kazhdan, M., and Funkhouser, T. Early expe-riences with a 3D model search engine. In Proceedings of the EighthInternational Conference on 3D Web Technology (St. Malo, France, Mar.9–12). ACM Press, New York, 2003, 7–18.

9. Regli, W. and Gaines, D. A repository of designs for process and assem-bly planning. Computer Aided Design 29, 12 (Dec. 1997), 895–905.

10. Shilane, P., Min, P., Kazhdan, M., and Funkhouser, T. The Princetonshape benchmark. In Proceedings of Shape Modeling International(Genoa, Italy, June 7–9). IEEE, 2004, 167–178.

11. Sundar, H., Silver, D., Gagvani, N., and Dickinson, S. Skeleton-basedshape matching and retrieval. In Proceedings of Shape Modeling Interna-tional (Seoul, Korea, May 12–15). IEEE, 2003, 130–142.

12. Tangelder, J. and Veltkamp, R. A survey of content-based 3D shaperetrieval methods. In Proceedings of Shape Modeling International(Genoa, Italy, June 7–9). IEEE, 2004, 145–156.

Thomas Funkhouser (funk@cs.princeton.edu) is an associateprofessor in the Department of Computer Science at Princeton University, Princeton, NJ.Michael Kazhdan (misha@cs.jhu.edu) is an assistant professor inthe Department of Computer Science at John Hopkins University,Baltimore, MD.Patrick Min (min@cs.princeton.edu) is an assistant professor inthe Department of Computer Science at John Cabot University,Rome, Italy.Philip Shilane (pshilane@cs.princeton.edu) is a graduate studentin the Department of Computer Science at Princeton University,Princeton, NJ.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. To copy otherwise, to republish, to post on servers or to redis-tribute to lists, requires prior specific permission and/or a fee.

© 2005 ACM 0001-0782/05/0600 $5.00

c

64 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

A mechanical engineer might use a search engineto find a particularCAD model in aparts catalog basedon its 3D shape characteristics.