ieee transactions on pattern analysis and machine intelligence, vol. x, no. xx...

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 1

The Geometry of Reflectance SymmetriesPing Tan, Member, IEEE, Long Quan, Fellow, IEEE, and Todd Zickler Member, IEEE

Abstract—Different materials reflect light in differentways, and this reflectance interacts with shape, lighting, andviewpoint to determine an object’s image. Common materialsexhibit diverse reflectance effects, and this is a significantsource of difficulty for image analysis. One strategy fordealing with this diversity is to build computational toolsthat exploit reflectance symmetries, such as reciprocity andisotropy, that are exhibited by broad classes of materials. Bybuilding tools that exploit these symmetries, one can createvision systems that are more likely to succeed in real-world,non-Lambertian environments. In this paper, we developa framework for representing and exploiting reflectancesymmetries. We analyze the conditions for distinct surfacepoints to have local view and lighting conditions that areequivalent under these symmetries, and we represent theseconditions in terms of the geometric structure they induceon the Gaussian sphere and its abstraction, the projectiveplane. We also study the behavior of these structures underperturbations of surface shape and explore applications toboth calibrated and un-calibrated photometric stereo.

Index Terms—Reflectance symmetry, projective geometry,auto-calibration, photometric stereo.

I. INTRODUCTION

AN image of an object is determined through com-plex interactions between its reflectance, shape, and

surrounding environment. In order to invert this processand recover scene information, vision systems rely onsimplified models of image formation, and these often in-clude reduced models of surface reflectance. One commonapproach is to assume that surfaces are Lambertian, orperfectly matte. According to the Lambertian model, thebi-directional reflectance distribution function (BRDF) isa constant function of the viewing and illumination direc-tions; and by assuming that surfaces are well-representedby this simple model, one can build powerful tools forstereo reconstruction, shape from shading, motion estima-tion, segmentation, photometric stereo, and so on.

Most surfaces are not Lambertian, however, so we oftenseek ways of generalizing these powerful Lambertian-based tools. One possibility is to assume that non-Lambertian phenomena, such as specular highlights, are

P. Tan is with the Department of Electrical and Computer En-gineering, National University of Singapore, Singapore 128044. E-mail:[email protected].

L. Quan is with the Department of Computer Science and En-gineering, HKUST, Clear Water Bay, Kawloon, Hong Kong. E-mail:[email protected].

T. Zickler is with the School of Engineering and Applied Science,Harvard University, 33 Oxford Street, Cambridge, MA 02138. E-mail:[email protected].

restricted to small regions of an image, so that they canbe treated as outliers or ‘missing data’. Another approachis to model these non-Lambertian phenomena using para-metric reflectance models that are more complex than theLambertian model. This has the important advantage ofusing all available image data, but it also has a significantlimitation: Even relatively simple reflectance models (e.g.,Phong [1]), severely complicate image analysis, and sincethey are only accurate for limited classes of materials, thisapproach generally requires new and complex analysis foreach application and each material class.

A third approach to dealing with complex surfacereflectance is to exploit more general properties. This isthe approach that we follow here, and its central thesisis that even though there is a wide variety of materialsin the world, there are common reflectance phenomenaexhibited by broad classes of these materials. By buildingcomputational tools to exploit these properties, one cancreate vision systems that are more likely to succeed inreal-world environments. One successful example of thisapproach is the dichromatic reflectance model [2], whichprovides the means to exploit the fact that additive diffuseand specular reflectance components of non-conductingmaterials are spectrally distinct.

In this paper, we focus on reflectance symmetries. Ageneral BRDF is a function of four angular dimensions—two for each of the input and output directions—andmany materials exhibit symmetries over this 4D domain.Almost all materials satisfy reciprocity ( [3], p. 231), forexample, which states that the BRDF is unchanged whenthe input and output directions are exchanged; and manymaterials satisfy isotropy, according to which the BRDFis unchanged for rotations about the surface normal ofany input/output direction-pair. These symmetries inducejoint constraints on shape, lighting and viewpoint, andexisting work suggests they can be exploited for visualtasks such as 3D reconstruction (e.g., [4], [5]). The goalof this paper is to provide a framework for understandingthese symmetries and exploiting them more broadly.

We advocate studying reflectance symmetries in termsof the geometric structures they induce on the Gaussiansphere. An image of a curved surface under point-sourcelighting contains observations of distinct surface pointswith symmetrically-equivalent local view and lightingdirections, and these equivalences induce geometric struc-tures on the field of surface normals. We show that byrepresenting these structures on the Gaussian sphere and


Fig. 1. The generalized bas-relief ambiguity [7] is a shape/lightingambiguity that exists for Lambertian surfaces (left). Reflectance symme-tries resolve this ambiguity for any surface that has an additive specularreflection component that is isotropic and spatially-uniform (right).

its abstraction, the projective plane, we obtain conciseand intuitive descriptions of the symmetries, as well asconvenient tools for applying them to vision problems.

To demonstrate the utility of the proposed framework,we use it to develop new techniques for both calibratedand un-calibrated photometric stereo. In uncalibrated pho-tometric stereo, we prove that constraints induced byisotropy and reciprocity can resolve the generalized bas-relief ambiguity (Fig. 1) and provide two practical al-gorithms for doing so. In the calibrated case, we showthat isotropy and reciprocity constraints can be used torecover Euclidean structure from images captured undera known, view-centered cone of light sources. This isachieved by improving the partial reconstruction providedby the method of Alldrin and Kriegman [6].

II. BACKGROUND AND RELATED WORK

At an appropriate scale, the reflectance of opaque andoptically-thick materials is described by the bi-directionalreflectance distribution function, or BRDF [8]. The BRDFis a positive function of four angular dimensions and iswritten f(ωi,ωo), where ωi and ωo are the directionsof incident and reflected flux, respectively. As mentionedin the introduction, many materials exhibit reflectancesymmetries. Reciprocity guarantees that the BRDF is sym-metric about the input and output directions: f(ωi, ωo) =f(ωo, ωi). In many cases, the BRDF is unchanged byrotations of the input and output directions (as a fixedpair) about the surface normal and by reflections of theoutput direction across the incident (input/normal) plane.Materials that satisfy these two symmetries are said to beisotropic and bilaterally-symmetric, respectively [9]. It isalso common to use the term isotropic to mean both, andwe will do so here.

When one or more of these symmetries is apparent,radiance measurements that are captured at symmetrically-equivalent local view and illumination directions mustbe equal, and this induces joint constraints on shape,viewpoint and illumination. This has been exploited, forexample, for surface reconstruction using isotropy [10],

bilateral symmetry [6], [11], and reciprocity [12]. Theadvantage of such symmetry-based approaches is that theyavoid the use of low-parameter BRDF models (Lamber-tian, Lafortune [13], Ward [14], Cook-Torrance [15], etc.)that have limited accuracy [16], [17] and often introducenon-linearities that severely complicate vision problemsthat are already ill-posed.

These symmetries are traditionally represented in a lo-cal (normal-defined) coordinate system by parameterizingthe BRDF domain in terms of halfway and differenceangles [18]. Accordingly, the complete 4D domain iswritten in terms of the spherical coordinates (θh, φh) ofthe halfway vector h = (ωi + ωo)/||ωi−ωo||, and thoseof the input direction with respect to the halfway vector,(θd, φd). Then, the folding due to reciprocity correspondsto φd → φd+π, and the projection due to isotropy (withoutbilateral symmetry) is one onto (θh, θd, φd). Bilateralsymmetry enables the additional folding φd → φd + π/2which gives the 3D domain (θh, θd, φd) ⊂ [0, π/2]3.

For vision problems, it is more useful to describethese symmetries in a global, camera-centered coordinatesystem, and that is what we do here. Building on earlierversions of our work [11], [19], we begin by examiningthese symmetries on the hemisphere of surface normalsthat is visible to an orthographic camera (i.e., the visibleportion of the Gaussian sphere), and then we show that theprojective plane provides a useful abstraction of the result-ing structure. In the next, we explore their applications inphotometric stereo.

In photometric stereo, we seek to infer three-dimensional shape from multiple images recorded from afixed viewpoint under multiple illuminations. Photometricstereo techniques can be either ‘calibrated’ or ‘uncal-ibrated’ depending on whether the lighting conditionsare known or unknown a priori. The simplest formula-tion of the problem assumes Lambertian reflectance andcalibrated acquisition [20]. When the lighting directionsare uncalibrated, the surface shape can only be recov-ered up to certain shape ambiguity [7], [21], [22] evenwhen the surface is Lambertian. Many works have beenproposed to investigate when and how this ambiguity isresolved, which often assume parametric non-Lambertianreflectance models [23]–[25]. Similar parametric modelsare also used in calibrated photometric stereo (e.g., [26]–[30]) to handle general non-Lambertian surfaces. However,these models are only valid for limited types of surfaces.Further, they are often non-linear and complicate theoriginal problem. In comparison, we apply the geometricconstraints derived from reflectance symmetries which arevalid for broad class of surfaces. We show these constraintscan resolve various shape ambiguities in both calibratedand uncalibrated photometric stereo with simple geometricanalysis.


V S

n

n`

d2

d1

Fig. 2. On the visible hemisphere of Gauss sphere, two normals forman isotropic pair if they lie at the intersections of two circles centeredat source direction s and view direction v. If the BRDF is isotropic (butotherwise arbitrary) the observed intensity at these points will be equal.

III. STRUCTURE ON THE GAUSSIAN SPHERE

We restrict our attention to surfaces observed by an idealorthographic camera under directional lighting, and weignore the global effects of mutual illumination and castshadows. This means that the camera’s measurements ofscene radiance can be completely described in terms ofthe relative orientation of each local surface patch, andour analysis can be performed on the Gaussian sphere. Fornow, we also assume that the surface has uniform materialproperties so that the BRDF is the same everywhere (thisis relaxed in Sect.VI.)

Consider the orthographic observer to be fixed in direc-tion v ∈ S2, and choose a global coordinate system so thatthe z-axis is aligned with this direction (i.e., v = (0, 0, 1)).Let s ∈ S2 represent the illumination direction. (We useboldface font to represent unit-vectors throughout thispaper.) Since the great circle through v and s will becomesignificant, we give it a name and refer to it as the principalmeridian.

A. Points and sets of points

We identify sets of surface points having equivalent locallight and view directions under isotropy and reciprocity.

Definition 1. Two surface normals n and n′ form anisotropic pair with respect to source s if and only if theysatisfy

n′>s = n>s and n′>v = n>v.

A geometric interpretation is shown in Fig. 2. Two nor-mals form an isotropic pair if they lie at the intersectionsof two circles centered at v and s on the Gaussian sphere.If we consider the local coordinate system for the BRDFdomain at each of two such surface normals, it is clearthat the incoming and outgoing elevation angles (θi andθo) are the same in both cases. This is because the tetrahe-dron formed by unit vectors {n, s, v} is equivalent underreflection across the principal meridian to that formed by{n′, s, v}. Also, if we were to project v and s onto the planeorthogonal to each normal, it is clear that the magnitudeof the angular difference between these projections (i.e.,|φi−φo|) would also be the same. It follows that when the

V S

m n

n`

d1

d2

d1

d2

m`

Fig. 3. On the visible hemisphere of Gauss sphere, two normalsform a reciprocal pair if they lie at the intersections of circles that arecentered at s and v and whose radii are exchanged. In this example, thereare four reciprocal pairs: m ↔ n, m′ ↔ n′, m ↔ n′, and m′ ↔ n.If the BRDF is isotropic and reciprocal, the observed BRDF value (butnot necessarily the radiance) at these points will be equal.

BRDF is isotropic, the radiance emitted from two pointsis equal if their normals form an isotropic pair.

Definition 2. Two surface normals n and m form areciprocal pair with respect to source s if and only if theysatisfy

m>s = n>v and m>v = n>s.

This condition can similarly be interpreted in termsof the intersections of circles centered at s and v. Theimportant difference is that the radii of the two circlesare swapped for the two normals. As depicted in Fig. 3,there are four possible intersections derived from allcombinations of circles with centers at s or v and havingtwo different radii. This family of four normals comprisestwo isotropic pairs ((m,m′) and (n,n′)) and four recip-rocal pairs ((m,n), (m′,n′), (m′,n) and (m,n′)). Unlikeisotropic pairs, not every point on the visible hemispherehas a visible point as its reciprocal partner. This is becausen>s < 0 implies m>v < 0, so m is not observed.

Using an argument similar to the isotropic case above, itis clear that the local light and view directions at any twonormals of a reciprocal pair are such that when a BRDF isisotropic and reciprocal, the observed BRDF value at twopoints is equal if their normals form a reciprocal pair. Itis important to realize that this statement does not referto emitted radiance, which is the BRDF multiplied by theforeshortening factor n>s.

According to our definitions, isotropic pairs are locatedsymmetrically with respect to the plane containing theprincipal meridian, and reciprocal pairs are located sym-metrically with respect to the plane perpendicular to theprincipal meridian and passing through the half-vector,or bisector, of v and s. With the exception of points onthe symmetry planes, most isotropic and reciprocal pairscontain two distinct normals.

Definition 1 induces an equivalence relation (i.e., onethat satisfies reflexivity, transitivity and symmetry), so wecan say that two normals are ‘isotropically equivalent’ ifthey form an isotropic pair. The conditions for a reciprocalpair do not induce an equivalence relation, however, as anormal is generally not reciprocal to itself. To form an


equivalence relation that exploits reciprocity, the condi-tions for isotropic and reciprocal pairs must be combined;we say that two normals are ‘equivalent under combinedisotropy and reciprocity’ if they form either an isotropicpair or a reciprocal pair. Thinking this way leads to thefollowing.

Definition 3. Four surface normals n,n′,m and m′ forma isotropic-reciprocal quadrilateral with respect to sources if both (n,n′) and (m,m′) are isotropic pairs and both(n,m) and (n′,m′) are reciprocal pairs.

According to this definition and our previous obser-vations, it is evident that when a BRDF is isotropicand reciprocal, the observed BRDF values at four pointsis equal if their normals form an isotropic-reciprocalquadrilateral.

B. Curves

It is worth defining two families of curves that are formedby the equivalence relations described above.

Definition 4. On the Gauss sphere, an isotropic curve withrespect to s is defined as

α1(s>x) + α2(v>x) = 0

for two constants α1, α2.1

An isotropic curve is a great circle, and it has thefollowing properties. First, if a normal is on an isotropiccurve, its isotropic corresponding normal is also on thecurve. (This can be easily verified by substituting Defini-tion 1 into the isotropic curve equation.) In other words, anisotropic curve is a union of isotropic equivalence classes.Second, the family of isotropic curves partitions the Gaus-sian sphere. Third and finally, due to the arrangementof isotropic pairs along the curve, when the BRDF isisotropic, the emitted radiance along an isotropic curveis a symmetric function when the curve is parameterizedby the signed angle to the principal meridian. Figure 7provides an example.

Definition 5. On the Gauss sphere, a reciprocal curve withrespect to s is defined as

α1(v>x)(s>x) + α2(v, s, x)2 = 0

for two constants α1, α2.2

Here the notation (x, y, z) = (x × y)>z denotes thescalar triple product. A reciprocal curve has properties thatare analogous to its isotropic counterpart. If a normal is

1In our previous work [11], this curve is defined as (s>x)+α(v>x) =0, which cannot include points on the curve v>x = 0.

2In our previous work [11], this curve is defined as (v>x)(s>x) +α(v, s, x)2 = 0, which cannot include points on the principal meridian(v, s, x) = 0.

on a reciprocal curve, all of its isotropic/reciprocal corre-sponding normals are on the same curve. So it is a unionof isotropic-reciprocal quadrilaterals. Also, the family ofthe reciprocal curves partitions the illuminated half ofthe Gaussian sphere (i.e., for which s>x > 0) . Finally,due to the symmetric arrangement of isotropic-reciprocalquadrilaterals within a reciprocal curve we observe thatwhen a BRDF is isotropic and reciprocal, we can choose aparameterization of a reciprocal curve such that the BRDFvalue along the curve is a symmetric function. One suchparameterization is given by the azimuthal angle betweennormal x and the plane perpendicular to the principalmeridian and containing the half-vector. Again, note thatthis statement involved the BRDF value, which is theemitted radiance divided by the foreshortening factor n>s.An example is shown in Figure 8.

IV. STRUCTURE ON THE PROJECTIVE PLANE

An alternative and arguably more powerful representationof these symmetry-induced geometric structures can beconstructed by considering the visible hemisphere as atwo-dimensional projective plane. As shown in Fig. 4, eachvisible normal n = (n1, n2, n3)> is considered a point inthe plane created by a gnomonic (or central) projectionof the unit hemisphere onto the tangent plane passingthrough the viewing direction v = (0, 0, 1)>. The planeis equipped with an elliptic metric: the distance betweenany two points is given by the angular difference betweenthe corresponding rays in R3.

Both points and lines are represented using homoge-nous three-vectors, for which we use the notation x =(x1, x2, x3), while noting that x ' αx represent the samepoint for any α ∈ R/{0}. (We use ' to indicate equalityup to scale.) As before, we use bold font to representnormalized vectors, so that x = x/||x||. We use thenotation vs , v×s to represent the principal meridian andthe notation h ' v + s for the half-vector. For illustrativepurposes, we assume that the source direction s lies inthe upper hemisphere and can be associated with a pointon the projective plane. This assumption can be easilyrelaxed, however, by applying our analysis to the half-vector associated with each source direction instead of thesource direction itself.

A. Points and sets of pointsWe first provide definitions for isotropic and reciprocalpairs in the projective plane. The equivalence of thesedefinitions is proved in Propositions 7 and 8 of Appendix.

Definition 6. [Alternative to Definition 1] Two surfacenormals n and n′ form an isotropic pair with respect tosource s if and only if they satisfy

(v× s)>(n + n′) = 0, (1)s>nv>n

=s>n′

v>n′. (2)


Fig. 4. The hemisphere of surface normals visible from direction v isrepresented by a plane obtained by gnomonic projection. This is the realprojective plane, where great circles map to lines and the equator maps toa line at infinity. Reciprocity, isotropy, and other reflectance symmetriescan be studied in terms of the geometric structures that they induce inthis plane.

Fig. 5. In projective plane, an isotropic-reciprocal quadruplet is anisosceles trapezoid centered at h with two sides perpendicular to vs andtwo sides intersect at p = h× (v× s).

Equations (1) and (2) can be interpreted geometricallyas saying that the principal meridian vs is the perpendic-ular bisector of the line segment nn′. Specifically, Eq. 1says that the mid-point of the line segment nn′ (given byn + n′3) lies on the principal meridian vs, and Eq. 2 saysnn′ is perpendicular to vs (see Proposition 9 of Appendix).This leads to the following remark, which is visualizedFigure 5, where both (n, n′) and (m, m′) are isotropicpairs.

Remark 1. Two points n,n′ on the projective plane forman isotropic pair if and only if the principal meridian vsis the perpendicular bisector of line segment nn′.

Definition 7. [Alternative to Definition 2] Two surfacenormals n and m form a reciprocal pair with respect tosource s if and only if they satisfy

s>(n + m) = v>(n + m), (3)s>(n×m) = v>(n×m), (4)

Geometrically, Eq. 3 says the line connecting pointn + m and the halfway vector h is perpendicular to theprincipal meridian (multiply the equation by v>(v + s) =s>(v + s) and apply Proposition 9 of Appendix). Equa-tion 4 says the line nm intersects the principal meridianat the point (v + s) × (v × s) = h × (v × s) (multiplythe equation by v>(v + s) = s>(v + s) and apply

3n + n′ is the bisector between n and n′. It is also the middle pointof nn′ in the sense of elliptic metric.

Proposition 10 in Appendix). Notably, this intersectionpoint is independent of m and n. From this we can statethe following.

Remark 2. Two points n,m on the projective planeform a reciprocal pair if and only if: 1) nm intersect theprincipal meridian vs at h × (v × s); 2) the join of thebisector of n and m and h (the bisector of v and s) isperpendicular to the principal meridian.

In Figure 5, both (n, m) and (n′, m′) are reciprocalpairs. Given any point n, its reciprocal correspondence mcan be determined as follows. First, find the point n + mas the intersection of two lines: 1) the join of points n andh× (v× s); and 2) the line through h that is perpendicularto the principal meridian. Then, using the elliptic metric,m is uniquely determined by n and n + m.

According to this discussion, the four normals(n, n′,m,m′) comprising an isotropic-reciprocal quadri-lateral are such that the principal meridian is the perpen-dicular bisector of both nn′ and mm′; and both nm andn′m′ intersect the principal meridian at point h× (v× s).The quadrilateral is thus an isosceles trapezoid, and thecenter of the quadrilateral is the half-vector h (because thediagonals of the quadrilateral intersect there). This leadsto the following.

Remark 3. Four points n,m,n′ and m′ on the projectiveplane form a isotropic-reciprocal quadrilateral if and onlyif they form an isosceles trapezoid centered at h, with twoedges perpendicular to the principal meridian vs and theother two edges intersecting vs at h× (v× s).

We will refer to the points c = h and p = h× (v× s)as the quadrilateral center and quadrilateral peak respec-tively.

B. Curves

The definition of an isotropic curve (Definition 4) can bere-written as

(α1s> + α2v>)x = 0, (5)

from which it is clear that this curve is a pencil (one-parameter family) of lines in the projective plane. Thispencil is formed by the linear combination of the two linesl1 = {x | s>x = 0} and l2 = {x | v>x = 0}, and sincethe coordinate system is such that v = (0, 0, 1)>, the linel2 is the line at infinity. The intersection of l1 and l2, i.e.v×s, is a common point of all isotropic curves, and sincethis point is a point at infinity (an ideal point), all isotropiccurves are parallel lines in the plane as shown in Figure 6.The common ideal point v×s is also dual to the principalmeridian vs, and hence all isotropic lines are perpendicularto the principal meridian. Hence, a pencil of lines areisotropic curves if and only if they are perpendicular tothe principal meridian.


Fig. 6. Curves of equivalence points in the projective plane. Theblack dash line is the line at infinity v. Isotropic curves (red lines)are parallel lines perpendicular to the principal meridian, i.e. goingthrough the common infinite point v × s. Isotropic curves partition thewhole projective plane. Reciprocal curves (green conics) are parabolassymmetric about the principal meridian and going through two commonpoints, s× (v× s) and v× (v× s). Reciprocal curves partition the halfplane lower the line s× (v× s). Points above this line have not visiblereciprocal corresponding point. The principal meridian vs is the pole ofthe infinite point v × s with respect to the reciprocal curves.

The definition of a reciprocal curve (Definition 5) isequivalent to

x>(α1(vs> + sv>) + α2(v × s)(v × s)>)x = 0, (6)

which is a pencil of conics in the projective plane. Thispencil is formed by linearly combining the two conicsc1 = {x | x>(vs> + sv>)x = 0} and c2 = {x | x>(v×s)(v×s)>x = 0}. These conics pass through two commonpoints, v × (v × s) and s × (v × s), which can beverified by substitution into Eq. 6. As one of these points,v×(v×s), is a point at infinity, these conics are parabolas( [31], p. 117), and these parabolas are symmetric aboutthe principal meridian. Since a conic is determined byfive points, and since all reciprocal curves pass throughtwo common points, a reciprocal curve is completelydetermined from any three points of an isotropic-reciprocalquadrilateral. In short, a pencil of conics are reciprocalcurves if and only if they are parabolas symmetric to theprincipal meridian and intersect the principal meridian atthe two common points (one of them is an infinite point).

Finally, we note that another way of expressing thelink between isotropic and reciprocal curves is througha pole-polar relationship. The common ideal point of allisotropic curves, v×s, is the pole of the principal meridianwith respect to any reciprocal curve; likewise, the principalmeridian is the polar of the common infinite point. Thisfollows from

(α1(vs> + sv>) + α2(v × s)(v × s)>)(v × s)= α2(v × s)||v × s||2 ' v × s.

V. TRANSFORMATIONS

The geometric structures and radiometric constraints de-scribed in previous sections exist whenever one observes a

curved surface with uniform, isotropic BRDF under direc-tional illumination. In the remainder of this paper, we ex-plore their application to three-dimensional reconstructionproblems, and in particular, to the problem of photometricstereo, where one seeks to infer three-dimensional shapefrom multiple images recorded from a fixed viewpointunder multiple illuminations.

In photometric stereo and some other photometry-basedreconstruction problems4, one encounter the situation inwhich the field of surface normals can be recovered up toa projective ambiguity. That is, instead of the true normalfield {nj}, we recover a transformed field {nj} that isrelated to the true field by an unknown linear transform:nj = A−>nj/||A−>nj ||, for unknown A ∈ GL(3).

In this section, we describe the behavior of oursymmetry-induced structures under these transformations.We see that the required radiometric relationships betweenisotropic and reciprocal pairs are destroyed by them, andthus, these radiometric relationships can be used to resolvethe reconstruction ambiguity.

A. General linear transformations

As motivation, consider the uncalibrated Lamber-tian photometric stereo problem, as formulated byHayakawa [21]. Given three or more images of a Lam-bertian surface under varying, but unknown, directionallighting, the field of surface normals and the source direc-tions can be recovered up to a 3× 3 linear transformationA that acts according to [21]

n = A−>n/||A−>n||, s = As/||As||, (7)

and inversely as

n = A>n/||A>n||, s = A−1s/||A−1s||. (8)

In the projective plane, these can be written much moreconveniently as

n ' A−>n, s ' As, and n ' A>n, s ' A−1s,

where the transformation can now be considered an ele-ment of the projective general linear group; A ∈ PGL(3).

We consider the scenario in which a normal field {nj}and set of sources {si} have been estimated from three ormore images {Iij}, and the estimated normals are in theorbit of the true normals {nj} under group PGL(3).

Definition 8. Transformation A preserves isotropic(resp. reciprocal) pairs with respect to s if for all pairs(n,n′), the pair being isotropic (resp. reciprocal) withrespect to source s implies the transformed pair (n, n′) =(A−>n, A−>n′) is isotropic (resp. reciprocal) with respectto transformed source si = As.

4This arises in far-field uncalibrated Helmholtz stereo [32], and itwould arise in Lambertian photogeometric reconstruction [33] if generalaffine cameras were used.


Within the group PGL(3), there are transformationsthat preserve isotropic and reciprocal pairs, and there areothers that do not. This means that constraints based onisotropy and reciprocity can be used to restrict the groupof allowable transformations and reduce the reconstructionambiguity. In the following, we say that two sourcedirections are in general position if they are distinct andnot coplanar with the view direction.

Proposition 1. Rotations about the view direction com-posed with changes in scale and a mirror reflection are theonly linear transformations that simultaneously preserveisotropic pairs with respect to two source directions ingeneral position.

Proof: First, from Definition 1 it is easy to verifythat isotropic pairs are preserved by all transformationsin the group, RSM say, of view-axis rotations composedwith changes in scale. To complete the proof we providean algorithm that, given an estimated normal field in theorbit of the true normal field under group PGL(3), findsa normal field in its orbit under subgroup RSM.

Suppose we have the ability to determine within theestimated normal field, pairs of normals that are imagesunder the unknown projectivity of isotropic pairs withrespect to the true source direction. (Such an ability willbe developed in Sec. VI.) Two such pairs (n1, n′1) and(n2, n′2) with respect to a single source direction define apoint on the line at infinity: n1n

′1×n2n

′2. Thus, from two

source directions in general position, two distinct pointscan be located on the line at infinity, and the projectiveambiguity can be reduced to an affine ambiguity by restor-ing the line at infinity (i.e., by finding a transformation thatmaps it to (0, 0, 1)).

Now, affine transformations preserve the mid-point of aline segment. The join of the mid-point of n1n

′1 with the

mid-point of n2n′2 is a line l perpendicular to n1n

′1. This

provides a pair of perpendicular directions, l and n1n′1.

From two lighting directions in general position, two suchperpendicular pairs can be identified, and this reduces theaffine ambiguity to a similarity which consists of a rota-tion, translation, scaling and a mirror reflection. Finally,similarities preserve the perpendicular bisector of a linesegment. As the original is on the perpendicular bisectorof n1n

′1, it can be determined by intersecting two such

perpendicular bisectors from two lighting directions non-coplanar with the view direction. Hence, the translationcan be resolved and the similarity is reduced to a rotationabout the origin (the view direction), a scale change and amirror reflection (about an arbitrary line passing the originin the projective plane).

Note that this proof provides an 8-normal algorithm(two isotropic pairs in each of two images) for reducinga projective ambiguity to a scale, a mirror reflection andview-axis rotation. In addition, if the principal meridian

vs is known for at least one image, the rotation ambiguitycan be resolved by rotating the perpendicular bisector ofan isotropic pair to align it with vs. Further, the axis ofthe mirror reflection is constrained to overlapping withvs. This mirror reflection is fully resolved if the principalmeridian is known for at least two images. However, thescaling cannot be resolved from the isotropic structures– principal meridian and isotropic pairs, because thesestructures are preserved by a uniform scaling.

Proposition 2. If the principal meridian is known inat least two images, the identity transformation is theonly linear transformation that simultaneously preservesisotropic-reciprocal quadrilaterals with respect to twosources in general position.

Proof: This proof presents an algorithm to resolvethe general linear ambiguity with the two known princi-pal meridians and two isotropic-reciprocal quadrilaterals.According to the discussion in last paragraph, the generallinear ambiguity can be simplified to a scaling from twoisotropic pairs in each image when the principal meridiansin both images are known. Next, we will show this scalingcan be resolved by an isotropic-reciprocal quadrilateral inone image. Suppose A = diag(λ, λ, 1) is the unknownscaling. The vertex n of the quadrilateral is transformedto A−>n. In comparison, the light source s is transformedto As. This suggests that λ can be determined by a 1Dsearch. If we gradually increase λ from 0 to +∞ (λ mustbe positive to ensure the surface facing to the camera),the quadrilateral is constantly expanding. Its center cmoves away from the origin on vs. However, the sources (and hence the half-vector h) moves toward the originon vs. According to the structure described in Sects. IV,h overlaps with c. Hence, λ can be uniquely determinedwhen these two points meet.

B. GBR Transformations

When a normal field obtained by photometric stereo isknown to be from a differentiable surface, one can imposean “integrability constraint” on the normals to reduce theambiguity. In this case, the normal field can be determinedup to the generalized bas-relief (GBR) ambiguity, whichis represented by linear transformations of the form [7]

G =

1 0 00 1 0µ ν λ

, µ, ν, λ ∈ R.

In the case of µ = ν = 0, this transformation is knownas the classic bas-relief ambiguity. In the projective plane,an element of the GBR group has the effect of a scalechange by λ composed with a translation by (−µ,−ν).The principal meridian is unchanged by a GBR trans-formation, so that vs = vs. Hence, a GBR ambiguity is


−90 −45 0 45 900

0.2

0.4

0.6

0.8

Theta (degree)

Inte

nsity

Intensity Function

Fig. 7. (Left) Green points represent isotropic pairs, which are arrangedsymmetrically about the principal meridian on an isotropic curve priorto a GBR transformation. These points are mapped to the yellow pointsby a GBR transformation, and are no longer symmetrically arrangedwithin the curve. (Right) Emitted radiance as a function of signed anglefrom the principal meridian along the isotropic curves before and afterthe GBR. Since isotropy is not preserved by the GBR, the symmetry inthe radiance function is lost.

resolved according to Propositions 2 if isotropic-reciprocalquadrilaterals can be identified in two images.

Proposition 3. A GBR transformation maps each isotropiccurve with respect to s ‘as a set’ to an isotropic curve withrespect to s.

Proof: Isotropic curves are a pencil of lines perpen-dicular to the principal meridian vs. As the GBR transfor-mation is a uniform scaling composed with a translation,the set of transformed lines are still perpendicular tovs = vs. Therefore, they are isotropic curves with respectto the transformed lighting direction s.

A GBR transformation maps isotropic curves toisotropic curves, but the relative positions of points withinthe curve is not preserved. As a result, it destroys the sym-metry of the intensity function along the curve. This fact isdemonstrated in Figure 7. This property will be used in thenext section to estimate an unknown GBR transformationfrom identified isotropic curves by establishing the lostintensity symmetry.

Proposition 4. A bas-relief transformation maps eachreciprocal curve with respect to s ‘as a set’ to a reciprocalcurve with respect to s.

Proof: Reciprocal curves are a pencil of parabolassymmetric about the principal meridian vs and intersectingit at two common points. The classic bas-relief transfor-mation is a uniform scaling by λ, and hence, the set oftransformed parabolas remain symmetric to vs, passingthrough two common points. Therefore, they are reciprocalcurves with respect to the transformed lighting directions.

Similar to the isotropic case, although reciprocal curvesare mapped as a set by the bas-relief transform, therelative positions of points within the curve is changed.Hence, the intensity symmetry along the reciprocal curveis lost after transformation. This property is illustrated inFigure 8 and later used to estimate the unknown bas-relief

0 90 180 270 3600

0.2

0.4

0.6

0.8

Theta (degree)

BR

DF

BRDF Function

Fig. 8. (Left) Green points represent reciprocal and isotropic pairs,which are symmetrically arranged on a reciprocal curve prior to a bas-relief transformation. These are mapped to the yellow points by a bas-relief transform and are no longer symmetrically located. (Right) TheBRDF as a function of position along the reciprocal curves before andafter the transformation. This function is initially symmetric, but thesymmetry is destroyed by the bas-relief transformation.

transformation from a reciprocal curve.

Proposition 5. There are at most four discrete GBRtransformations that preserve isotropic-reciprocal quadri-laterals with respect to a source.

Proof: The proof consists of an algorithm that, givena normal field in the orbit of the true normal field undergroup GBR, finds the true normal field. As shown in Fig-ure 9, suppose a quadrilateral nmm′n′ is identified whosepre-image nmm′n′ is an isotropic-reciprocal quadrilateralunder illumination s. The center and peak of this quadri-lateral are c and p respectively. According to the knowntransformed lighting direction s, we can also compute thecenter and peak as c′ = h′ w s + v and p′ = h′ × (v× s).The goal is then to find a GBR that maps c to c′ andp to p′. However, all of these four points depend on theGBR parameters and the lighting and normal directions aretransformed differently, which makes a geometric proofcomplicated.

Here, we provide an algebraic approach to proof theproposition. By isotropy and reciprocity, an isotropic-reciprocal quadrilateral (n, m, n′, m′) with respect to s,satisfies:

(v× s)>(n + n′)=0,

s>(n + m)=v>(n + m),s>(n×m)=v>(n×m).

Substituting n = G>n/||G>n||, s = G−1s/||G−1s||, weobtain three equations in the unknown parameters µ, ν, λ5:

(v, s, g)=C1, (9)s>(diag(λ2, λ2, 0) + gg>)s=C2, (10)

s>(diag(λ2, λ2, 0) + gg>)(n× m)=√

C2(g, n, m).(11)

Here, C1 = 12

((v,s,n)v>n + (v,s,n′)

v>n′)

, C2 = (s>n)(s>m)(v>n)(v>m)

are constants. diag(·) represents a 3× 3 diagonal matrix,

5We use the following fact to simplify the derivation. G>a ×G>b/||G>a× G>b|| = G−1(a× b)/||G−1(a× b)||.


Fig. 9. In the projective plane, the original isotropic-reciprocalquadruplet is transformed by a scaling followed with a translation. Theillumination direction is transformed separately.

(·, ·, ·) is the triple scalar product of its arguments, and gis the translation vector [−µ,−ν, 1]>. Equations 9-11 arelinear and quadratic equations about the GBR parameters.There are at most four solutions from them 6.

This proof provides an one-image algorithm to resolvethe GBR ambiguity. Similar to the case of multi-viewstereo, where the shape ambiguity is resolved by identify-ing the absolute conic, identifying the isotropic-reciprocalquadrilateral can resolve the ambiguity in photometricstereo.

VI. APPLICATIONS

A. Uncalibrated photometric stereo

Our objective in this part of the paper is to use isotropyand reciprocity to ‘autocalibrate’ photometric stereo with-out the restrictions of low-parameter BRDF models. Weconsider the case in which the spatially-varying BRDF ofa surface can be represented as linear combination of aLambertian diffuse component and an isotropic specularcomponent:

f(x, y, ωi,ωo) = f1(x, y) + f2(θi, θo, |φi − φo|), (12)

where (x, y) denotes a surface point. In this model,the diffuse component varies spatially (i.e., the surfacehas ‘texture’), while the specular component is homo-geneous. Given three or more images of a surface withreflectance of this form, one can obtain a reconstructionup to the GBR ambiguity using existing techniques fordiffuse/specular image separation (e.g. [34]), and thenapplying uncalibrated Lambertian photometric stereo withintegrability [22] to the diffuse component.

6We can solve for µ, ν from Equation 9-10 as functions of λ2. Thenwe substitute these solutions into Equation 11 to obtain a second orderequation about λ2. We can obtain two solutions of λ because it shouldbe positive to ensure the surface is visible. Each choice of λ will lead totwo possible configurations of µ and ν. Hence, there are four possiblesolutions in total.

Accordingly, we assume that we are given as inputa normal field {nj} and set of sources {si} that havebeen estimated from three or more images {Iij}, andthat the estimated normals are in the orbit of the truenormals {nj} under group GBR. In what follows, wepresent two methods for resolving the GBR ambiguityusing isotropy and reciprocity constraints derived fromthe specular component of the input images. The firstfollows from Proposition 5 and uses the constraints froma single image, but it has the disadvantage of requiringan exhaustive search over a two-dimensional parameterspace. The second follows from Proposition 2 whichemploys two images instead of one, and has the advantageof requiring an exhaustive search over only one dimension.

Constraints from a single image. According to Propo-sition 5, the GBR ambiguity is reduced to a discretechoice once we identify a quadrilateral nmn′m′ whosepre-image nmn′m′ is an isotropic-reciprocal quadrilateralwith respect to the pre-image of s. It still remains todiscuss how to identify such a quadrilateral in an inputimage.

Given an arbitrary normal n, we must locate the appro-priate corresponding normals n′ and m (computationally,m′ is not required.) These two normals can be locatedsequentially by making use of the fact that the BRDF (or,in the case of n′, the intensity) is equal to that at n. Wewill see that locating the first (isotropic) correspondencen′ resolves one degree of freedom in the GBR ambiguity,and locating the second (reciprocal) correspondence mresolves the other two. For this discussion, it is helpful tore-parameterize the translational portion (µ, ν) of the GBRin terms of components that are parallel and orthogonal tothe principal meridian. These components are

ν′ , s1µ + s2ν, (13)µ′ , s2µ− s1ν. (14)

Since the GBR includes only translation and scale, theisotropic match n′ must lie on the known line that passesthrough n and is orthogonal to the principal meridian. Thisknown line is the isotropic curve passing through n and isunchanged by the GBR according to the Proposition 3. Thematch n′ can be obtained as the intersection of this lineand the isophote on the projective plane (i.e., the equal-intensity contour) that contains n. Once n′ is determined,one translational component (µ′) is resolved as the valuethat takes the bisector of nn′ to the principal meridian.

Next, we locate the reciprocal match, m, and resolvethe remaining two degrees of freedom (ν′, λ) throughan exhaustive search over this 2D parameter space. Ahypothesis (ν′1, λ1) yields a hypotheses for the true source,the quadrilateral center and peak: s1, c1 = h1, p1 =h1× (v× s1). It also yields a hypotheses for the intensity


Fig. 10. Left: typical example of the objective function for the single-image method. The existence of local minima at which the functiontakes values close to the global minimum (white circle) means that anexhaustive search can be sensitive to noise. Right: typical example ofthe objective function for the two-image method. This objective functionhas a clear global minimum.

value at m, as

I(m) = I(n)m>sn>s

= I(n)n>vn>s

= I(n)λ||G−1(ν′, λ)s|| n>v

n>s.

We exhaustively check all points with this intensity (i.e.all points on the same equal-intensity contour as m). Foreach potential match m1 on this contour, the transformedquadrilateral center c2 and peak p2 can be re-estimatedby intersecting the perpendicular bisector of nn′ with theline m1n

′ and the line m1n respectively. These in turnprovide an estimation of (ν′2, λ2) by mapping c2, p2 backto c1, p1 with a translation and scaling. If the hypothesizedGBR parameters (ν′1, λ1) are correct, (ν′2, λ2) should beconsistent with (ν′1, λ1). We define the cost of the hypoth-esis (ν′1, λ1) as the minimum of |ν′1 − ν′2|2 + |λ1 − λ2|2for all m1 on the equal-intensity contour. The final resultof (ν′, λ) are obtained by searching a position with theminimum cost.

An example of this cost over a large interval of the(ν, λ) parameter-space is shown in the left of Figure 10.We only search for positive λ to ensure the surface facingthe camera. It should be noted that this method doesnot guarantee to find a single minimum. As proved inProposition 5, there could be 4 discrete solutions. Thismotivates an alternative procedure that uses constraintsfrom two images and eliminates the need for a 2D search,and relies on a simpler 1D search instead.

Constraints from two images. The single-image methoddescribed above begins by resolving one translationaldegree of freedom (µ′ in Eq. 14) using one pair (n, n′) thatcan be easily identified as the intersection of the isophoteand isotropic curve passing through the point n. Given twoimages under two sources in general position, the sameprocedure can be applied in each image to resolve bothtranslational degrees of freedom.

Given an image under source si, a pair (ni, n′i) inducesa linear constraint on the translational parameters (µ, ν)

(i.e. Equation (9)), we can write this constraint as

s2µ− s1ν − C1 = 0. (15)

Geometrically, this is interpreted as a requirement forsymmetry of isotropic pairs about the principal meridian,which is characterized by (v, s, n + n′) = 0.

In this way, isotropic constraints from at least twosources in general position can be used to recoverboth translational parameters, µ, ν, without an exhaustivesearch. This reduces the GBR ambiguity to a classic bas-relief ambiguity diag(1, 1, λ), and it remains to determinethe scale parameter λ. This is consistent with the discus-sion following Proposition 1 which claims the classic bas-relief ambiguity cannot be resolved by isotropy.

The scale ambiguity can be resolved using reciprocalconstraints from a single image. The procedure is muchlike that of the single-image case described above, with theimportant difference being that it requires only a 1D searchover λ instead of a 2D search over (ν′, λ). Specifically,a hypothesis λ1 yields hypotheses for the true source,the quadrilateral center and peak: s1, c1, and p1. It alsogenerates a hypothesis of the intensity at m. Different fromthe single-image case, there is no need to check every pointon the equal-intensity contour. Since the classic bas-reliefambiguity does not break a reciprocal curve (accordingto the Proposition 4), the reciprocal match m must lieon the reciprocal curve passing through n. Hence, anhypothesis of m can be obtained as the intersection ofof this reciprocal curve and an equal-intensity contour inthe projective plane. The transformed quadrilateral centerc2 and peak p2 can be re-estimated by intersecting theperpendicular bisector of nn′ with the line mn′ and mnrespectively as the single image case. These in turn providean estimation of λ2 by the ratio of the distance betweenc1, p1 and that between c2, p2. If the hypothesized valueλ1 is correct, λ2 should be the same as λ1. So we definea cost of the hypothesis λ1 as |λ1 − λ2|2. The final resultof λ1 is obtained by searching a position with minimumcost.

An example of this measure of disagreement over alarge interval of λ values is shown in the right of Figure 10.Unlike the 2D search, we typically see a clear globalminimum in the objective function at the true value ofλ.

Special case of ‘specular-spike’ reflectance. The needof exhaustive search in the two described methods comesfrom the difficulty of identifying isotropic/reciprocal pairs.In the special case of ‘specular-spike’ reflectance, a specu-lar pixel forms isotropic and reciprocal pairs with itself, i.e.n = n′ = m. Substituting this condition into Equations 9–11, we find the Equation 11 is degenerated. The first twoequations are still valid. Hence, from a specular pixelin each of two images, we can obtain four equationsto solve the three GBR parameters. We reach the same


conclusion as [24] that two specular pixels resolve theGBR ambiguity of ‘specular-spike’ surfaces. Please notethat in our formulation µ, ν are solved first in linearequations. In comparison, [24] solves all three parameterstogether in a nonlinear formulation. The performance ofthese two methods are further compared in Sects. VII.

B. Calibrated photometric stereo

The auto-calibrated approach described in the previoussection can be applied whenever the BRDF (or a separablecomponent of it) is isotropic and spatially uniform on thesurface. More general reflectance can be accommodatedwhen more information is available about the light sources.A very general method is that of Alldrin and Kriegman [6],which provides a partial reconstruction for surfaces withisotropic reflectance that varies arbitrarily between surfacepoints. Given a set of images I(x, y, t) captured using acone of known source directions s(t), t ∈ [0, 2π) centeredabout view direction v, this method yields one componentof the normal at every image point (x, y). Specifically, foreach pixel it provides the plane spanned by the unknownsurface normal and the view direction, but the remainingdegree of freedom in each normal cannot be recoveredwithout additional information. In other words, if thesurface is differentiable, the surface gradient direction canbe recovered at each point, but the gradient magnitude isunknown. This means that one can recover the ‘iso-depthcontours’ of the surface, but that these curves cannot beordered [6].

In this section, we use reflectance symmetries on theprojective plane to reconstruct additional 3D structures.Consider a surface S = {x, y, z(x, y)} that is de-scribed by a height field z(x, y) on the image plane.A surface point with gradient zx, zy is mapped via theGaussian sphere to point n ' (zx, zy,−1) in the pro-jective plane, and the ambiguity in gradient magnitudefrom [6] corresponds to a transformation of normal fieldn(x, y) ' diag(1, 1, λ(x, y))n(x, y), where the per-pixelscaling λ(x, y) is unknown. As depicted in Figure 11, thiscan be interpreted as a per-pixel bas-relief transformation,where each normal n is translated arbitrarily along theline vn. Now, an isotropic pair (n, n′) has two properties:1) n, n′ are equally distant from the origin; and 2) linesvn and vn′ are symmetric across the principal meridian.The per-pixel transformation destroys the first property butpreserves the second. In what follows, we seek to resolvethe shape ambiguity by restoring the former.

Having equal distance to v, isotropic pairs lie on circlescentered about the view direction. Any two normals onsuch a circle is necessarily an isotropic pair with respectto one of the sources in {s(t)}. Thus, for any normal n,the view-centered circle on which it lies can be interpretedas the union of its isotropic matches with respect to the setof sources {s(t)}. Now, a view-centered circle is also the

Fig. 11. In calibrated photometric stereo, one component of each surfacenormal is recovered by exploiting isotropy at each point [6]. Determiningthe remaining degree of freedom at each point can be interpreted asfinding an unknown translation of the normal along the line through von the projective plane.

Gaussian-image of surface points having equal gradientmagnitude: ||∇z|| = constant. Thus, if we could locatesurface points with normals on such circle, we wouldrecover a surface curve of constant gradient magnitude—acurve we will refer to as an ‘iso-slope contour’. To get asense of how this would constrain the surface, considerthat when only the iso-depth contours are known, thesurface can be recovered—at best—up to a differentiablefunction [6]. This is because any two differentiable heightfields z1(x, y) and z2 = h(z1) will have the same setof iso-depth contours for any differentiable function h(·).Here we show that if they also possess the same iso-slope contours, then this arbitrary differentiable function isreduced to a classic bas-relief transformation (i.e., a linearscaling of depth).

Proposition 6. In the general case, if differentiable heightfields z1(x, y) and z2 = h(z1) are related by a differen-tiable function h and possess equivalent sets of iso-slopecontours, the function h is linear.

Proof: From the functional relationship betweenz1, z2:

||∇z2||2 = (∂h/∂z1)2||∇z1||2.If z1 and z2 possess the same set of iso-slope contours,either (∂h/∂z1)

2 is constant or ||∇z1||2 = ||∇z2||2 = 0along each contour. Since sets of iso-depth and iso-slopecontours are generically distinct, this implies that ∂h/∂z1

is constant and h is linear.It can be shown that the ‘accidental’ case in which the setsof iso-depth and iso-slope contours are equivalent corre-sponds to a surface-of-revolution with the view directionv as the symmetry axis.

In light of Proposition 6, it is desirable to be able toidentify the iso-slope contour passing through a givenimage point. With the partial reconstruction, a normal nis recovered up to a per-pixel bas-relief transformation.This transformation does not change the line vn, hence,the isotropic match n′ must lie on the line symmetric


to vn across the principal meridian. If the surface hasuniform reflectance (or has a uniform separable compo-nent), the match n′ can be located by intersecting this linewith the euqal-intensity contour passing through n. Suchisotropic matches n′(t) with respect to all light directionss(t), t ∈ [0, 2π) define the iso-slope contour.

This simple method for identifying iso-slope contourscould be ambiguous because the equal-intensity contourmight intersect the line vn′ at two points. This problemis solved by the following extension which also enablesus to deal with surfaces having spatially-varying isotropicreflectance of the form:

f(x, y, ωi,ωo)=f1(x, y) + f2(x, y)f3(θi, θo,|φi−φo|),(16)

which is a generalization of Eq. 12 that allows a ‘textured’specular component.

Let I(x, y, t) be the recorded radiance, and at eachimage point (x, y), shift and normalize these observationsas

In(x, y, t) , (I(x, y, t− φn)−mint I(x, y, t))(maxt I(x, y, t)−mint I(x, y, t))

, (17)

where φn ∈ [0, 2π) is the azimuthal component of thesurface normal as recovered by [6], and t is extendedperiodically: t → t + 2kπ for integer k. Then, if thespatially-varying BRDF is of the form in Eq. 16, anecessary condition for two points (x1, y1) and (x2, y2)to have normal directions forming an isotropic pair isIn(x1, y1, t) = In(x2, y2, t) ∀ t ∈ [0, 2π). This is becausenormalizing the temporal radiance at each pixel to [0, 1]removes the effects of the spatially-varying reflectanceterms f1 and f2. This constraint can be used in thematching procedure above by using it in place of the equal-intensity contours.

VII. EXPERIMENTS

In order to test our methods, we conducted experimentswith both real and synthetic data. Each set of images wascaptured/rendered using directional illumination and anorthographic camera. In the case of real data, the camerawas radiometrically calibrated so that image intensitycould be directly related to emitted scene radiance.

A. Uncalibrated photometric stereo

In order to test the methods for resolving the GBRambiguity, we first captured high dynamic range imagesand separated diffuse and specular reflection componentsof the input images using a color-based technique [34],and then applied the method of Yuille and Snow [22] tothe diffuse images to obtain a reconstruction up to a GBRambiguity. We then corrected this ambiguity using both thesingle-image and two-image methods described above.

Figure 12 shows results on a synthetic example usingthirty-six rendered input images of a Venus model using

Fig. 12. Resolving the GBR using constraints from reflectance symme-try. The top row shows linearly encoded normal map. The second and thethird row are the depth maps and surfaces respectively. Columns left toright: calibrated photometric stereo; uncalibrated photometric stereo [22];our single-image auto-calibration method; our two-image auto-calibrationmethod.

the Cook-Torrance BRDF model. The top row of thisfigure shows a linearly coded normal map, where the RGBchannels represent x, y, and z components, respectively.The second row shows the corresponding surface heightfields obtained by integrating the normal fields. The thirdrow shows additional validation by rendering the resultsurfaces. From left to right, the columns show results ofcalibrated photometric stereo (i.e., ‘ground truth’); uncali-brated photometric stereo [22]; and the results obtainedusing the single-image method and two-image methoddescribed in Sec. VI-A. For the latter two methods, oneor two input images were chosen at random, and onenormal n was chosen in each image to provide a ‘seed’ forresolving the three GBR parameters. These results demon-strate that the two proposed auto-calibration methods cansuccessfully resolve the GBR ambiguity in uncalibratedphotometric stereo [22], and gives results that are veryclose to the calibrated case.

Figure 13 shows results for experiments on real images.In this figure, the top row shows one of the input imagesalong with the separated diffuse and specular components.The second shows the normal maps as before, and thethird and fourth rows show the integrated depth maps andsurfaces. The columns are arranged in the same way asin Figure 12. Again, our auto-calibration methods resolvethe GBR ambiguity. However, the two-image method oftengenerates results closer to the ‘ground truth’ than the one-image method. This can be consistently observed from thedepth of the ‘fish’, ‘apple’ and ‘pear’ examples.

Figure 14 shows the angular errors in recovered normalmaps with respect to the result from calibrated photometricstereo. The first row shows the error of uncalibrated


Fig. 13. Resolving the GBR using constraints from reflectance symmetry. Top row: one input image with separated diffuse and specular components.Middle row: linearly encoded normal map. Bottom rows: depth maps and surfaces. Columns left to right: calibrated photometric stereo; uncalibratedphotometric stereo [22]; our single-image and two-image auto-calibration method, which successfully resolves the GBR ambiguity.

photometric stereo. The second and third rows show theerror of our single-image and two-image methods. It isevident that both methods can significantly reduce theerror and the two-image method is consistently moreaccurate than the single-image method. We also observedseveral limitations of our method. First, the separation ofdiffuse and specular reflectance could be inaccurate. Whenthe specular reflection is weak, the specular component haslow signal-to-noise ratio, which causes problems for theidentification of isotropic and reciprocal structures. Thisexplains the large error in the ‘fish’ example. Second, weassume the diffuse reflectance to be Lambertian to obtain areconstruction up to the GBR ambiguity. This assumptionmight not be true for certain objects. Last, we require asmoothly curved surface so that the geometric structuresare presented. Our method cannot handle surfaces withonly a few discrete normals.

We further compared our method with [24] for objects

with ‘specular spike’ reflectance. As shown in Figure 15,our method generates results closer to the calibratedmethod. We believe this is because our linear formulation(for µ, ν) is more robust than the non-linear equations in[24].

B. Calibrated photometric stereo

In this section, we provide suggestive results for the cal-ibrated reconstruction procedure described in Sec. VI-B.Given a set of images captured under a known cone ofsource directions, we obtain iso-depth contours (i.e., onecomponent of the normal at each point) by applying themethod of Alldrin and Kriegman [6]. Then, we applythe method described in Sec. VI-B to compute iso-slopecurves.

Figure 16 shows two sets of results, each using fortysynthetic images rendered using the Cook-Torrance BRDF


Fig. 14. Surface normal angular difference (in degrees) between. From top to bottom, they are the errors in uncalibrated method, our single-imageand two-image method.

Fig. 15. Comparison with the method proposed in [35]. Top rows:surface normals and depth. Bottom row: one of the input image andangular errors. Columns left to right: calibrated photometric stereo,uncalibrated photometric stereo [25], the method presented in [35], ourtwo-image method.

model and forty source directions that are uniformlydistributed along a view-centered cone. On the right inred are shown iso-slope contours that are recovered usingEq. 17 to identify isotropic matches n′(t) for the seedpoints marked in yellow. For comparison, the (generallydistinct) iso-depth contours recovered by [6] are shown inblue. The ground-truth curves (computed using the knowngeometry) are shown on the left, and our results matchthese quite closely.

Figure 17 shows analogous results on real images. Theiso-slope curves recovered by our procedure are shownon the right, and as a form of ‘ground truth’, we showthe corresponding contours taken from the complete re-constructions obtained by Alldrin et al. [35]. Despite thedifferences between these two reconstruction proceduresand their underlying BRDF models, the recovered curvesare quite consistent. The gourd example uses 100 input

Fig. 16. Recovered iso-slope and iso-depth curves. Red and bluecurves are the iso-slope and iso-depth curves respectively. These twoset of curves often do not coincide. Our result is shown on the right.For comparison, on the left is ground truth curves obtained from theunderlying normal fields.

images, and the helmet example uses 252 input images,and in each case, the source directions were measuredduring acquisition.

VIII. DISCUSSION

Isotropy and reciprocity, which are common symmetriesin the BRDF, induce joint constraints on shape, view-point and lighting that can be used for the radiometricanalysis of images. Any image of a surface (convex ornot) under directional illumination and orthographic viewcontains observations of distinct surface points havinglocal lighting and viewing directions that are equivalentunder these symmetries, and these equivalences inducegeometric structure on the Gauss sphere. We argue that theprojective plane provides a useful abstraction for describ-ing this geometric structure and for exploiting it for three-dimensional reconstruction. By developing reconstruction


Fig. 17. Recovered iso-slope and iso-depth curves. Red and blue curvesare the iso-slope and iso-depth curves respectively. Our result is shownon the right. For comparison, on the left is result computed from thereconstructions of [35].

techniques that exploit common BRDF symmetries—andonly these symmetries—we hope to enable systems thatare more likely to succeed in the presence of real-world,non-Lambertian materials.

We consider applications to both calibrated and un-calibrated photometric stereo, and in the uncalibratedcase, we show that reciprocity and isotropy induce con-straints in a single image that are sufficient to resolve thegeneralized bas-relief (GBR) ambiguity. Practically, thisleads to an auto-calibrating reconstruction procedure thatrequires only a simple acquisition system (a hand-heldlight sources) and is likely to succeed for a broad classof objects. Theoretically, it generalizes previous work thatuses the ‘specular spike’ model of reflectance to resolvethe GBR [23], [24]—that model is a special case of thearbitrary isotropic BRDFs that are considered here. Ouranalysis also provides an alternative to existing methodsthat resolve the GBR using parametric BRDF models [25],and it has the relative advantages of working for a broaderclass of surfaces and allowing precise statements regardingthe conditions for uniqueness of reconstructions.

The fact that isotropic and reciprocal constraints from asingle image are sufficient to resolve the GBR ambiguitymay help motivate studies of shape perception in the pres-ence of non-Lambertian reflectance. Existing perceptualstudies suggest that given a single image of a Lambertiansurface (left in Fig. 1), different people perceive differentsurfaces, and these surfaces are related to one anotherby GBR-like transforms [36]. The results in this paperprovide a motive for conducting a similar experiment usingnon-Lambertian objects with relatively uniform specularcomponents (right in Fig. 1). Perhaps peoples’ perceptionsof these surfaces will exhibit less variance.

While we restricted our attention to applications inphotometric stereo in this paper, it is likely that some of

the tools we develop can be applied to other tasks. Onedirection to explore might be photogeometric reconstruc-tion techniques that combine geometric and photometricconstraints from images captured under varying viewpointand lighting. Some of these methods also yield ambigu-ities that can be described as affine transformations ofthe normal field [32]. Other directions to consider arereflectometry, illuminant estimation, and compression ofappearance data.

Finally, while we restrict our attention to reciprocityand isotropy in this paper, the same framework (and theprojective plane) can be used to exploit other symmetriesas well. In a number of cases, the three-dimensional BRDFdomain can be further reduced to a two-dimensionaldomain because of half-vector symmery [17], [37]. Thissymmetry provides an even stronger constraint on view,shape, and lighting, and it might be interesting to studyin the context of calibrated photometric stereo, where itcould provide conditions for uniqueness that supplementrecent empirical results [35]. Conditions for uniquenessbased on parametric BRDF models exist [28], but for themost part, conditions that avoid the restrictions of low-parameter BRDF models have yet to be discovered.

ACKNOWLEDGMENT

We thank the anonymous reviewers for their valuablefeedbacks and Neil Alldrin for sharing his code and data.This work was supported by the Singapore MOE grantR-263-000-555-112 and HOME 2015 project R-263-000-592-305. Long Quan was supported by the Hong KongRGC GRF 619409 and 618510, and the National NaturalScience Foundation of China (60933006). Todd Zicklerwas supported by the US National Science Foundationunder Career Award IIS-0546408, the US Office of NavalResearch through award N000140911022, the US ArmyResearch Laboratory and the US Army Research Officeunder contract/grant 54262-CI, and a fellowship from theAlfred P. Sloan Foundation.

REFERENCES

[1] B. T. Phong, “Illumination for computer generated pictures,” Com-munications of the ACM, vol. 18, no. 6, pp. 311–317, June 1975.

[2] S. Shafer, “Using color to separate reflection components,” COLORresearch and applications, vol. 10, no. 4, pp. 210–218, 1985.

[3] H. v. Helmholtz, Treatise on Physiological Optics, volume 1 ed.Dover (New York), 1925.

[4] J. Lu and J. Little, “Reflectance and shape from images using acollinear light source,” IJCV, vol. 32, no. 3, pp. 1–28, 1999.

[5] T. Zickler, P. Belhumeur, and D. Kriegman, “Helmholtz stereopsis:Exploiting reciprocity for surface reconstruction,” IJCV, vol. 49,no. 2/3, pp. 215–227, September 2002.

[6] N. Alldrin and D. Kriegman, “Toward reconstructing surfaceswith arbitrary isotropic reflectance: A stratified photometric stereoapproach,” in Proc. CVPR, 2007.

[7] P. N. Belhumeur, D. J. Kriegman, and A. L. Yuille, “The bas-reliefambiguity,” International Journal of Computer Vision, vol. 35,no. 1, pp. 33–44, 1999.


[8] F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W. Ginsberg,and T. Limperis, “Geometric considerations and nomenclature forreflectance,” National Bureau of Standards (US), Monograph 161,Oct. 1977.

[9] S. R. Marschner, “Inverse rendering for computer graphics,” Ph.D.dissertation, Cornell University, 1998.

[10] J. Lu and J. Little, “Reflectance and Shape from Images Using aCollinear Light Source,” IJCV, vol. 32, no. 3, pp. 213–240, 1999.

[11] P. Tan, S. P. Mallick, L. Quan, D. Kriegman, and T. Zickler,“Isotropy, reciprocity and the generalized bas-relief ambiguity,” inProc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.

[12] T. Zickler, P. Belhumeur, and D. Kriegman, “Helmholtz stereopsis:Exploiting reciprocity for surface reconstruction,” in Proc. ECCV,2002.

[13] E. P. F. Lafortune, S.-C. Foo, K. E. Torrance, and D. P. Greenberg,“Non-linear approximation of reflectance functions,” in SIGGRAPH’97: Proceedings of the 24th annual conference on Computergraphics and interactive techniques. New York, NY, USA: ACMPress/Addison-Wesley Publishing Co., 1997, pp. 117–126.

[14] G. Ward, “Measuring and modeling anisotropic reflection,” Com-puter Graphics, vol. 26, no. Annual Conference Series, pp. 265–273, 1992.

[15] R. L. Cook and K. E. Torrance, “A reflection model for computergraphics,” ACM Transactions On Graphics, vol. 1, no. 1, pp. 7–24,Jan. 1982.

[16] A. Ngan, F. Durand, and W. Matusik, “Experimental analysis ofbrdf models,” Eurographics Symposium on Rendering, pp. 117–126, 2005.

[17] M. Stark, J. Arvo, and B. Smits, “Barycentric parameterizations forisotropic BRDFs,” IEEE T-VCG, vol. 11, no. 2, pp. 126–138, 2005.

[18] S. Rusinkiewicz, “A new change of variables for efficient BRDFrepresentation,” Eurographics Rendering Workshop, vol. 98, pp.11–22, 1998.

[19] P. Tan and T. Zickler, “A projective framework for radiometricimage analysis,” in Proc. CVPR, 2009.

[20] R. Woodham, “Photometric stereo: A reflectance map technique fordetermining surface orientation from image intensities,” in Proc.SPIE 22nd Annual Technical Symposium, 1978, pp. 136 – 143.

[21] K. Hayakawa, “Photometric stereo under a light source witharbitrary motion,” J. Opt Soc. Am., vol. 11, no. 11, 1994.

[22] A. Yuille and D. Snow, “Shape and Albedo from Multiple ImagesUsing Integrability,” in Proc. IEEE Conf. Computer Vision andPattern Recognition, 1997.

[23] O. Drbohlav and R. Sara, “Specularities reduce ambiguity ofuncalibrated photometric stereo,” in ECCV, vol. 2, 2002, pp. 46–60.

[24] O. Drbohlav and M. Chantler, “Can two specular pixels calibratephotometric stereo?” in ICCV, vol. 2, 2005, pp. 850–1857.

[25] A. Georghiades, “Incorporating the Torrance and Sparrowmodel of reflectance in uncalibrated photometric stereo,”Proc. IEEE Int. Conf. Computer Vision, pp. 816–823, 2003.

[26] K. Ikeuchi, “Determining surface orientations of specular surfacesby using the photometric stereo method,” IEEE TRANS. PATTERNANALY. AND MACH. INTELLIG., vol. 3, no. 6, pp. 661–669, 1981.

[27] S. Nayar, K. Ikeuchi, and T. Kanade, “Determining shape andreflectance of hybrid surfaces by photometricsampling,” IEEETransactions on Robotics and Automation, vol. 6, no. 4, pp. 418–431, 1990.

[28] H. D. Tagare and R. J. P. deFigueiredo, “A theory of photometricstereo for a class of diffuse non-lambertian surfaces,” IEEE T-PAMI,vol. 13, no. 2, pp. 133–152, 1991.

[29] D. Goldman, B. Curless, A. Hertzmann, and S. Seitz, “Shapeand spatially-varying brdfs from photometric stereo,” in IEEEInternational Conference on Computer Vision, 2005.

[30] P. Tan, S. Lin, and L. Quan, “Resolution-enhanced photometricstereo,” in Proc. European Conf. Computer Vision, 2006.

[31] H. S. M. Coxeter, The Real Projective Plane, 3rd ed. Springer-Verlag, 1993.

[32] T. Zickler, P. Belhumeur, and D. Kriegman, “Toward a stratificationof Helmholtz stereopsis,” in Proc. IEEE Conf. Computer Vision andPattern Recognition, vol. 1, 2003, pp. 548–555.

[33] N. Joshi and D. Kriegman, “Shape from varying illumination andviewpoint,” in Proc. IEEE Int. Conf. Computer Vision, 2007.

[34] Y. Sato and K. Ikeuchi, “Temporal-color space analysis of reflec-tion,” Journal of Optical Society of America A, vol. 11, no. 11, pp.2990 – 3002, November 1994.

[35] N. Alldrin, T. Zickler, and D. Kriegman, “Photometric stereo withnon-parametric and spatially-varying reflectance,” in Proc. CVPR,2008.

[36] J. Koenderink, A. v. Doorn, A. Kappers, and J. Todd, “Ambiguityand the ‘mental eye’ in pictorial relief,” Perception, vol. 30, pp.431–448, 2001.

[37] F. Romeiro, Y. Vasilyev, and T. Zickler, “Passive reflectometry,” inProc. ECCV, 2008.

APPENDIX

Proposition 7. For v = (0, 0, 1)′ and any other threeunit vectors s, n, n′ where n 6= ±n′, s>n = s>n′ andv>n = v>n′, if and only if (v × s)>(n + n′) = 0 ands>nv>n = s>n′

v>n′ .

Proof: We first prove the forward direction. The results>nv>n = s>n′

v>n′ is trivial since it is simply the quotient of thetwo equations. Next, (v× s)>(n + n′) = 0 ⇔ −s2(n1 +n′1)+s1(n2+n′2) = 0 ⇔ s1

s2= n1+n′1

n2+n′2. On the other hand,

since n3 = v>n = v>n′ = n′3, we have s>n = s>n′ ⇒n1s1 + n2s2 = n′1s1 + n′2s2 ⇔ s1

s2= −n2−n′2

n1−n′1. Hence,

(v× s)>(n + n′) = 0 ⇔ n1+n′1n2+n′2

= −n2−n′2n1−n′1

⇔ n21 + n2

2 =n′21 + n′22 ⇔ n3 = n′3, which is true.

Now, we prove the reverse direction. s>nv>n = s>n′

v>n′ ⇔s1s2

= −n2n′3−n′2n3n1n′3−n′1n3

and s>nv>n = s>n′

v>n′ ⇔ s1s2

= n1+n′1n2+n′2

.

Hence, from these two equations we have −n2n′3−n′2n3n1n′3−n′1n3

=n1+n′1n2+n′2

⇔ (n1n′3−n′1n3)(n1 +n′1)+(n2n

′3−n′2n3)(n2 +

n′2) = 0 ⇔ (1−n23)n

′3−(1−n′23 )n3+(n1n

′1+n2n

′2)(n

′3−

n3) = 0 ⇔ (n′3−n3)(1 + n>n′) = 0. Since n 6= −n′, wehave v>n′ = n′3 = n3 = v>n. Then from s>n

v>n = s>n′v>n′

we immediately obtain s>n′ = s>n.

Proposition 8. Four unit vectors v, s, n, m where n, mnon-collinear satisfy s>n = v>m and v>n = s>m, ifand only if s>(n + m) = v>(n + m) and s>(n × m) =v>(n×m).

Proof: As n, m are distinct, they define a coordinatesystem in R3 along with vector n×m. The result followsdirectly from representing v and s in this coordinatesystem.

Proposition 9. Given that v = (0, 0, 1) is the origin inthe 2D projective plane, for any other three points a, b, son the plane, the line ab is perpendicular to the line vs ifand only if (s>a)/(v>a) = (s>b)/(v>b).

Proof: The line ab and vs are a × b and v × srespectively. Let the intersection of ab and vs with theline at infinity l∞ = (0, 0, 1) be A , ab × l∞ andB , vs × l∞. Then ab⊥vs if and only if A, B areharmonic conjugate points with respect to the circularpoints I = (1, i, 0)′ and J = (1,−i, 0)′, i.e., the crossratio cr(A, B, I, J) = −1. Now, cr(A, B, I, J) = −1 ⇐⇒


A1B1 +A2B2 = 0, and this reduces to (s1a1 +s2a2)b3 =(s1b1 + s2b2)a3 ⇐⇒ (s>a)(v>b) = (s>b)(v>a) ⇐⇒(s>a)/(v>a) = (s>b)/(v>b).

Proposition 10. For any four points a, b, s and v, the lineab intersects sv at the point (v + s)× (v× s) if and onlyif (s>(a× b))/(v>(a× b)) = (s>(v + s))/(v>(v + s)) 7

Proof: The line ab and vs are a × b and v × srespectively. It is clear that (v + s)× (v× s) is a point onthe line vs. This point is the intersection of ab and vs if itis also a point on ab, i.e., (a× b)> ((v + s)× s× v) = 0.On the other hand, (v + s) × s × v = s(v>(v + s)) −v(s>(v + s)). Therefore, (a × b)> ((v + s)× s× v) = 0⇐⇒ (a× b)>s(v>(v + s)) = (a× b)>v(s>(v + s)) ⇐⇒(s>(a× b))/(v>(a× b)) = (s>(v + s))/(v>(v + s)).

Ping Tan received the B.S. degree in AppliedMathematics from the Shanghai Jiao Tong Uni-versity, China, in 2000. He received the Ph.D.degree in Computer Science and Engineeringfrom the Hong Kong University of Science andTechnology in 2007. He joined the Departmentof Electrical and Computer Engineering at theNational University of Singapore as an assis-tant professor in October 2007. His researchinterests include image-based modeling, photo-metric 3D modeling and image editing. He has

served on the program committees of ICCV, CVPR, ECCV. He is amember of the IEEE and ACM

Todd Zickler received the B.Eng. degree inhonors electrical engineering from McGill Uni-versity, Montreal, QC, Canada, in 1996 andthe Ph.D. degree in electrical engineering fromYale University, New Haven, CT, in 2004, un-der the direction of P. Belhumeur. He joined theSchool of Engineering and Applied Sciences,Harvard University, Cambridge, MA, as an As-sistant Professor in 2004 and was appointedJohn L. Loeb Associate Professor of the NaturalSciences in 2008. He is the Director of the

Harvard Computer Vision Laboratory, and his research is focused onmodeling the interaction between light and materials and developingalgorithms to extract scene information from visual data. His workis motivated by applications in face, object, and scene recognition;image-based rendering; content-based image retrieval; image and videocompression; robotics; and human-computer interfaces. Dr. Zickler isa recipient of the National Science Foundation Career Award and aResearch Fellowship from the Alfred P. Sloan Foundation. His researchis funded by the National Science Foundation, the Army Research Office,the Office of Naval Research, and the Sloan Foundation. He is a memberof the IEEE.

7 According to Prop. 9, this is equivalent to the join of (v + s) and(a× b) being perpendicular to vs.

Long Quan is a Professor of the Departmentof Computer Science and Engineering and theDirector of Center for Visual Computing andImage Science at the Hong Kong Universityof Science and Technology. He received hisPh.D. in 1989 in Computer Science from INPL,France. He entered into the CNRS (Centre Na-tional de la Recherche Scientifique) in 1990 andwas appointed at the INRIA (Institut Nationalde Recherche en Informatique et Automatique)in Grenoble, France. He joined the HKUST

in 2001. He works on vision geometry, 3D reconstruction and image-based modeling. He supervised the first Best French Ph.D. Dissertationin Computer Science of the Year 1998 (le prix de these SPECIF), thePiero Zamperoni Best Student Paper Award of the ICPR 2000, and theBest Student Poster Paper of IEEE CVPR 2008. He co-authored oneof the six highlight papers of the SIGGRAPH 2007. He was electedas the HKUST Best Ten Lecturers in 2004 and 2009. He has servedas an Associate Editor of IEEE Transactions on Pattern Analysis andMachine Intelligence (PAMI) and a Regional Editor of Image andVision Computing Journal (IVC). He is on the editorial board of theInternational Journal of Computer Vision (IJCV), the Electronic Letterson Computer Vision and Image Analysis (ELCVIA), the Machine Visionand Applications (MVA), and the Foundations and Trends in ComputerGraphics and Vision. He was a Program Chair of IAPR InternationalConference on Pattern Recognition (ICPR) 2006 Computer Vision andImage Analysis, is a Program Chair of ICPR 2012 Computer and RobotVision, and is a General Chair of the IEEE International Conference onComputer Vision (ICCV) 2011. He is a Fellow of the IEEE ComputerSociety.

ieee transactions on pattern analysis and machine intelligence, vol. x, no. xx...

Documents