Enhanced Photometric Stereo with Multispectral Images

Tsuyoshi Takatani∗† Yasuyuki Matsushita‡ Stephen Lin‡

Yasuhiro Mukaigawa† Yasushi Yagi†

†Osaka University{takatani,mukaigaw,yagi}@am.sanken.osaka-u.ac.jp

‡Microsoft Research Asia{yasumat,stevelin}@microsoft.com

Abstract

We introduce a technique based on multispectralimages aimed at improving Lambertian photometricstereo. Many photometric stereo algorithms assumeLambertian reflectance, but deviations from this idealproduce errors in shape reconstruction. To alleviatethis problem, we exploit the wavelength-dependence ofmaterial reflectance. Based on the observation that re-flectance at certain wavelengths for a given object ismore Lambertian than at others, we propose a methodfor identifying such wavelengths by a matrix rank anal-ysis, and use them to achieve more accurate photo-metric stereo. We merge reconstruction results fromdifferent wavelengths to produce the final surface nor-mal map. Experimental results on synthetic and realdata demonstrate the greater accuracy of this methodcompared to conventional photometric stereo based onbrightness images.

1 Introduction

Photometric stereo, a technique originally proposedby Woodham [1] and Silver [2], is a traditional methodfor estimating surface orientations from multiple im-ages taken under different lighting conditions. Akey factor in this process is the surface reflectance,which describes how the shading at each surface pointchanges in relation to the lighting direction and thesurface normal. By varying the light direction, thecorresponding changes in the shading are used to inferthe surface normals according to the reflectance model.

Surface reflectance is normally assumed to followLambert’s law, whereby the intensity I of the reflectedlight is proportional to the inner product of the lightingvector l and the surface normal vector n:

I(x) = ρ(x)nTl, (1)

where ρ(x) denotes the albedo, or intrinsic color, ofthe surface at point x. This Lambertian model of re-flectance is frequently used in photometric stereo be-cause of its simplicity and convenience. However, thesmall deviations from the Lambertian model displayedby many materials produces inaccuracies in shape re-construction.

Various methods have been proposed for dealingwith non-Lambertian reflectance. Non-Lambertianhighlights are treated as outliers within a Lambertianphotometric stereo framework [3], or removed from im-ages [4]. Some instances of photometric stereo areformulated for specific parametric reflectance models,

∗This work has been done while the author was a researchintern at Microsoft Research Asia.

Combination of normal maps estimated at optimal wavelength in each segment according to evaluation values

Evaluation in each segment

....

wlen1 wlen2 wlen3 wlenm

Observed images at various wavelengths

segment1segment2segment3

segmentk

....

Segmentation using multispectral reflectance

wlen1 wlen2 wlen3 wlenm

....

Photometric stereo using observation at each wavelength

+

Figure 1. Overview of our algorithm for enhancingthe performance of photometric stereo using multi-spectral images.

such as the Torrance-Sparrow model [5], a combinationof isotropic Ward models [6], and m-lobed reflectancemaps [7]. Several work have been presented for anarbitrary form for the reflectance [8, 9]. Some meth-ods analyze color for photometric stereo, but mainly tominimize image acquisition with differently colored il-luminations [10], to improve robustness by incorporat-ing additional measurements [11], or to aid in removingnon-Lambertian reflectance components [4, 12].

Our algorithm is outlined in Fig. 1. Our input isa set of images captured at various wavelengths andfor different lighting directions. For each wavelength,we apply Lambertian photometric stereo. In addition,the multispectral image constructed from the set ofwavelengths is segmented into different multispectralregions. In each of these, we evaluate the photometricstereo results for each wavelength, using matrix rankanalysis to identify which wavelength is most consis-tent with the Lambertian model. The surface normalmap computed for each wavelength are then mergedby selecting the optimal wavelength for each region.We thus obtain the final normal map. This approachis validated with simulated images. Experiments onreal images support this multispectral approach, show-ing visible improvements in surface normal estimationover conventional photometric stereo based on bright-ness images.

2 Photometric Stereo using MultispectralImages

2.1 Wavelength-dependent Reflectance

We first examine the wavelength-dependence of re-flectance and its effects on photometric stereo recon-

struction. When accounting for a continuous rangeof wavelengths, the Lambertian reflectance model inEq. (1) can be expressed as

I(λ, x) = σ(λ)ρ(λ, x)e(λ)n(x)Tl, (2)

where λ ranges over the measurable wavelengths of thesensor, σ is the spectral sensitivity of the camera, ρ isthe spectral reflectance at a given surface point, and eis the spectrum of the light source.

The diffuse reflection that is conventionally usedfor photometric stereo arises from a physical processin which light penetrates an object surface, scattersoff material particles, and is emitted out of the sur-face. According to the Lambertian model, this outgo-ing light is distributed uniformly in all directions, withan intensity given by Eq. (2).

In reality, the directional distribution of the outgo-ing light is determined by how the light is scatteredwithin the material. This subsurface scattering is phys-ically determined with respect to the optical propertiesof the material, virtually all of which are wavelength-dependent. For example, the wavelength can affectthe number of scattering events. For details on thewavelength-dependence of optical properties in mate-rials, we refer the reader to [13].

This wavelength-dependence of optical propertiessuggests that surface reflectance is also spectrum-sensitive, thereby affecting the reconstruction qualityof photometric stereo. We demonstrate this in Fig. 2with a simulated sphere rendered using real reflectancedata for red plastic, obtained from the MERL BRDFdataset [14]. Forty images with different lighting direc-tions were used for photometric stereo, and the colorspectrum was coarsely sampled in red, green, and bluechannels. The angular error maps in the reconstruc-tions reveal different degrees of accuracy in the differ-ent color channels.

2.2 Region-based Identification of OptimalWavelengths

Given a set of multispectral images captured witha stationary camera and different lighting directions,our algorithm first computes Lambertian photometricstereo, performs a region segmentation, and identifiesthe optimal wavelength for each region, which is thenused to merge the normal maps computed with dif-ferent wavelengths. The details of the process are asfollows.

Region segmentation As mentioned previously,the optimal wavelength for Lambertian photometricstereo may vary across the image because of varyingBRDFs. In particular, our investigations have shownthat regions of different color often have different op-timal wavelengths. To improve the reconstruction ac-curacy, we therefore segment the scene by normalizedmultispectral information, and determine the optimalwavelength for each of the regions. We apply k-meansclustering for the segmentation, using an image wherethe object is fully illuminated (i.e., no shadows).

Optimal wavelength identification in each re-gion To find the optimal wavelength in each seg-mented region, we evaluate the data in each region.

(a)

(b) (c) (d) (e) (f)

Avg. 1.42Var. 1.02

Avg. 5.37Var. 11.42

Avg. 13.43Var. 44.13

Avg. 2.56Var. 2.63

Figure 2. Effect of different wavelengths on photo-metric stereo. (a) One of 40 images rendered withdifferent lighting directions; (b) ground truth of thenormal map; normal maps and angular error mapsusing the (c) red, (d) green, and (e) blue color chan-nels for Lambertian photometric stereo; and (f) re-sults with brightness images.

For a region with N pixels and M lighting conditions,we can express Eq. (2) in matrix form as

I = g(λ)NL, (3)

where g(λ) = σ(λ)ρ(λ)e(λ), I ∈ RN×M is a matrixof image intensities, N ∈ RN×3 is a matrix of surfacenormals, and L ∈ R3×M is a matrix of lighting vectors.

As seen in Eq. (3), the Lambertian model predictsthat the rank of the observation matrix I must equalthat of the normal vector matrix N, given that thesampled lighting directions are not co-planar (i.e.,rank(L) = 3). As rank(I) deviates further fromrank(N), the data becomes more poorly fitted by theLambertian model and then Lambertian photometricstereo should produce less accurate results as discussedin [15].

In view of this effect, we evaluate the wavelengthsby determining how close the rank of its observationmatrix is to rank(N). The evaluation function used inthis work is

E =σk+1

σk, (4)

where k = rank(N) and σi denotes the i-th eigenvalueof I. This function becomes large when the rank of Ifalls below k (i.e., σk close to 0) or exceeds k (i.e., σk+1

becomes large). Thus, the wavelength that minimizesE is that for which the rank of I is closest to rank(N).A small additive term may be included in the denomi-nator to avoid division by zero, but in practice we havefound σk never to be zero.

Assuming the rank of N to be three, Eq. (4) is eval-uated with k = 3. Corresponding values of k shouldbe used in degenerate cases of lower rank. For eachsegmented region, the optimal wavelength minimizesE, and to enhance Lambertian photometric stereo, weestimate the final normal map by combining the nor-mal maps of each region that correspond to its optimalwavelength.

3 Experimental Results

3.1 Simulation images

We first validate our method on a simulated scenefor which the ground truth surface normals are known.Our scene consists of an egg-shaped surface renderedwith the reflectances of seven different materials fromthe MERL BRDF dataset [14]. Forty images under

(a) (b)

(c) Avg. 4.16; Var. 0.49

(d) Avg. 5.88; Var. 0.73

(e) Avg. 10.40; Var. 1.51

(f) Avg. 4.29; Var. 0.37

(g) Avg. 3.14; Var. 0.16

Figure 3. Normal map estimation for simulation im-ages. (a) One of 40 images rendered under differentlight directions; (b) ground truth of the normal map;normal maps and angular error maps using the (c)red, (d) green, and (e) blue color channels; (f) withbrightness images; (g) with our method.

Table 1. Evaluation function values.

Material name Red Green Blue

blue-acrylic 0.960 0.985 0.642green-plastic 0.840 0.831 0.885light-brown-fabric 0.687 0.852 0.968orange-paint 0.767 0.812 0.897purple-paint 0.348 0.837 0.847red-phenolic 0.403 0.932 0.900yellow-matte-plastic 0.434 0.604 0.898

different lighting directions are generated. One of theseis shown in Fig. 3(a). The ground truth is exhibitedin Fig. 3(b). We note that the MERL BRDF datasetprovides reflectance data in terms of only red, green,and blue, so our multispectral technique is performedwith three color channels in this instance.

Photometric stereo is first computed for each of thecolor channels separately, and also for the brightnessimages obtained by summing the three color chan-nels. The estimated normal maps, angular error maps,and error statistics are shown in Fig. 3(c-f). Over-all, the brightness images yield better results than theimages of individual color channels, probably becauseof a higher signal-to-noise ratio (SNR) resulting fromsumming multiple measurements. However, in someregions, using an individual color channel yields a bet-ter estimate than the brightness images (see the redrectangles in Fig. 3 for example).

The result of combining normal map regions fromdifferent color channels is shown in Fig. 3(g). Each re-gion from the three color channel images is evaluatedusing Eq. (4), giving the values shown in Table 1. Pix-els with sharp highlights detected by intensity thresh-olding were discarded from the evaluation by removingtheir rows from the observation matrix. With this mul-tispectral technique, we achieve reconstructions thatsurpass those derived from brightness images, despiteour technique not using summed measurements to im-prove the SNR.

3.2 Real images

We conducted an experiment with a real scenecaptured using a fixed camera (Point Grey ResearchGrasshopper), a halogen lamp (OSRAM), and nine

narrow band filters at 450, 488, 580, 650, 694, 730,780, 880 and 940 nm (Edmond Optics). A translationstage (Sigma) was used to switch the narrow band fil-ters automatically. Images were recorded under twelvedifferent lighting directions. We applied photometricstereo to the images under perspective projection [16]to estimate the surface normals.

The target object in this experiment is a plushieshown in Fig. 4, made of fabric of different colors.

Normal estimation using each wavelength Thenormal maps of the plushie estimated from differentwavelengths are shown in Fig. 4. Subtle but visibledifferences are apparent in the results obtained fromthe different wavelengths. Long wavelengths tend toyield smoother reconstruction results, possibly becausethey penetrate deeper into the surface. By contrast,shorter wavelengths often yield sharper surface details.At certain wavelengths, such as 650 nm, surface normaldiscontinuities may appear, coinciding with sharp colorgradients on the surface. These effects may indicateproblems in fitting the Lambertian model to the dataat these wavelengths.

From the surface normal reconstructions for the dif-ferent wavelengths, we next determine which wave-lengths are most consistent within the Lambertianmodel with our segmentation and evaluation scheme.

Segmentation and region evaluation Segmenta-tion is performed using the nine color channels of ourmultispectral images. The segmentation result of thetarget object is shown in Fig. 5(a). The plushie is di-vided into nine different region types. Disconnectedareas on the image with the same region index aretreated collectively as one region.

After segmentation, Eq. (4) is used to determine theoptimal wavelength for each region. The values of theevaluation function for the different wavelengths areshown in Fig. 5(c) for each region. The evaluationcurves tend to be even over the color spectrum, pos-sibly because the fabric material is less transparent,and hence more similar degrees of light penetrationinto the surface at different regions and for differentwavelengths.

Combination of normal maps using evaluationfunction values The evaluation function minima inFig. 5(c) yield the optimal wavelength for each region,displayed in the label map in Fig. 5(b). Long wave-lengths in the visible range are often selected.

A reasonable supposition is that the optimal wave-length corresponds to the surface color of a region,since these wavelengths produce the brightest imageand a high SNR. However, our results suggest other-wise. For example, the optimal wavelength is in thered spectral range even for a green surface, because ofthe subsurface scattering properties of this region.

The normal map computed by our method from thelabel map and the normal maps at different wave-lengths is shown in Fig. 4. Though the ground truthis unavailable for this object, our normal map appearsqualitatively more correct than the normal map com-puted from brightness images. With our method, thenormal map is less sensitive to texture.

Plushie 450nm 488nm 580nm 650nm 694nm

730nm 780nm 880nm 940nm Brightness OursFigure 4. Picture and normal maps estimated using different wavelengths, brightness images, by applying our method.

(a) (b)

012345678

Reg

ion

inde

x 450488580650694730780880940

(c)

Figure 5. Our algorithm illustrated. (a) Nine re-gions segmented by k-means clustering in multispec-tral space. (b) Optimal wavelength for each regionidentified by Eq. (4). (c) Evaluation function valuesversus wavelength for each region.

4 Conclusion

In this paper, we proposed a new technique basedon multispectral images to improve Lambertian pho-tometric stereo. The resulting normal map is obtainedby combining different normal maps computed at anoptimal wavelength determined in each region of theimage. Optimal wavelength determination does notrequire ground truth geometry, but is based insteadon matrix rank analysis. Our experiments demon-strated significant wavelength-dependence of variousmaterials in photometric stereo reconstruction. Ourmethod demonstrably shows improvements over tradi-tional photometric stereo based on brightness images.

Some materials have reflectance properties that dif-fer significantly from the Lambertian model for anywavelength. To broaden the applicability of our tech-nique, future work will extend our approach to otherparametric reflectance models used in photometricstereo. In addition, we intend to investigate thewavelength-dependent reflectance of various objects forthe purpose of material recognition.

AcknowledgmentsThis research is granted by the Japan Society forthe Promotion of Science (JSPS) through the “Fund-ing Program for Next Generation World-Leading Re-searchers (NEXT Program),” initiated by the Councilfor Science and Technology Policy (CSTP).

References

[1] R. J. Woodham, “Photometric method for determin-ing surface orientation from multiple images,” Optical

Engineering, vol. 19, no. 1, pp. 139–144, 1980.[2] W. Silver, “Determining shape and reflectance using

multiple images,” tech. rep., Master thesis, MIT, 1980.[3] S. Barsky and M. Petrou, “The 4-source photometric

stereo technique for three-dimensional surfaces in thepresence of highlights and shadows,” IEEE Trans. onPattern Anal. Mach. Intell., vol. 25, no. 10, pp. 1239–1252, 2003.

[4] S. P. Mallick, T. E. Zickler, D. J. Kriegman, and P. N.Belhumeur, “Beyond lambert: Reconstructing specu-lar surfaces using color,” in Proc. CVPR, vol. 2, pp. 619–626, IEEE, 2005.

[5] A. Georghiades, “Incorporating the torrance and spar-row model of reflectance in uncalibrated photometricstereo,” in Proc. ICCV, pp. 816–823, IEEE, 2003.

[6] D. B. Goldman, B. Curless, and A. H. S. M. Seitz,“Shape and spatially-varying brdfs from photometricstereo,” in Proc. ICCV, vol. 1, pp. 341–348, IEEE,2005.

[7] H. Tagare and R. de Figueiredo, “A theory of photo-metric stereo for a class of diffuse non-lambertian sur-faces,” IEEE Trans. on Pattern Anal. Mach. Intell.,vol. 13, no. 2, pp. 133–152, 1991.

[8] N. Alldrin and D. Kriegman, “Photometric stereo withnon-parametric and spatially-varying reflectance,” inProc. CVPR, IEEE, 2008.

[9] T. Higo, Y. Matsushita, and K. Ikeuchi, “Consensusphotometric stereo,” in Proc. CVPR, pp. 1157–1164,IEEE, 2010.

[10] R. Anderson, B. Stenger, and R. Cipolla, “Color pho-tometric stereo for multicolored surfaces,” in Proc.ICCV, IEEE, 2011.

[11] P. H. Christensen and L. G. Shapiro, “Three-dimensionalshape from color photometric stereo,” InternationalJournal of Computer Vision, vol. 13, no. 2, pp. 213–227, 1994.

[12] M. Liao, X. Huang, and R. Yang, “Interreflection re-moval for photometric stereo by using spectrum-dependentalbedo,” in Proc. CVPR, pp. 689–696, IEEE, 2011.

[13] M. Bass, ed., Handbook of Optics: Optical properties ofmaterials, nonlinear optics, quantum optics. OpticalSociety of America, 2009.

[14] W. Matusik, H. Pfister, M. Brand, and L. McMillan,“A data-driven reflectance model,” ACM Trans. onGraph., vol. 22, no. 3, pp. 759–769, 2003.

[15] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang,and Y. Ma, “Robust photometric stereo via low-rankmatrix completion and recovery,” in Proc. ACCV, 2010.

[16] A. Tankus and N. Kiryati, “Photometric stereo un-der perspective projection,” in Proc. ICCV, vol. 1,pp. 611–616, IEEE, 2005.