Predicting Environment Illumination Effects on Material Appearance

Jirı Filip Michal HaindlInstitute of Information Theory and Automation of the AS CR

jiri.filip@utia.cz

Jaroslav StancıkVaros s.r.o.

AbstractEnvironment illumination is a key to achieving a re-

alistic visualization of material appearance. One wayto achieve such an illumination is an approximation byrendering of the material surface lit by a finite set ofpoint light sources. In this paper we employed visualpsychophysics to identify a minimal number of pointlight sources approximating realistic illumination. Fur-thermore, we analyzed stimuli images and correlationof their statistics with obtained psychophysical data. Fi-nally, image statistics were identified which can predictsuch a minimal environment representation for threetested materials, depending on the visual properties ofthe illumination environment.

1 IntroductionAchieving a realistic appearance of real-world

materials in virtual scenes is impossible withoutenvironment illumination. Such an environment can beapproximated by its model to any required precision.In an idealized model we could assume that parametersof the environment are known. Unfortunately, theexact environment parameters are seldom known,therefore an application of the model mostly introducesunknown approximation errors. One way to achievethe desired illumination is to use a global illuminationapproach, generating an image by tracing the path oflight through pixels in an image plane and simulatingthe effects of its encounters with virtual objects. Thisapproach theoretically provides excellent quality at thecost of high computational requirements, so achievingreal-time rendering in this way requires specializedgraphics hardware. Several image-based approachesemerged to remedy these drawbacks. These approachestypically represent captured illumination environmentby a finite set of directional lights. Although methodsexist nowadays allowing real-time rendering fromsuch environment representations, they rarely takeinto account the visual properties of environmentillumination (further denoted as EI) and/or the materialbeing illuminated.

01 02 03 04 05

06 07 08 09 10

Figure 1. Tested illumination environments.

Prior work: The idea of illumination by means of en-vironment maps was first introduced in [2] and furtherapplied in [3]. As global illumination algorithms basedon ray-tracing were computationally demanding, im-portance sampling methods were developed [1], [4] rep-resenting the environment map by means of a finite setof directional lights. The importance of context envi-ronment information and specular reflection for percep-tion of surface reflectance was studied by [7] with theconclusion that observers can match correct environ-ment based only on the surface reflectance. [10, 11, 8]studied effect of shape, EI and reflectance on humanperception. Subjects’ perception of realistically tex-tured scenes was analyzed [9, 6] and used for [5] ma-terial appearance data compression. We are not awareof any work that has applied psychophysical methods toassess both bidirectional textured material appearanceand EI-dependent perceptual effects.Paper contribution: In this paper, we use a psy-chophysical study (Section 3) to determine the simplestrepresentation of EI which preserves visual fidelity ofthe original environment. Furthermore, we use this in-formation to derive a statistical feature (Section 4) pre-dicting such representations automatically for novel EIs(Section 5).

2 Data Preparation

Environment Acquisition and RepresentationFor purposes of our work we acquired ten different

illumination environments (see Fig. 1). Each environ-ment was obtained by merging eight overlapping pho-tos taken with a fish-eye lens. Each such environment(2048 × 1024 pixels) was represented by a finite setof directional lights using the median-cut algorithm [4]as follows. Every environment image is first gamma-

21st International Conference on Pattern Recognition (ICPR 2012)November 11-15, 2012. Tsukuba, Japan

978-4-9906441-1-6 ©2012 IAPR 2075

correction compensated (γ = 2.2) to obtain the orig-inal linear response of the camera sensor. The algo-rithm then iteratively splits the input environment mapimage into even sets of regions by cutting the regionsat the point where the medians of the cut regions’ lu-minance are mutually equal. For rendering, the envi-ronment map is wrapped over a sphere surrounding theilluminated object. To compensate over-representationof regions near the poles, the luminance pixels of theenvironment map are weighted by cosine of the incli-nation angle. Fig. 2-top shows the compensated envi-ronment 02 represented by 128 light by means of themedian-cut algorithm. An example of the sphere illu-

environment map 02 represented by 128 lights

16 lights 32 lights 64 lights 128 lights

Figure 2. Example of environment illumination. Thespheres are illuminated by different numbers of lights.

minated by a different number of lights approximatingthe environment by median-cut algorithm is shown inFig. 2-bottom. The quality of shading and presence ofdisturbing seams on the object’s surface are clearly de-pendent on a number of lights. Our goal is to identifythe lowest number of lights that preserve the naturalnessof EI and suppress these visible artifacts.

Materials RepresentationContrary to using reflectance only (BRDF) [10, 11, 8]

we represented material appearance using view- andillumination-dependent textures (BTF) [5] samples alu(Fig. 2), corduroy, and leather (Fig. 3) from the BTFDatabase Bonn1.

Rendering of BTF using a finite set of ni directionallights can be described as their convolution with eachpixelLr(x, y, v) =

ni∑i=1

BTF (x, y, ωi, ωv)Li(x, y, ωi), (1)

where ωi, ωv are illumination and viewing direc-tions, Li(x, y, ωi) is radiance of individual lights andLr(x, y, v) is final radiance reflected by the sample.

3 Psychophysical ExperimentThe goal of the experiment was to analyze the

material-dependent sensitivity of human subjects to a1http://btf.cs.uni-bonn.de/

simplified representation of different real-world illumi-nation environments.

Pairs of static images of size 640 × 640 pixels wereused as experimental stimuli, representing a materialBTF rendered on a sphere. Each pair consisted of a ref-erence rendering using the 256 lights and another ren-dering using a set of 16, 32, 64, 128, or 256 lights. Pairsof images were displayed simultaneously, side-by-sideas shown in Fig. 3. We used the shape of a spherebecause its geometry provides a wide range of illumi-nation and viewing combinations without introducingunwanted higher curvatures, which could mask soughtartifacts. All three test materials were rendered in tentest environments. As some environments were used inmultiple views, the total number of different tested illu-minations was fifteen. Azimuthal positions of views areshown in Fig 1 as red (the first view), green (the secondview), and blue (the third view) dots at the bottom ofenvironment maps. Given three materials, fifteen envi-ronments, and five quality levels of illumination repre-sentation, the total number of stimuli was 225.

Figure 3. Experimental stimuli showing a sphere inillumination environment (256 vs. 32 lights).

Twenty volunteers (8 females, 12 males) participatedin the experiment. All participants had normal or cor-rected to normal vision and were naive with respect tothe purpose and design of the experiment. The partic-ipants were shown the stimuli in a random order andasked a yes-no question: ’Can you spot any differencesin the material covering the two objects?’. Participantswere asked to answer in 3 seconds to avoid decisionbased on a minor local difference. It turned out that 94% of the responses were made in 2 seconds. All stim-uli were presented on a calibrated monitor during nightunder dim room ceiling light.

By averaging the responses of all participants, weobtained psychometric data relating the average re-sponse to different numbers of lights, environments, andmaterials. The numbers of lights required for each EIand material were obtained from a psychometric func-tion using 25% threshold [5] and are shown in Fig. 4.

4 Relation to Illumination Maps StatisticsWe analyzed the relation between obtained human

visual attention and feasibly computable statistics ofstimuli and environment maps. We were inspired bythe image salience methods, therefore we took a cir-

2076

Figure 4. Estimated minimal number of lights for EIsand materials.

cular Gaussian-weighted window (with shape speci-fied by a parameter σ) to obtain the local mean valueµL, standard deviation σL, and energy eL (eL(x, y) =√(∂I/∂x)2 + (∂I/∂y)2). These features were applied

on environment maps (I) down-sampled to resolution256 × 128 pixels. Results for the Gaussian windowσ = 5 are illustrated on the environment map in Fig. 5.As these features describe local properties of individualpixels, their mean µ(µL), µ(σL), µ(eL) and standarddeviation values σ(µL), σ(σL), σ(eL) were computedfor EI map description.

EI map µL σL eLFigure 5. Example of the tested statistics (σ = 5)

.Stimuli Images AnalysisFirst we used the six proposed features for descriptionof stimuli images used in the experiments. As these im-ages comprise two distinct parts – an environmentallyilluminated textured sphere and background – we ana-lyzed these parts separately. Tab. 1 shows correlationof the features with results of the main psychophysicalexperiment. The correlation was computed as a Pear-

Table 1. Correlation R of the statistics with results ofthe experiment (σ = 10).

µ(µL) µ(σL) µ(eL) σ(µL) σ(σL) σ(eL)

alu -0.13 0.35 0.28 0.10 -0.11 0.17corduroy -0.15 0.35 0.52 0.35 -0.02 0.23leather -0.23 0.37 0.48 0.15 0.23 0.17backgr. 0.37 0.30 0.36 0.63 -0.22 0.01EI 0.39 0.19 0.45 0.61 0.25 0.62

son correlation coefficient RX,Y = E[(X−µX)(Y−µY )]σXσY

,where X,Y are compared average subject responseswith a feature’s values, µ and σ are their means andstandard deviations. The table shows coefficients forthree tested materials and background for a width ofGaussian window σ = 10. Correlation of the featuresfor textured sphere was relatively low (the highest val-ues R ≈ 0.5) for µ(σL) and µ(eL). What is surprising

is a higher correlation (R = 0.63) of σ(µL) computedin a stimuli background. This suggests the hypothesisthat variance in stimuli background or contrast betweenthe background and textured sphere significantly influ-ences perception of visual artifacts caused by insuffi-cient representation of environment illumination.

Environment Maps AnalysisThe results obtained from analysis of the stimuli im-age opens a question of whether it might be general-ized to environment map images. For this reason weidentified the region in the maps roughly correspond-ing to the background visible in the stimuli images (seeFig. 5-left) and computed the proposed features directlyon such parts of environment maps. Results of the fea-tures’ correlation with subjects’ responses from the ex-periment are shown in the last row of Tab. 1. Againthere is a high (R = 0.61) correlation of σ(µL) (vari-ance of the locally averaged part of the environmentmap) regardless of the size of σ. Such correlation val-ues were also achieved by a local energy feature σ(eL),however only for σ > 9.

5 Illumination Complexity PredictionIn the applied context of this research we aimed at

development of a simple computational method for au-tomatic prediction of a minimal acceptable representa-tion of illumination environment. Such a method woulddescribe dependency of an optimal number of lights onmaterial statistics and environment map statistics. Forthis purpose, we plotted dependency of the estimatednumber of lights from the experiment on values of thebest feature σ(µL) in Fig. 6. While (a) shows such adependency of σ(µL) computed on the tested materialsand background from stimuli images, (b) shows depen-dency of σ(µL) on the tested materials and their aver-age, computed directly from environment maps. Theplot for each material contains 15 points correspond-ing to the tested illuminations. In all plots we can seethe tendency toward requirement of an increasing num-ber of lights with the increase of a feature’s value. Toexpress this dependence analytically we fitted the datapoints by polynomials of different ranks. Due to a lownumber of data points and their low consistence, how-ever, this resulted in data overfitting. Therefore, the datapoints were finally approximated by a linear regression(y = kd+ q) in a least square sense, resulting in the redoutlines shown in the plots. From the slants of the out-lines it is apparent that this tendency is clearly material-dependent. Vertical offset of the fitted outlines confirmsour previous finding that visual artifacts, introduced byrepresentation of minimal number of lights, were theleast apparent for structured corduroy and the most ap-parent for diffuse leather.

2077

a)k=0.2,q=99 k=0.4,q=70 k=1.8,q=107 k=1.0,q=74

b)k=0.3,q=96 k=1.1,q=44 k=2.1,q=84 k=1.1,q=77

Figure 6. Dependence of number of lights estimatedfrom the experiment on σ(µL): feature values (a) com-puted from stimuli images, (b) computed directly fromenvironment maps. The red outlines - LS linear fit.

To show a practical application of our conclusionswe acquired an additional five environment maps asis shown in the first row of Fig. 7, and computedthe σ(µL) feature on backgrounds of selected views(red dots at the bottom of the images). A number oflights was predicted from the regression parameters k, q(Fig. 6) and the feature values. When observing render-ings of materials in all test environments there are notany visible artifacts, and the minimal numbers of lightsrequired for rendering vary across both tested environ-ments and materials.

6 ConclusionsIn this paper we analyzed human observers’ ability

to detect visual artifacts caused by under-representationof environmental illumination by a limited set of direc-tional lights. We have found that this ability signifi-cantly depends on the type of illuminated material aswell as on the type of illumination environment. Gener-ally, an increase of the standard deviation of local meanvalue in an environment map increases the ability of hu-man observers to detect visual artifacts introduced bymeans of an insufficient number of lights representingthe environment. Therefore, the more visually com-plex/rich the environment is the more lights are requiredfor its proper representation. We have shown how thisnumber of lights depends on the type of illuminated ma-terial, and how it can be predicted from an environmentmap’s statistics.

7 AcknowledgmentsWe would like to thank the University of Bonn for

providing BTF measurements and volunteers for partic-ipation in the experiment. This work has been supportedby the Grants GACR 103/11/0335 and 102/08/0593, ECMarie Curie ERG 239294, and CESNET grant 409.

σ(µL)=29.2 19.5 41.7 41.0 38.2

105 102 109 109 108 lights

75 65 88 87 84 lights

147 126 173 172 166 lights

Figure 7. Illumination of the tested materials by thenumber of lights predicted from the EI statistics.

References

[1] S. Agarwal, R. Ramamoorthi, S. Belongie, andH. Jensen. Structured importance sampling of environ-ment maps. In SIGGRAPH 2003, pages 605–612, 2003.

[2] J. Blinn and M. Newell. Texture and reflection in com-puter generated images. Communications of the ACM,19(10):542–547, 1976.

[3] P. Debevec. Rendering synthetic objects into realscenes: Bridging traditional and image-based graphicswith global illumination and high dynamic range pho-tography. In ACM SIGGRAPH 1998, pages 1–10, 1998.

[4] P. Debevec. A median cut algorithm for light probe sam-pling. In SIGGRAPH 2005 (Posters), 2005.

[5] J. Filip, M. Chantler, P. Green, and M. Haindl. A psy-chophysically validated metric for bidirectional texturedata reduction. ACM TOG, 27(5):138, 2008.

[6] J. Filip, M. Chantler, and M. Haindl. On uniform resam-pling and gaze analysis of bidirectional texture func-tions. ACM TAP, 6(3), August 2009.

[7] R. W. Fleming, R. O. Dror, and E. H. Adelson. Real-world illumination and the perception of surface re-flectance properties. Journal of Vision (2003), (3):347–368, 2003.

[8] J. Krivanek, J. Ferwerda, and K. Bala. Effects ofglobal illumination approximations on material appear-ance. ACM TOG, 29(4):112, 2010.

[9] J. Meseth, G. Muller, R. Klein, F. Roder, and M. Arnold.Verification of rendering quality from measured BTFs.In APGV, 2006.

[10] G. Ramanarayanan, J. Ferwerda, B. Walter, and K. Bala.Visual equivalence: Towards a new standard for imagefidelity. ACM TOG, 26(3), 2007.

[11] P. Vangorp, J. Laurijssen, and P. Dutre. The influenceof shape on the perception of material reflectance. ACMTOG, 26(3), 2007.

2078