Physical, computational andperceptual factors in color-based object identification

Qasim Zaidi

Color-Based Object Identification

Can you identify which parts are sunny or shady?Why do you infer illuminant rather than material changes?Can you identify similar foliages from sunny to shady parts?

Strategies for color-based object identification

Color Constancy: Constancy of subjective appearance byadaptation to the illuminant ( Ives, Helson, etc.).

Inverse Optics Methods: Estimate spectrum of illuminant andinfer reflectance spectra which are physical invariants (Maloney& Wandell, D’Zmura & Iverson, Brainard & Freeman etc.).

Alternatives: (i) Neural algorithms based on cone-catches.(ii) Judgments based on percepts of color and contrast.

Wallach (1963). The perception of neutral colors. Scientific American, 208,107-116.

Flat achromatic objects with identical poses and backgrounds

(Contrast?)

Ripamonti, Bloj, Hauck, Mitha, Greenwald, Maloney & Brainard (2004). Measurementsof the effect of surface slant on perceived lightness. Journal of Vision, 4(9), 717-763.

Flat objects with different poses and without a background

(Separating illuminant and reflectance contributions to brightness?)

Robilotto & Zaidi (2004). Limits of lightness identification for real objects under naturalviewing conditions. Journal of Vision, 4(9), 779-797.

3-D objects with shadowing, highlights etc.

(Brightness dissimilarity?)

•Three of the objects are made of identical paper while one is made ofa different paper.

•Illumination on the right is a quarter of the illumination on the left.

What is the number of the odd object?

Experiment 1a

Patterned 3-D objects

“2” is the number of the odd object!

What strategy did you follow?

Stimuli

Calibration

Equipment

Method of constant-stimuli

•Discrimination threshold (within illuminants) from % side-correct

•Identification threshold (across illuminants) from % object-correct

Two Hypothetical Observers

PhotometricSimilarity

InverseOptics

Results of Mean Reflectance Identification

Note that identification is worse than discrimination for darker testsunder higher illumination and lighter tests under lower illumination

Results of Mean Reflectance Identification

Note that identification is worse than discrimination for darker testsunder higher illumination and lighter tests under lower illumination

Results of Mean Reflectance Identification

Note that identification is worse than discrimination for darker testsunder higher illumination and lighter tests under lower illumination

Mean reflectance identification based on brightnessdissimilarity responses of an adaptation model

Mean Luminanceof Stimulus

Brightnessof Stimulus

gain

0

1

xi

G(I) = k/(k+I)

R(I) = Φ(G(I)*I)

•Choose the object that is most different in brightness.•Disregard the material and the illumination difference.

What is the number of the odd object?

Experiment 2

Identification predicted from Brightness Dissimilarity

Note that identification is worse than discrimination for the sameconditions as the material identification experiment.

Comparison of discrimination and identification thresholdsExp. 1a (Material identification) vs Exp. 2 (Brightness dissimilarity)

Does pattern contrast help in mean reflectance identification?

•Three of the objects are made of identical paper while one is made ofa different paper.

•Illumination on the right is a quarter of the illumination on the left.

What is the number of the odd object?

Experiment 1b

“3” is the number of the odd object!

What strategy did you follow?

Results of Reflectance Contrast Identification

Identification is worse than discrimination for lower-contrast tests underlower illumination and higher-contrast tests under higher illumination

Results of Reflectance Contrast Identification

Identification is worse than discrimination for lower-contrast tests underlower illumination and higher-contrast tests under higher illumination

Object identification predicted by similarity of perceived contrast(surface scatter reduces physical contrast)

Contrast constancy Surface scatter

•Choose the object that is most different in contrast.•Disregard the material and the illumination difference.

What is the number of the odd object?

Experiment 3

Identification predicted by Contrast Dissimilarity

Note that identification is worse than discrimination for the sameconditions as the material identification experiment.

Comparison of discrimination and identification thresholds

Exp. 1b (Material identification) vs Exp. 3 (Contrast dissimilarity)

STRATEGIES FOR IDENTIFICATION OFPATTERNED ACHROMATIC OBJECTS

For patterned 3-D objects, in some conditions material identification islimited solely by the limen of brightness or contrast discrimination,but in other conditions material identification is considerably worse.

Brightness dissimilarity and contrast dissimilarity strategies, notinverse optics, reproduce both types of results.

A brightness dissimilarity strategy is limited in accuracy byincomplete adaptation, and leads to large biases.

A contrast dissimilarity strategy is limited in accuracy by contrastreduction due to surface scatter, and biases will be proportional to theroughness of surfaces.

Lightness vs Color Identification

Can you identify which parts are sunny or shady?Can you identify trees with similar foliage from sun to shade?Is it harder to separate local reflectance from local shading?

Physical invariants for color identification

For natural and man-made materials, changes in spectra of reflectedlights between two illuminant conditions are complex, but changes incone-absorptions for large sets are simply multiplicative, hence rank-orders and cone-contrasts are preserved (Dannemiller; Nascimento &Foster; Zaidi, Spehar & DeBonet).

Neural invariants for color identification

In post-receptoral cordinates, the color conversion is translationalfor L/(L+M), and multiplicative for S/(L+M) (Zaidi, Spehar &DeBonet).

ALTERNATIVES TO INVERSE OPTICS

0

0.01

0.02

0.03

0.04

0.05

0.06

0.5 0.6 0.7 0.8

L/(L+M)

Equal Energy Light

0.5 0.6 0.7 0.8

L/(L+M)

Skylight

0.5 0.6 0.7 0.8

L/(L+M)

EE --> Skylight

Recovery of Lambertian Objects

L/(L+M)

To solve the correspondence problem for a subset of reflectances underequal-energy light with the superset under skylight, assume that theaffine transformation holds, and derive its parameters by matchingpatterns of chromaticities.This process should lead to generally correct identification, with errorsonly when reflectances diverge from the affine color conversion.

Using parallel color difference vectors or color contrasts, will lead toapproximately correct identifications for many but not all reflectances.

Divergences from parallelism will depend on the perceptual color space.

ALTERNATIVES TO INVERSE OPTICS

Three of the objects are identical, but the fourth is made from adifferent paper. The illumination on the left is bluer and the illumination

on the right is yellower.

Which object is made from a different paper than the other three?

Exp 4: COLOR BASED OBJECTIDENTIFICATION

What strategy did you follow?

Number 1 is made from a differentpaper than the other three objects.

Stimuli

LegendEach data point is the result of 10 observations

O = side correct < 80%, i.e. observer cannot discriminate betweenstandard and test under the same illuminant.

O = object correct ≥ 80%, i.e. observer identifies correct objects acrossilluminants.

O = object correct ≤ 20%, i.e. observer misidentifies the standard as theodd object.

O = 20% < object correct < 80%, i.e. no reliable identification acrossilluminants.

O = test object under bluish light

O = test object under yellowish light

Instructions: Three of the objects are made from identical papers whileone is made from a different paper. Indicate the number of the object

that is made from a different paper than the other three

Observer: RH – uninformed, experienced

Object Identification ResultsObject identification across illuminants is not perfect.

Even when tests can be discriminated from the standards, insome cases identification is not possible and in others thestandard is misidentified as the odd object.

The results form a complex pattern and demonstrate thatidentification/misidentification is not solely a function of colordistance from the standard.

Can these patterns be explained by a simple perceptual strategythat compares perceived colors of objects to perceived colors ofilluminants?

Exp 5: RELATIVE COLORCATEGORIZATION

Instructions: Compared to the standard object, is the test predominantlyyellower, bluer, redder, or greener?

Observer: QZ – informed, experienced

MethodsStandard and test under different lights

Standard and test under the same light

Legend

The C O L O R of the circle corresponds to observer QZ’s judgment ofthe test object being predominately yellower, bluer, redder or greener

than the standard

O = no judgment could be made because the standard and test couldnot be discriminated

O = test object under bluish light

O = test object under yellowish light

Relative Color Categorization ResultsStandard and test under different lights

* Standards under different lights are reliably seen as differing in color,i.e. the observer cannot rely on appearance constancy to do objectidentification.* Color categorization with respect to the standard under the other lightcannot explain the observer’s results in the identification experiment.

Standard and test under the same lights

* In all cases the tests separated into two relative color groups withrespect to the standard under the same light.* QZ’s relative color categories correspond to clusters of correct andincorrect identification by RH.

Observer algorithm for color basedobject identification across illuminants

1.Identify the illuminant on the test (It) by findingthe side on which the two objects havedifferent colors.

2. Compare the colors of the backgrounds tojudge the change in color from Is to It.

3. Pick the object most dissimilar to the othersalong the perceived It - Is color dimension asthe odd object.

Predictions

2: Tests under the bluish/greenish light:Tests perceived as bluer than the standard under the same light inthe categorization experiment will be picked correctly as the oddobject in the identification experiment.Tests perceived as yellower than the standard under the same lightwill not be picked as the odd object unless they are percieved to bealmost as yellow or yellower than the standards under the yellowlight.

1. Tests under the yellowish/reddish light:

Tests perceived as yellower/redder than the standard under thesame light in the categorization experiment, will be picked correctlyas the odd object in the identification experiment.Tests perceived as bluer/greener than the standard under thesame light, will not be picked as the odd object unless they areperceived to be almost as blue/green or bluer/greener than thestandards under the blue light.

Results

241554M78191247TOTAL

544743ITOTALMGI

Of the tests that observer RH could discriminate reliably from the standard,92% of correct identifications and 79% of misidentifications were predictedcorrectly by observer QZ’s color categorizations.

Observed

Predicted

* We used QZ’s relative color categorizations to predict RH’s objectidentification results by using the above algorithm.* In the table, I = correct identification, M = misidentification, G = neithercorrect nor incorrect identification

STRATEGIES FOR IDENTIFICATION OFCHROMATIC MATERIALS

•In color-based object identification, observers followa heuristic strategy that only requires judging therelative colors of the illuminants, and the relativecolors of the objects.•This strategy is essentially similar to the strategy forlightness based object identification.

Color-Based Object Identification

It is clear that the green foliage under the sun is not the samefoliage as the red, orange or yellow foliage in the shade.However, is it the same as the paler green or the darker green?

References•Zaidi, Q., Spehar, B. and DeBonet, J.S. Color constancy invariegated scenes. J. Opt. Soc. Am., A14, 2608-2621, 1997.•Zaidi, Q. Identification of illuminant and object colors: heuristicsbased algorithms. J. Opt. Soc. Am., A15, 1767-1776, 1998.•Zaidi, Q. Color constancy in a rough world. Color Research andApplication, 26, S192-S200, 2001.•Khang, B. and Zaidi, Q. Cues and strategies for color constancy.Vision Research, 42, 211-226, 2002.•Smithson, H. and Zaidi, Q. Color constancy in context: roles of localadaptation and reference levels. Journal of Vision, 4, 693-710, 2004.•Robilotto, R. and Zaidi, Q. Lightness identification of patterned three-dimensional real objects. Journal of Vision, 6, 18-36, 2006.•Bostic, M., Robilotto, R. and Zaidi, Q. Reflectance identification ofreal colored objects across illuminants. VSS 2006.

Acknowledgments: NEI EY07556 & EY13312