04/18/23 CIC 10, 2002 2

Color constancy at a pixel [Finlayson et al. CIC8, 2000]

Idea: plot log(R/G) vs. log(B/G):

14 daylights14 daylights

24 p

atch

es24

pat

ches

04/18/23 CIC 10, 2002 3

Log(R/G)

Log(R/G)

Log(B

/G)

Log

(B/G

)

For every patch, the direction from light color change is about the same!

04/18/23 CIC 10, 2002 4

Why all linear and same direction?

color shading

intensitylight SPD

reflectancesensor

k=1..3

Now let’s make some assumptions:

The image formation equation:The image formation equation:

04/18/23 CIC 10, 2002 5

Assumption 1Assumption 1: Light is ~ Planckian: Light is ~ Planckian (or some other 1D assumption)

Wien’s approximation of a Planckian source:

Note: 1D parameter: T == temperature == light color.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

r/(r+g+b)

g/(

r+g

+b

)

Illuminant Chromaticities

P100

6

Assumption 2: Narrow band sensorsAssumption 2: Narrow band sensors

)()( kkk qq

SONY DXC-930

The Sony Camera has fairly narrow band sensitivities

Using spectral sharpening, we can make almost all sensor sets have this property.

[Finlayson, Drew, Funt]

T

ccSI

kkkk

251 ))(ln(

~lnln

Modified Image FormationModified Image Formation

T

c

kkkkecSI 2

51)(

~

dqSEI kk )()()(~

The kth response

Substituting Narrow-band and Planckian Assumptions

Take logs

Response = light intensity + surface + light color

II )(~

, na

8

ImplicationsImplications

T

ccSI

kkkk

251 ))(ln(

~lnln

We have k equations of the form:

I~

ln is common to all equations and can be removed bysimple differencing at this pixel

This results in k-1 independent equations of the form

T

kjkjkj

,,/ln

reflectanceterm

light colorterm

04/18/23 CIC 10, 2002 9

ImplicationsImplications

The log chromaticities of 7 surfacesviewed under 10 lights

T

kjkjkj

,,/ln

(1) If there are 3 sensors wehave two independent equations

of this form:

(2) For a single surface viewed underdifferent colored lights the log

chromaticities must fall on a line:

(3) Different surfaces induce lines withthe *same* orientation

04/18/23 CIC 10, 2002 10

Luminance

1D invariant

Gray

One degree of freedom is invariantOne degree of freedom is invariantto light changeto light change

11

More formally:

and define

form ratios

define vectors

line in 2D

12

What is this good for?

With certain restrictions, from a 3-band color image we can derive a 1-D

grayscale image which is:

- illuminant invariant

- and so, shadow free

13

Then use edge info. to integrate back without shadows [ECCV02 Finlayson, Hordley, and Drew]

These are approximately the same, except that the invariant edge map has

no shadow edges

14

Other tasks: Tracking, etc.

Tracking result for moving hand under lamp light.[Jiang and Drew, 2003]

04/18/23 CIC 10, 2002 15

But problem: doesn’t always remove all shadows:

Depends on camera sensors

16

How do we find light color change direction?

Sony DXC-930 camera

Mean-subtracted log-chromaticity

(Use robust line-finder)

04/18/23 CIC 10, 2002 17

Problem: invariant image isn’t invariant across illuminants

04/18/23 CIC 10, 2002 18

Gets worse: Kodak DCS420 camera is much less sharp

04/18/23 CIC 10, 2002 19

How to proceed? Try spectral sharpening, since wish to make sensors more narrowband….Or just optimize directly, making invariant image more invariant.

E.g. optimize color-matching functions :

)(),(),( zyx

04/18/23 20

Invariant image for patches apply optimized sensors to any image

Before optimization of sensors After optimization of sensors

21

How to optimize?Firstly, let’s use a linear matrixing transform, taking 31 x 3 sensor matrix Q to a new sensor set:

Should we sharpen to get M?

sensors colors

3 x 3

22

Should we sharpen to get M?

There’s a problem: If we made sensors that were all the same, the definition

makes the invariant go to zero… The more the sensors are alike, the “better”

Sharpening & flattening both work…

04/18/23 CIC 10, 2002 23

So need to use a term to steer away from a rank-reduced M

Optimize on the (correlation coefficient)2 R 2

and encourage high effective rank

are singular values of M

Initialize with data-driven spectral sharpening matrix.

04/18/23 CIC 10, 2002 24

So optimize M:

E.g., color-matching functions: R2 goes from 0.43 to 0.94

04/18/23 CIC 10, 2002 25

HP912 camera:

R2 : 0.86 0.93

entropy : 5.856 5.590 bits/pixel

26

Real image:

entropy 5.295 4.939 bits/pixel with an M

27

Than

ks!