ECCV 2002

Removing Shadows From Images

G. D. Finlayson1, S.D. Hordley1 & M.S. Drew2

1School of Information Systems, University of East Anglia, UK

2School of Computer Science, Simon Fraser University, Canada

ECCV 2002

Overview

Introduction

Shadow Free Grey-scale images

- Illuminant Invariance at a pixel

Shadow Free Colour Images

- Removing shadow edges using illumination invariance

- Re-integrating edge maps

Results and Future Work

ECCV 2002

The Aim: Shadow Removal

We would like to go from a colour image with shadows, to the same colour image, but without the shadows.

ECCV 2002

Why Shadow Removal?

For Computer Vision- improved object tracking, segmentation etc.

For Image Enhancement- creating a more pleasing image

For Scene Re-lighting

- to change for example, the lighting direction

ECCV 2002

What is a shadow?

Region Lit by Sunlight and

Sky-light

Region Lit by Sky-light only

A shadow is a local change in illumination intensity and (often) illumination colour.

ECCV 2002

Removing Shadows

So, if we can factor out the illumination locally (at a pixel) it should follow that we remove the shadows.

So, can we factor out illumination locally? That is, can we derive an illumination-invariant colour representation at a

single image pixel?

Yes, provided that our camera and illumination satisfies certain restrictions ….

ECCV 2002

Conditions for Illumination Invariance

(1) If sensors can be represented as delta functions (they respond only at a single wavelength)

(2) and illumination is restricted to the Planckian locus

(3) then we can find a 1-D co-ordinate, a function of image chromaticities, which is invariant to

illuminant colour and intensity

(4) this gives us a grey-scale representation of our original image, but without the shadows

(it takes us a third of the way to the goal of this talk!)

ECCV 2002

dSEBb

dSEGg

dSERr

)()()(

)()()(

)()()(

)(S

)(E )()( SE

Image Formation

Camera responses depend on 3 factors: light (E), surface (S), and sensor (R, G, B)

ECCV 2002

G()

Sen s

i tiv

i ty

B() R()

=

Delta functions “select” single wavelengths:

RR

Using Delta Function Sensitivities

RRR SEdES

GG BB

RR SEr

GG SEg

BB SEb

ECCV 2002

Characterising Typical Illuminants

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

r/(r+g+b)

g/(

r+g

+b

)

Illuminant Chromaticities Most typical illuminants lie on, or close to, the Planckian locus (the red line in the figure)

So, let’s represent illuminants by their

equivalent Planckian black-body

illuminants ...

ECCV 2002

1

51 1)(

2

T

c

eIcE Here I controls the overall intensity of light, T is the

temperature, and c1, c2 are constants

Planckian Black-body Radiators

But, for typical illuminants, c2>>T. So, Planck’s eqn.

is approximated as:

T

c

eIcE 2

51)(

ECCV 2002

How good is this approximation?

2500 Kelvin

10000 Kelvin

5500 Kelvin

ECCV 2002

For, delta function sensors and Planckian Illumination we have:

Back to the image formation equation

T

c

RRRecISr 2

51)(

T

ccSIr

RRR

251)(lnlnln

Surface Light

Or, taking the log of both sides ...

ECCV 2002

c

b

a

Tk

k

k

b

g

r

s

s

s1

ln

ln

ln

Where subscript s

denotes dependence on reflectance and

k,a,band c are constants. T

is temperature.

Summarising for the three sensors

Constant independentof sensor

Variable dependentonly on reflectance

Variable dependenton illuminant

ECCV 2002

),,(')(

)(' sssfb

bc

bar

That is: there exists a weighted difference of log-opponent chromaticities that depends only on surface reflectance

Factoring out the illumination

bc

ba

Tgb

gr

b

r

ss

ss 1

lnln

lnln

'

'

First, let’s calculate log-opponent chromaticities:

Then, with some algebra, we have:

ECCV 2002

-0.5 0 0.5 1 1.5 2 2.5 3-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

log(r/g)

log

(b/g

)

An example - delta function sensitivities

400 450 500 550 600 650 7000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Wavelength

Re

lati

ve

Se

ns

itiv

ity

B

W R

YG

P

Narrow-band (delta-function sensitivities)

Log-opponent chromaticities for 6 surfaces under 9 lights

ECCV 2002

Deriving the Illuminant Invariant

-0.5 0 0.5 1 1.5 2 2.5 3-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

log(r/g)

log

(b/g

)

Log-opponent chromaticities for 6 surfaces under 9 lights

This axis is invariant to illuminant colour

Rotate chromaticities

ECCV 2002

Normalized sensitivities of a SONY DXC-930 video

camera

A real example with real camera data

400 450 500 550 600 650 7000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Wavelength

Re

lativ

e S

en

siti

vity

Log-opponent chromaticities for 6

surfaces under 9 different lights

ECCV 2002

Deriving the invariant

Log-opponent chromaticities for 6

surfaces under 9 different lights

The invariant axis is now only approximately illuminant

invariant (but hopefully good enough)

Rotate chromaticities

ECCV 2002

Some Examples

ECCV 2002

A Summary So Far

With certain restrictions, from a 3-band colour image we can derive a 1-d grey-scale

image which is:

- illuminant invariant

- and so, shadow free

ECCV 2002

What’s left to do?

To complete our goal we would like to go back to a 3-band colour image, without shadows

We will look next at how the invariant representation can help us

to do this ...

ECCV 2002

Looking at edge information

Consider an edge map of the colour

image ...

And an edge map of the 1-d invariant

image ...

These are approximately the same, except that the invariant edge map has

no shadow edges

ECCV 2002

Removing Shadow Edges

From these two edge maps we can remove shadow edges thus:

Edges = Iorig & Iinv

(Valid edges are in the original image, and in the invariant image)

ECCV 2002

Using Shadow Edges

So, now we have the edge map of the image we would

like to obtain (edge map of the original

image with shadows edges set to zero)

So, can we go from this edge information back to

the image we want? (can we re-integrate the edge

information?).

ECCV 2002

Re-integrating Edge Information

Of course, re-integrating a single edge map will give us a grey-scale image:

So, we must apply any procedure to each band of the colour image separately:

Red

Green

Blue

Original

Colour Channels

Edge Maps of Channels

Shadow Edges

Removed

Re-integrated

ECCV 2002

Re-Integrating Edge Information

The re-integration problem has been studied by a number of researchers:

- Horn

- Blake et al

- Weiss ICCV ‘01 (Least-Squares)

- ...

- Land et al (Retinex)The aim is typically to derive a reflectance image from an

image in which illumination and reflectance are confounded.

ECCV 2002

Weiss’ Method

Weiss used a sequence of time varying images of a fixed scene to determine the reflectance edges of

the scene

His method works by determining, from the image sequence, edges which correspond to a change in

reflectance(Weiss’ definition of a reflectance edge is an edge which

persists throughout the sequence)Given reflectance edges, Weiss re-integrates the

information to derive a reflectance image

In our case, we can borrow Weiss’ re-integration procedure to recover our shadow-free image.

ECCV 2002

Re-integrating Edge Information

Let Ij(x,y) represent the log of a single band of a colour image

x is the derivative operator in the x

direction

y is the derivative operator in the y

direction

We first calculate:

jYXj ITTI

T is the operator that sets shadow

edges to zero

This summarises the process of detecting and removing shadow edges

ECCV 2002

Re-integrating Edge Information

jYXj ITTI

To recover the shadow free, image we want to invert this Equation

To do this, we first form the Poisson Equation

jYYXXj ITTI

We solve this (subject to Neumann boundary conditions) as follows:

ECCV 2002

Re-integrating Edge Information

jYYXXrec IIj

1

jYYXXj ITTI We solve by applying the

inverse Laplacian

Note: the inverse operator has no

Threshold

Applying this process to each of the three channels recovers a log image without shadows.

ECCV 2002

A Summary of Re-integration

1. Iorig = Original colour image, Iinv = Invariant image

2. For j=1,2,3 Ijorig = jth band of Iorig

3. Remove Shadow Edges: Edges = Ijorig & Iinv

4. Differentiate the thresholded edge map

5. Re-integrate the image

6. Goto 3

ECCV 2002

Some Remarks

The re-integration step is unique up to an additive constant (a multiplicative constant in linear image space

Fixing this constant amounts to applying a correction for illumination colour to the image. Thus we choose suitable

constants to correct for the prevailing scene illuminant

In practice, the method relies upon having an effective thresholding step T, that is, on effectively locating the

shadow edges.

As we will see, our shadow edge detection is not yet perfect

ECCV 2002

Shadow Edge Detection

The Shadow Edge Detection consists of the following steps:

1. Edge detect a smoothed version of the original (by channel) and the invariant images

2. Threshold to keep strong edges in both images

3. Shadow Edge = Edge in Original & NOT in Invariant

4. Applying a suitable Morphological filter to thicken the edges resulting from step 3.

Canny or SUSAN

This typically identifies the shadow edges plus some false edges

ECCV 2002

An Example

Detected Shadow Edges

OriginalImage

InvariantImage

Shadow Removed

ECCV 2002

A Second Example

Detected Shadow Edges

OriginalImage

InvariantImage

Shadow Removed

ECCV 2002

More Examples

Detected Shadow Edges

OriginalImage

InvariantImage

Shadow Removed

ECCV 2002

More Examples

Detected Shadow Edges

OriginalImage

InvariantImage

Shadow Removed

ECCV 2002

A Summary

We have presented a method for removing shadows from images

The method uses an illuminant invariant 1-d image representation to identify shadow edges

From the shadow free edge map we re-integrate to recover a shadow free colour image

Initial results are encouraging: we are able to remove shadows, even when shadow edge definition is not perfect

ECCV 2002

Future Work

We are currently investigating ways to more reliably identify shadow edges ...

… or to derive a re-integration which is more robust to errors (Retinex?)

Currently deriving the illuminant invariant image requires some knowledge of the capture device’s characteristics

- We show in the paper how to determine these characteristics empirically and we are working on making this

process more robust

ECCV 2002

Acknowledgements

The authors would like to thank Hewlett-Packard Incorporated for their support of this

work.