© leo burnett. 2 “moe”: april 23 “moe” : june 6 3 population size estimation behavioral...

Automatic Zebra Identification from PhotographsMayank Lahiri

University of Illinois at Chicago

© Leo Burnett

2

The goal, in two pictures

“Moe”: April 23

“Moe” : June 6

3

Identifying zebras by their stripes:Why identify zebras?

Population size estimation

Behavioral studies

Phenotype analysis

4

Identifying zebras by their stripes:And why use stripes?

Tracking devices:

Require anesthetization

Expensive

Unreliable

Broad coverage all but impossible

5

The algorithm: ZebraGoogle

1.Moe 93%2.Alice 5%3.Bob 2%

The “ALGORITHM

”

Database of zebra pictures

Input – “the query”

Output – “zebra ranking”

6

Attempts #1 and #2

Face recognition algorithms▪ Early algorithms too rigid▪ Modern algorithm could work, but are black

boxes

Fingerprint recognition▪ Generally look for well-defined features▪ Rarely deal with occlusion, perspective skew,

varying distance to camera

7

Attempt #3: Shape matching

© Colchester Zoo

y

x

1. Shape contour tracing2. Spline function fitting3. Query and retrieve splines

8

Attempt #3: Shape matching

9

Is there a simpler way?

How much information is there in the data?

10

Is there a simpler way?

How much information is there in the data?

11

Attempt #4: ZEBRA BARCODE Hypothesis: width and spacing of

stripes are distinctive when measured finely

12

Zebras vs. Barcodes (seriously)Zebras are/have:

seldom two-dimensional

frequently obscured

non uniform stripes

high tendency towards pregnancy and violence

afraid of barcode scanners

© Barcodeman

13

The approach

Build a solution by eliminating the problems!

14

PROBLEM #1:“Width” of stripes change with distance

Closer to camera, more pixels for the body

Further away, fewer pixels for the body

10 Megapixel camera = 3648 pixels across, 2736 pixels down1080p HD TV = 1920 pixels across, 1080 pixels down15” MacBook Pro screen = 1440 pixels across, 900 pixels down

15

PROBLEM #1:“Width” of stripes change with distance

Solution: measure widths relative to the previous stripe

52 46 64 66 63 88 75 104

16

PROBLEM #1:“Width” of stripes change with distance

Solution: measure widths relative to the previous stripe

52 46 64 66 63 88 75 1.38

17

PROBLEM #1:“Width” of stripes change with distance

Solution: measure widths relative to the previous stripe

52 46 64 66 63 88 0.85 1.38

18

PROBLEM #1:“Width” of stripes change with distance

Solution: measure widths relative to the previous stripe

52 .8 1.4 1.0 .95 1.39 0.85 1.38

19

PROBLEM #2:Camera is rarely perpendicular to zebra

20

PROBLEM #2:Camera is rarely perpendicular to zebra

Shear transformation flattens small amount of perspective skew [1].

Original image With shear transformation

21

PROBLEM #2:Camera is rarely perpendicular to zebra

Shear transformation flattens small amount of perspective skew [1].

Shear transformation is a special case of affine transformation.

Affine transformation: Ratios of distances along a line are

preserved

22

PROBLEM #2: Camera is rarely perpendicular to zebra

Solution: measure widths relative to the previous stripe

52 .8 1.4 1.0 .95 1.39 0.85 1.38

23

PROBLEM #2: Camera is rarely perpendicular to zebra

Solution: measure widths relative to the previous stripe

52 .8 1.4 1.0 .95 1.39 0.85 1.38

0.8 1.4 1.0 0.95 1.39 0.85 1.38

A “strip” of stripes

24

PROBLEM #3:Zebras move, and computers are stupid

1.4 1.0 0.95 1.39 0.85 1.38 0.7

From original photograph in database:

0.83 1.39 0.85 1.38 0.7

Zebra occluded from the left side:

Missed the rightmost black stripe:

1.4 1.0 0.95 1.39 0.85 1.98

Extremely oblique viewing angle:

1.5 1.1 1.2 1.7 1.01 1.66 1.3

25

PROBLEM #3:Zebras move, and computers are stupid

Solution: Dynamic programming Align two strips to minimize “errors” a.k.a. Spell-check, DNA sequence alignment, Needleman-Wunsch

algorithm, Smith-Waterman algorithm, edit distance, dynamic time warping, etc.

Stripe-alignment! occlusion = indel cost image processing errors = indel + matching cost stripe distortion = matching cost strong perspective skew = matching cost

Low alignment “cost” = fewer differences in strips = zebras are very similar

27

PROBLEM #3:Zebras move, and computers are stupid

For a new picture (the “query”):

Read a strip off the body at a known location

Align against all the zebra strips in the database, also from the same location

Rank zebras in the database by the alignment cost of their strips

28

In Theory...

One click per zebra

Analogous to a barcode scanner

Handles occlusion, minor perspective skew

Can be applied to any part of the body

Computationally efficient

29

In Practice. 20 zebras, ~6 photos per zebra =

109 pictures.

“Transcription” errors

30

Evaluating accuracy

Photos were manually identified by Rosemary at Ol’Pejeta Conservancy. Manually coded stripes along the

shoulder

For each photo, rank the closest matches using dynamic programming.

Metric: rank of the correct zebra in the list of closest matches.

31

Results: Rank of correct zebraPro

port

ion

of

qu

eri

es

at

or

belo

w

rank

Average rank = 1.5

32

Results: Where does it fail? No love from Zebra #3 – “01_700”

Photo 87 Zebra 3 Flank R PicID 8159 CORRECT_RANK 4 Photo 65 Zebra 3 Flank L PicID 8142 CORRECT_RANK 9 Photo 63 Zebra 3 Flank R PicID 8170 CORRECT_RANK 5 Photo 52 Zebra 3 Flank R PicID 8166 CORRECT_RANK 2

8159

8142

8170

8166

33

Results: Where does it fail? Worst performance on this picture:

34

Performance

Time to search database of 108 pictures: 0.023 seconds -- my ageing 2006 laptop

If the number of stripes on a zebra is O(1), then the time complexity of a single search is linear in the number of photographs. 1,000 pictures ~ 0.2 seconds 10,000 pictures ~ 3 seconds

35

Applicability

36

Questions?Future work:1. Build an effective user interface2. Get field biologists to discover better ways to

use it!3. Run tests on 3,000+ pictures from January

Kenya trip4. Tweak image processing algorithms

“So you see! There’s no end to the thing you might know, depending how far beyond Zebra

you go.”- Dr. Seuss, Beyond Zebra

References:[1] Hutchison and Barrett . Fourier-Mellin registration of line-detained tabular document images. Intl. J. Doc. Analysis 8(2):87-110, 2006.

mlahiri@gmail.com