© leo burnett. 2 “moe”: april 23 “moe” : june 6 3 population size estimation behavioral...
TRANSCRIPT
Automatic Zebra Identification from PhotographsMayank Lahiri
University of Illinois at Chicago
© Leo Burnett
2
The goal, in two pictures
“Moe”: April 23
“Moe” : June 6
3
Identifying zebras by their stripes:Why identify zebras?
Population size estimation
Behavioral studies
Phenotype analysis
4
Identifying zebras by their stripes:And why use stripes?
Tracking devices:
Require anesthetization
Expensive
Unreliable
Broad coverage all but impossible
5
The algorithm: ZebraGoogle
1.Moe 93%2.Alice 5%3.Bob 2%
The “ALGORITHM
”
Database of zebra pictures
Input – “the query”
Output – “zebra ranking”
6
Attempts #1 and #2
Face recognition algorithms▪ Early algorithms too rigid▪ Modern algorithm could work, but are black
boxes
Fingerprint recognition▪ Generally look for well-defined features▪ Rarely deal with occlusion, perspective skew,
varying distance to camera
7
Attempt #3: Shape matching
© Colchester Zoo
y
x
1. Shape contour tracing2. Spline function fitting3. Query and retrieve splines
8
Attempt #3: Shape matching
9
Is there a simpler way?
How much information is there in the data?
10
Is there a simpler way?
How much information is there in the data?
11
Attempt #4: ZEBRA BARCODE Hypothesis: width and spacing of
stripes are distinctive when measured finely
12
Zebras vs. Barcodes (seriously)Zebras are/have:
seldom two-dimensional
frequently obscured
non uniform stripes
high tendency towards pregnancy and violence
afraid of barcode scanners
© Barcodeman
13
The approach
Build a solution by eliminating the problems!
14
PROBLEM #1:“Width” of stripes change with distance
Closer to camera, more pixels for the body
Further away, fewer pixels for the body
10 Megapixel camera = 3648 pixels across, 2736 pixels down1080p HD TV = 1920 pixels across, 1080 pixels down15” MacBook Pro screen = 1440 pixels across, 900 pixels down
15
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 75 104
16
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 75 1.38
17
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 0.85 1.38
18
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
19
PROBLEM #2:Camera is rarely perpendicular to zebra
20
PROBLEM #2:Camera is rarely perpendicular to zebra
Shear transformation flattens small amount of perspective skew [1].
Original image With shear transformation
21
PROBLEM #2:Camera is rarely perpendicular to zebra
Shear transformation flattens small amount of perspective skew [1].
Shear transformation is a special case of affine transformation.
Affine transformation: Ratios of distances along a line are
preserved
22
PROBLEM #2: Camera is rarely perpendicular to zebra
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
23
PROBLEM #2: Camera is rarely perpendicular to zebra
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
0.8 1.4 1.0 0.95 1.39 0.85 1.38
A “strip” of stripes
24
PROBLEM #3:Zebras move, and computers are stupid
1.4 1.0 0.95 1.39 0.85 1.38 0.7
From original photograph in database:
0.83 1.39 0.85 1.38 0.7
Zebra occluded from the left side:
Missed the rightmost black stripe:
1.4 1.0 0.95 1.39 0.85 1.98
Extremely oblique viewing angle:
1.5 1.1 1.2 1.7 1.01 1.66 1.3
25
PROBLEM #3:Zebras move, and computers are stupid
Solution: Dynamic programming Align two strips to minimize “errors” a.k.a. Spell-check, DNA sequence alignment, Needleman-Wunsch
algorithm, Smith-Waterman algorithm, edit distance, dynamic time warping, etc.
Stripe-alignment! occlusion = indel cost image processing errors = indel + matching cost stripe distortion = matching cost strong perspective skew = matching cost
Low alignment “cost” = fewer differences in strips = zebras are very similar
27
PROBLEM #3:Zebras move, and computers are stupid
For a new picture (the “query”):
Read a strip off the body at a known location
Align against all the zebra strips in the database, also from the same location
Rank zebras in the database by the alignment cost of their strips
28
In Theory...
One click per zebra
Analogous to a barcode scanner
Handles occlusion, minor perspective skew
Can be applied to any part of the body
Computationally efficient
29
In Practice. 20 zebras, ~6 photos per zebra =
109 pictures.
“Transcription” errors
30
Evaluating accuracy
Photos were manually identified by Rosemary at Ol’Pejeta Conservancy. Manually coded stripes along the
shoulder
For each photo, rank the closest matches using dynamic programming.
Metric: rank of the correct zebra in the list of closest matches.
31
Results: Rank of correct zebraPro
port
ion
of
qu
eri
es
at
or
belo
w
rank
Average rank = 1.5
32
Results: Where does it fail? No love from Zebra #3 – “01_700”
Photo 87 Zebra 3 Flank R PicID 8159 CORRECT_RANK 4 Photo 65 Zebra 3 Flank L PicID 8142 CORRECT_RANK 9 Photo 63 Zebra 3 Flank R PicID 8170 CORRECT_RANK 5 Photo 52 Zebra 3 Flank R PicID 8166 CORRECT_RANK 2
8159
8142
8170
8166
33
Results: Where does it fail? Worst performance on this picture:
34
Performance
Time to search database of 108 pictures: 0.023 seconds -- my ageing 2006 laptop
If the number of stripes on a zebra is O(1), then the time complexity of a single search is linear in the number of photographs. 1,000 pictures ~ 0.2 seconds 10,000 pictures ~ 3 seconds
35
Applicability
36
Questions?Future work:1. Build an effective user interface2. Get field biologists to discover better ways to
use it!3. Run tests on 3,000+ pictures from January
Kenya trip4. Tweak image processing algorithms
“So you see! There’s no end to the thing you might know, depending how far beyond Zebra
you go.”- Dr. Seuss, Beyond Zebra
References:[1] Hutchison and Barrett . Fourier-Mellin registration of line-detained tabular document images. Intl. J. Doc. Analysis 8(2):87-110, 2006.