counting crowded moving objects vincent rabaud and serge belongie department of computer science and...
TRANSCRIPT
![Page 1: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/1.jpg)
Counting Crowded Moving Objects
Vincent Rabaud and Serge BelongieDepartment of Computer Science and Engineering
University of California, San Diego{vrabaud,sjb}@cs.ucsd.edu
Presentation by: Yaron KoralIDC, Herzlia, ISRAEL
![Page 2: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/2.jpg)
AGENDA
• Motivation• Challenges• Algorithm• Experimental Results
2
![Page 3: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/3.jpg)
AGENDA
• Motivation• Challenges• Algorithm• Experimental Results
3
![Page 4: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/4.jpg)
Motivation
• Counting crowds of people• Counting herds of animals• Counting migrating cells
• Everything goes as long as the crowdis homogeneous!!
4
![Page 5: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/5.jpg)
AGENDA
• Motivation• Challenges• Algorithm• Experimental Results
5
![Page 6: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/6.jpg)
Challenges
• The problem of occlusion– Inter-object– Self occlusion
• Large number of independent motions– Dozens of erratically moving objects– Require more than two successive
frames
6
Surveillance camera viewing a crowd from a distant viewpoint, but zoomed in, such that the effects of perspective are minimized.
![Page 7: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/7.jpg)
AGENDA
• Motivation• Challenges• Algorithm• Experimental Results
7
![Page 8: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/8.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
8
![Page 9: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/9.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
9
![Page 10: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/10.jpg)
Harris Corner Detector – What are Good Features? C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988
• We should easily recognize a corner by looking through a small window
• Shifting a window in any direction should give a large change in intensity
![Page 11: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/11.jpg)
Harris Detector: Basic Idea
“flat” region:no change in all directions
“edge”:no change along
the edge direction
“corner”:significant
change in all directions
![Page 12: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/12.jpg)
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change of intensity for shift in [u,v] direction:
IntensityShifted intensity
Window function
orWindow function w(x,y)=
Gaussian1 in window, 0 outside
![Page 13: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/13.jpg)
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
yx vIuIyxIvyuxI ,,
v
uMvu
v
u
III
IIIyxwvu
v
uyxIyxIyxwvuE
yyx
yxx
yxyx
2
2
,
2
),(
,,,,
For small [u,v]:
We have:
![Page 14: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/14.jpg)
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
For small shifts [u,v] we have a bilinear approximation:
2
2,
( , ) x x y
x y x y y
I I IM w x y
I I I
where M is a 22 matrix computed from image derivatives:
![Page 15: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/15.jpg)
Harris Detector: MathematicsDenotes by ei the ith eigen-vactor of
M whose eigen-value is i:
Conclusions:
0 iiTi M ee
max1, ,maxarg e vuEvu
maxmax eE
![Page 16: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/16.jpg)
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of M
direction of the slowest change
direction of the fastest change
)max(-1/2
)min(-1/2
Ellipse E(u,v) = const
![Page 17: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/17.jpg)
Harris Detector: Mathematics
1
2
“Corner”1 and 2 are large,
1 ~ 2;
E increases in all directions
1 and 2 are small;
E is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of image points
using eigenvalues of M:
![Page 18: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/18.jpg)
Sum of Squared Differences – Tracking FeaturesTracking Features
• SSD is optimal in the sense of ML when1. Constant brightness assumption2. i.i.d. additive Gaussian noise
Ayx
tyxtvyuxvu,
2,,1,,,SSD RI
![Page 19: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/19.jpg)
Exhaustive Search
• Loop over all parameter space• No realistic in most cases
– Computationally expensive• E.g. to search 100X100 image in 1000X1000
image using only translation~1010 operations!
– Explodes with number of parameters– Precision limited to step size
![Page 20: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/20.jpg)
The Problem
Find (u,v) that minimizes the SSD
over region A.
Assume that (u,v) are constant over all A
Ayx
yxvyuxvuSSD,
2,,, RI
![Page 21: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/21.jpg)
Iterative Solution
• Lucas Kanade (1981)– Use Taylor expansion of I (the optical
flow equation)
– Find
Ayx
tyxvuvu
vuvuSSD,
2
,,min,min III
0 tyx vu III
![Page 22: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/22.jpg)
Feature Tracking with KLT(We’re back to crowd counting…)• KLT is a feature tracking algorithm• Driving Principle:
– Determine the motion parameter oflocal window W from image I to consecutive image J
– The center of the window defines the tracked feature
23
![Page 23: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/23.jpg)
Feature Tracking with KLT
• Given a window W– the affine motion parameters A and d
are chosen to minimize the dissimilarity
24
![Page 24: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/24.jpg)
Feature Tracking with KLT
• It is assumed that only d matters between 2 frames. Therefore a variation of SSD is used
25
• A window is accepted as a candidate feature if in the center of the window, both eigenvalues exceed a predefined threshold t
min(min(λλ11,, λ λ22) > t) > t
![Page 25: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/25.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
33
![Page 26: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/26.jpg)
Increased Efficiency #1• Associating only one window with
each feature– Giving a uniform weight function that
depends on 1/(window area |w|)– Determining quality by comparing:
• Computation of different Z matrices is accelerated by “integral image”[1]
34
|| w
z
[1] Viola & Jones 2004
![Page 27: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/27.jpg)
Increased Efficiency #2
• Run on sample training frames first– Determine parameters that lead to the
optimal windows sizes– Reduces to less than 5% of the possible
parameter set– All objects are from the same class
35
Surveillance camera viewing a crowd from a distant viewpoint, but zoomed in, such that the effects of perspective are minimized.
![Page 28: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/28.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
36
![Page 29: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/29.jpg)
Feature Re-Spawning
• Along time, KLT looses track:– Inter-object occlusion– Self occlusion– Exit from picture– Appearance change due to perspective
and articulation
• KLT recreates features all the time– Computationally intensive– Weak features are renewed
37
![Page 30: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/30.jpg)
Feature Re-Spawning
• Re-Spawn features only at specific locations in space and time
• Propagate them forward and backward in time– Find the biggest “holes”– Re-spawn features
in frame with theweighted average oftimes
38
![Page 31: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/31.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
39
![Page 32: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/32.jpg)
Trajectory Conditioning
• KLT tracker gives a set of trajectories with poor homogeneity– Don’t begin and end at the same times– Occlusions can result in trajectory
fragmentation– Feature can lose its strength resulting in
less precise tracks
• Solution: condition the data– Spatially and temporally
40
![Page 33: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/33.jpg)
Trajectory Conditioning
• Each trajectory is influenced by its spatial neighbors
• Apply a box to each raw trajectory• Follow all neighbor trajectories from
the time the trajectory started
41
![Page 34: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/34.jpg)
Algorithm Highlights
• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering
42
![Page 35: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/35.jpg)
Trajectory Clustering
• Determine number of object at time tt by clustering trajectories
• Since at time tt objects may be close, focus attention on a time interval (half-width of 200 frames)
• Build connectivity graph– At each time step, the present features form the
nodes of a connectivity graph G – Edges indicate possible membership to a common
object.
43
![Page 36: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/36.jpg)
Trajectory Clustering• Connectivity Graph
– Bounding Box: as small as possible, able to contain every possible instance of the object
– If two features do not stay in a certain box, they do not belong to the same object.
– The 3 parameters of this box are learned from training data.
44
Surveillance camera viewing a crowd from a distant viewpoint, but zoomed in, such that the effects of perspective are minimized.
Articulation factor
![Page 37: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/37.jpg)
Trajectory Clustering• Rigid parts merging
– Features share similar movement during whole life span, belong to a rigid part of an objectrigid part of an object, and consequently to a common objectcommon object
– RANSAC is applied to sets of trajectories • Within time window • Connected in graph G
45
![Page 38: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/38.jpg)
Trajectory Clustering
• Agglomerative Clustering– At each iteration, the two closest sets are
considered – If all features are linked to each other in the
connectivity graph, they are merged together.
– Otherwise, the next closest sets are considered
– Proceed until all possible pairs are analyzed
46
![Page 39: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/39.jpg)
AGENDA
• Motivation• Challenges• Algorithm• Experimental Results
47
![Page 40: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/40.jpg)
Experimental results
• Datasets– USC: elevated view of a crowd
consisting of zero to twelve persons– LIBRARY: elevated view of a crowd of
twenty to fifty persons– CELLS: red blood cell dataset consisting
of fifty to hundred blood cells
48
![Page 41: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/41.jpg)
Experimental results
49
![Page 42: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/42.jpg)
Experimental results
50
Estimated
Ground Truth
![Page 43: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/43.jpg)
Experimental results
51
![Page 44: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/44.jpg)
Conclusion
• A new way for segmenting motions generated by multiple objects in crowd
• Enhancements to KLT tracker• Conditioning and Clustering
techniques
52
![Page 45: Counting Crowded Moving Objects Vincent Rabaud and Serge Belongie Department of Computer Science and Engineering University of California, San Diego {vrabaud,sjb}@cs.ucsd.edu](https://reader036.vdocuments.us/reader036/viewer/2022081603/56649f285503460f94c4016e/html5/thumbnails/45.jpg)
Thank You!Thank You!
53