globally optimal solution to multi-object tracking with...
TRANSCRIPT
Globally Optimal Solution to Multi-Object Tracking with
Merged Measurements
João F. João F. João F. João F. HenriquesHenriquesHenriquesHenriques RuiRuiRuiRui CaseiroCaseiroCaseiroCaseiro Jorge BatistaJorge BatistaJorge BatistaJorge Batista
Institute of Systems and RoboticsUniversity of Coimbra
International Conference on Computer Vision, 2011
Motivation
• Tracking multiple people• Single, Single, Single, Single, uncalibrateduncalibrateduncalibrateduncalibrated camera• Realistic video surveillance scenes
Problem far from solved!Problem far from solved!Problem far from solved!Problem far from solved!
Proposed method:• Handles merged measurements• Global optima in polynomial time
•Tracking as global optimization1. Detect in all frames
2. Optimize all tracks simultaneously (minimize a cost function)
Solid theoretical treatment of groups
⇒
Contributions:Contributions:Contributions:Contributions:
Previous work
Multiple-matching
Target counts(group identification)
Recovering identities
⇒ ⇒
Previous work
Multiple-matching
Target counts(group identification)
Recovering identities
⇒ ⇒
C. Huang et al.ECCV’08
Global solutionin
No merges/splitsNo merges/splitsNo merges/splitsNo merges/splitsNo groups Obtained trivially
A. Perera et al.CVPR’06
Local solutionin
(Global in )Limited to groups of 2
Limited togroups of 2
J. Sullivan et al.ECCV/CVPR’06
Local solutionin
No convergence guarantees
Global solutionin
J. F. J. F. J. F. J. F. HenriquesHenriquesHenriquesHenriqueset al., ICCV’11et al., ICCV’11et al., ICCV’11et al., ICCV’11
GlobalGlobalGlobalGlobal solutionsolutionsolutionsolutionin
GlobalGlobalGlobalGlobal solutionsolutionsolutionsolutionin
GlobalGlobalGlobalGlobal solutionsolutionsolutionsolutionin
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.1. Multiple-matching
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒2.1. Multiple-
matching
Global optimization tracking review
1. Detect and track with a simple method
• Stop trackers when confidence is low
• Output: trackletstrackletstrackletstracklets(short track segments)
2. Each tracklet is a nodenodenodenode in a graph
Global optimization tracking review
3. Connect each node to another if it occurs later in time
4. Calculate a costcostcostcost for each of these arcsarcsarcsarcs
Low cost ⇒ same target
(hopefully!)
Use distance, appearance similarity, size difference…
time
c1,2
c1,3node 1 node 2
node 3
• These are just hypothesesjust hypothesesjust hypothesesjust hypotheses.
• Optimization will choose the best set of arcs.
Global optimization tracking review
5. Termination nodes –hypothesis of ending a track
6. Initialization nodes –hypothesis of starting a track
7. Costs based on entry/exit locations model
c1init c1
term
Global optimization tracking review
• A typical graph:
• Possibly occluded targets –long-range arcs
• Optimization is free to choose between many hypotheses
(Some init./term. nodes omitted for clarity)
init.
init.
term.
term.
Note:
• Long occlusions ⇒ many arcs⇒ mistakes more likely
(merged measurements(merged measurements(merged measurements(merged measurementssolves this)solves this)solves this)solves this)
Global optimization tracking review
Optimal matching
• Find set of arcs X with a minimum total cost
• Subject to nononono----overlap restrictionoverlap restrictionoverlap restrictionoverlap restriction: only 1 ingoing and 1 outgoing arc in each node
∑=ji
ijijX
xcX,
* minarg
∀=∀=
∑∑
jx
ixts
i ij
j ij
,1
,1..
Hungarian AlgorithmHungarian AlgorithmHungarian AlgorithmHungarian Algorithm Solution in ( : #nodes)⇒
init.
init.
term.
term.
Merged measurements
• A single node single node single node single node may actually represent several targetsseveral targetsseveral targetsseveral targets.
• Merges and splits violate violate violate violate theno-overlap restriction.
⇓
Can’t use Hungarian Algorithmdirectly!
Somehow…Somehow…Somehow…Somehow…Change graph to support:
All the previous arcs+
Merges/splits hypotheses
(Cannot be solved with:• Min. cost flow• QP• QBP)
mergemergemergemerge splitsplitsplitsplit
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.1. Multiple-matching
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒
Multiple-matching
node 1
node 2
node 3Don’t know a-priori if node 3 is:
• Same target as node 1• Same target as node 2• Both targets (merged)
Can calculate a cost for each hypothesis:
c13, c23, cmerge
????
Multiple-matching
node 1
node 2
node 3
cccc13131313
Multiple-matching
cccc23232323
node 1
node 2
node 3
Multiple-matching
cccc23232323
node 1
node 2
node 3
cccc13131313
Total cost
= cmerge
Problems:
• Total cost is not linearnot linearnot linearnot linear
• ViolatesViolatesViolatesViolates the no-overlap restriction
≠ c13 + c23
Multiple-matching
node 1
node 2
node 3
Conditions:
• Total cost of merge must be
⇒
Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal link and one termination.link and one termination.link and one termination.link and one termination.
Key idea:
cccc23232323
cccc13131313
cccctermtermtermterm
????
Multiple-matching
node 1
node 2
node 3
Conditions:
• Total cost of merge must be
⇒
• If is active, must be active
⇒ Disable all other incoming arcs tonode 3
Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal link and one termination.link and one termination.link and one termination.link and one termination.
Key idea:
cccc23232323
cccc13131313
cccctermtermtermterm
????
Multiple-matching
node 1
node 2
node 3
cccc13131313
Multiple-matching
node 1
node 2
node 3
cccc23232323
Multiple-matching
node 1
node 2
node 3
cccc13131313
cccctermtermtermterm
• Total cost= c13 + cterm = cmerge
as required.
• The proposed structure restricts the solutions to onlyonlyonlyonly these 3.
Multiple-matching
• For splits, reverse arc directions.
• The Hungarian Algorithmis free to choosefree to choosefree to choosefree to choose
• merges,• splits,• regular arcs
that better explain data.
• Same computational complexity Same computational complexity Same computational complexity Same computational complexity as simple matching.
Output
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.1. Multiple-matching
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒
Groups
• Targets travelling alone can be tracked unambiguously.
But:
• Individual identities are lost in groups.
TargetTargetTargetTarget #1#1#1#1
TargetTargetTargetTarget #2#2#2#2
TargetTargetTargetTarget #3#3#3#3
TargetTargetTargetTarget ????????
TargetTargetTargetTarget ????????
TargetTargetTargetTarget ????????
1. Identify nodes that correspond to groups.
2. Match individual targets across the groups.
Strategy:
Target counts as flow circulation
• To robustly identify groups,count targets count targets count targets count targets in each node.
• Solve min-cost circulation:
Solved in polynomial time with the Edmonds-Karp algorithm.
Minimize number of targets
Flow conservation
Boundary conditions(at least 1 target per node)
Target counts as flow circulation
• Costs inversely proportional tobox size(larger boxes ⇒ larger groups)
• Counts obtained solely fromsolely fromsolely fromsolely fromthe graph topologythe graph topologythe graph topologythe graph topology.
• Equivalent to locating occluded targets.
Mostly as a tie-breaker
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.1. Multiple-matching
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒
Recovering identities
Groups subgraph
={arcs and nodes with flow > 1}
Entering a group
={arcs pointing to }
Exiting a group
={arcs pointing from }
To track across groups, match targets entering to targets exiting.
Simple matching problem⇒
Recovering identities
• Match across groups:
1. Connect a node in entering to a node in exiting if it’s reachable through .
2. Calculate costs as usual.
3. Optimal matching with Hungarian Algorithm.
Recovering identities
• Unlike other methods, wedon’t link across arbitrary gapsdon’t link across arbitrary gapsdon’t link across arbitrary gapsdon’t link across arbitrary gaps.
• Target’s trajectory is restrictedrestrictedrestrictedrestricted to group’s trajectory.
• Only admits physically plausiblephysically plausiblephysically plausiblephysically plausibletracks for occlusions.
Outline
1. Global optimization tracking review
2. Proposed method:
3. Conclusion and results
2.2. Target counts(group identification)
2.3. Recovering identities
⇒ ⇒2.1. Multiple-
matching
Results
Detection method:foreground segmentation
• Simple to implement
• Frequent merges and splitsFrequent merges and splitsFrequent merges and splitsFrequent merges and splits⇒ Stress-test the framework
PETS 2006 PETS 2006 PETS 2006 PETS 2006 (single view)Good results, challenging videos.No other MOTA scores to compare.
PETS 2009 PETS 2009 PETS 2009 PETS 2009 (single view)Outperforms all 12 all 12 all 12 all 12 state-of-the-art systems at PETS 2010.
Results
Conclusion
Formalized group formationFormalized group formationFormalized group formationFormalized group formationas a global optimization problem.
PolynomialPolynomialPolynomialPolynomial----time algorithms time algorithms time algorithms time algorithms for:
• Multi-match problem.• Counting problem.• Identity match across groups.
Improves state-of-the-artwith simple detection and low-level stages.
Questions?