motion coherent tracking with multi-label mrf optimization
DESCRIPTION
Motion Coherent Tracking with Multi-Label MRF Optimization. David Tsai Matthew Flagg James M. Rehg Computational Perception Lab School of Interactive Computing Georgia Institute of Technology. Tracking Animals for Behavior Analysis. Lekking display. Stotting behavior. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/1.jpg)
Motion Coherent Tracking with
Multi-Label MRF OptimizationDavid Tsai Matthew Flagg James M. Rehg
Computational Perception LabSchool of Interactive ComputingGeorgia Institute of Technology
![Page 2: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/2.jpg)
Tracking Animals for Behavior Analysis
• Wide range of animal morphologies and biological questions
• “Tracking” via segmentation of the animal• http://www.kinetrack.org 2
Lekking display Stotting behavior
![Page 3: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/3.jpg)
Goal
• Automatic video object cut-out• Offline analysis with minimum user input• Behavior analysis via post-processing
Input video Output
![Page 4: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/4.jpg)
Previous Work
Bibby et al. ECCV’08 Bai et al. SIGGRAPH’09
* Real-time, automatic* Not focused on segmentation
* High-quality segmentation* Significant manual effort is required
![Page 5: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/5.jpg)
5
Approach
• Segmentation in video volume MRF– Formulate as joint label assignment– No shape priors, adaptation, etc.
• Joint label space encodes per-pixel segmentation and motion– Enforce motion coherence
• SegTrack database with video segmentation ground truth– Comparison to existing methods
![Page 6: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/6.jpg)
p
MRF Multi-Label Space
t+1
t
Temporalneighbors
Spatialneighbors
pl { , } X {-2, -1, 0, +1, +2}
)( plA )( plD
pixelper labels )12(2 :In 2MXYT
![Page 7: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/7.jpg)
p
MRF Multi-Label Space
t+1
t
Temporalneighbors
Spatialneighbors
pl { , } X {-2, -1, 0, +1, +2}
)( plA )( plD
pixelper labels )12(2 :In 2MXYT
pl { , +1}
![Page 8: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/8.jpg)
8
• Find joint label assignment minimizing:
Multi-Label MRF Assignment
Gp Gp Nq
qppqppp
llVlVLE ),()()(
Data Smoothness
![Page 9: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/9.jpg)
9
Data Term
Gp Gp Nq
qppqppp
llVlVLE ),()()(
t+1
t
foreground background
RGB likelihood
Optical flow
![Page 10: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/10.jpg)
10
Smoothness Term
Gp Gp Nq
qppqppp
llVlVLE ),()()(
p
t+1
t
AttributeCoherence
)]()([1 qpA lAlAw
![Page 11: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/11.jpg)
11
Smoothness Term
Gp Gp Nq
qppqppp
llVlVLE ),()()(
AttributeCoherence
p
t+1
t
)]()([1 qpA lAlAw
LowestCostAssignment
![Page 12: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/12.jpg)
12
Smoothness Term
Gp Gp Nq
qppqppp
llVlVLE ),()()(
p
t+1
t
MotionCoherence
)()( qpD lDlDw
![Page 13: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/13.jpg)
13
Smoothness Term
Gp Gp Nq
qppqppp
llVlVLE ),()()(
p
t+1
t
MotionCoherence
)()( qpD lDlDw
LowestCostAssignment
![Page 14: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/14.jpg)
Optimization• Challenge: Combinatorics of label space
– 162 labels/pixel => 19.4M labels/frame• Solution:
– Process video in subvolumes, constrain the solution across the boundaries
– Spatial mult-grid (factor of 16 savings)– Use Fast-PD by Komodakis et.al.
• We have found Fast-PD to converge significantly faster than graphcut– 20 seconds/frame for 300 x 400 image
![Page 15: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/15.jpg)
SegTrack Database• A new database for video segmentation with
ground truth• Three attributes that impact performance:
– Color overlap between target and background– Interframe motion– Change in target shape
![Page 16: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/16.jpg)
Experiment• Quantitative Comparison with Chockalingam et.al ICCV’09
Parach
ute Girl
Monkeyd
og
Penguin
Birdfal
l
Cheetah0
1000
2000
3000
4000
5000
6000
7000
Our scoreChockalingam et al ICCV'09
![Page 17: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/17.jpg)
18
Video Results
![Page 18: Motion Coherent Tracking with Multi-Label MRF Optimization](https://reader035.vdocuments.us/reader035/viewer/2022062315/568166a3550346895dda8fa6/html5/thumbnails/18.jpg)
19
Conclusion
• New approach to video object cut-out based on temporally-coherent MRF
• New SegTrack dataset to facilitate quantitative evaluations of segmentation performance
• Promising experimental results, including comparison to two recent methods