![Page 1: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/1.jpg)
Silhouette Lookup for Automatic Pose Tracking
NICK HOWE
![Page 2: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/2.jpg)
Goal: 3D Pose Tracking
Full 3D “motion capture” from 2D video Single camera Unmarked video
Difficulties: 3D ambiguity Self-occlusion Foreshortening Appearance changes Shadowing
↑(Uses hand-entered data)
![Page 3: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/3.jpg)
The “Old” Way:
Incremental Tracking
Previous frame
Compare withwith ImageRefine 2D PoseRefine 2D Pose
2D Pose2D Pose+ Appearance+ Appearance
NumericalNumericalOptimizationOptimization
NextNext frame
![Page 4: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/4.jpg)
Creeping Error
Incremental Errors accumulate and grow.
May be mitigated by: Better motion models (more guidance) Better appearance models (3D) Better tracking (multiple hypotheses)[Sidenbladh, et. al.; Sminchisescu, et. al.]
Intrinsic problems still remain.(initialization, error recovery)
![Page 5: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/5.jpg)
Direct Pose Estimation
Consider human abilities: Estimate pose from still photo Estimate pose from stick figure Estimate pose from silhouette
[Brand ’99; Rosales et. al. ’01)
![Page 6: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/6.jpg)
Recognition/Retrieval
Hypothesis: Humans can recognize pose by recalling similar examples. Pose Recognition Retrieval
![Page 7: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/7.jpg)
Recognition/Retrieval
Hypothesis: Humans can recognize pose by recalling similar examples. Pose Recognition Retrieval
New Approach: 1. Store many silhouettes with known poses
2. Given video, extract silhouettes3. Retrieve best candidate matches4. Look for plausible series of poses over time
![Page 8: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/8.jpg)
Some Related Work
Estimating Human Body Configuration Using Shape Context MatchingMori & Malik, ECCV 2002
3D Tracking = Classification+InterpolationTomasi, Petrov, & Sastry, ICCV 2003
Temporal Integration of Multiple Silhouette-based Body-part HypothesesKwatra, Bobick, & Johnson, CVPR 2001
3D Human Pose from Silhouettes by Relevance Vector RegressionAgarwal & Triggs, CVPR 2004
![Page 9: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/9.jpg)
Silhouette Comparison
Turning angle(Captures morphology)
Chamfer distance(Captures overlap)
Combine using Belkin technique(score = sum of individual ranks)
![Page 10: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/10.jpg)
Sample Retrievals
(Hits from a small library of 1600 poses)
![Page 11: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/11.jpg)
Coordination Between Frames
Need to pick from top matches at each frame. Want good image match at all frames Want small change between frames
Markov chain minimization!
Best local choices minimize global error
etc.
frame i-1 frame i frame i+1
![Page 12: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/12.jpg)
Too Much Coffee?
Initial solution shows “twitches”
![Page 13: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/13.jpg)
Smoothing it Out
Jitters in motion parameters smoothed via polynomial splines
![Page 14: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/14.jpg)
Making it Match
Problem: poor overlap between observed silhouette & smoothed solution Work with 11-frame splines
Optimize spline parameters to reduce chamfer distance
Result: better match to observations, still smooth
![Page 15: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/15.jpg)
Walking Sequence Result
![Page 16: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/16.jpg)
Re-rendering
Same scene, different viewpoint.
![Page 17: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/17.jpg)
Another Example
Tracked using library of ballet poses
![Page 18: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/18.jpg)
Incremental Tracking
Markov chain is best for offline use But: Convergence after ~10 frames
Incremental tracking with latency
![Page 19: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/19.jpg)
Key Points
Silhouette lookup provides set of potential poses for each frame
Markov chain selects best temporal pose sequence (HMM)
Smoothing & optimization based upon temporal splines
Result: simple tracker, tolerates errors
![Page 20: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/20.jpg)
Thank you! Questions?
![Page 21: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/21.jpg)
Continuing Challenges
Mistakes in rotational direction No data for parts not on silhouette
Incorporate optical flow Some unrealistic motions generated
Incorporate motion model Correct pose not always retrieved
Improve library coverage, retrieval
![Page 22: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/22.jpg)
Future Research
People carrying objects Multiple overlapping people (sports) Time considerations
Optimization slow Chaining currently slow Holy Grail: Real-time tracking
![Page 23: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/23.jpg)
2. Identify best (least expensive)
result
Markov Chain Minimization
Frame 1 Frame 2 Frame n
...
1. Compute least expense to reach each state from previous frame (cost = estimate of plausibility)
State 2A
State 2C
State 2B
State 1A
State 1C
State 1B
State nA
State nC
State nB
3. Backtrack, picking out path that gave best result.
![Page 24: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/24.jpg)
Silhouette Extraction
Many candidate approaches. Moving & fixed camera
This work: Static camera Graph-based segmentation
![Page 25: Silhouette Lookup for Automatic Pose Tracking N ICK H OWE](https://reader034.vdocuments.us/reader034/viewer/2022050908/56649cf85503460f949c9157/html5/thumbnails/25.jpg)
Making it Match
Solution doesn’t match exactly yet.