human movement summarization and depiction from videos hao jiang boston college yijuan lu texas...
TRANSCRIPT
![Page 1: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/1.jpg)
Human Movement Summarization and Depiction from Videos
Hao Jiang
Boston College
Yijuan Lu
Texas State University
![Page 2: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/2.jpg)
How do we describe human movementin videos?
2
![Page 3: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/3.jpg)
Methods to illustration motion
Muybridge’s photos to capture human movement
![Page 4: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/4.jpg)
Methods to illustration motion
4
![Page 5: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/5.jpg)
Manual drawing or depiction from mocap data
S. Bouvier-Zappa, V. Ostromoukhov and P. Poulin, “Motion cues for illustration of skeletal motion capture data”, Symposium on Non-Photorealistic Animation and Rendering 2007.
![Page 6: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/6.jpg)
Our method: depiction from 2D videos
Our method automatically extracts human motion and generates depictions.
video
![Page 7: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/7.jpg)
:wChallenges
Movement segmentation
Finding human body parts and linking them though time
Finding detailed trajectories of “particles” on the human subject
Generating depictions of the movement using arrows, colors, trajectories.
![Page 8: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/8.jpg)
Movement segmentation
8
Trajectories of particles on human subject can beused to quantify the movement at one instant.
![Page 9: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/9.jpg)
Movement segmentation
9
Comparing the trajectory features between successive frames:v u
![Page 10: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/10.jpg)
Body part detection and association
10
• Body part detections are in each single frame; we need to link them through time (tracking)• There are often errors that need to be corrected (error correction)
![Page 11: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/11.jpg)
Body part tracking
11
We formulate the tracking problem as a data association problem
![Page 12: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/12.jpg)
Body part tracking
12
We formulate the tracking problem as a data association problem
![Page 13: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/13.jpg)
Correct tracking errors
13
We use the following network to correct part tracking errors:
Predictions
![Page 14: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/14.jpg)
Correct tracking errors
14
We use the following network to correct part tracking errors:
Predictions
Error correction becomes a shortest path problem.
![Page 15: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/15.jpg)
Finding long term trajectories
15
motionsmoothness
structure
unary
![Page 16: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/16.jpg)
Finding long term trajectories
16
Gradient descent
![Page 17: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/17.jpg)
Generate depiction
17
Trick: polynomialcurve fitting togenerate pleasingResults.
![Page 18: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/18.jpg)
Results
18
![Page 19: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/19.jpg)
Results
19
![Page 20: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/20.jpg)
Results
20
![Page 21: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/21.jpg)
Results
21
![Page 22: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/22.jpg)
More Results
22
![Page 23: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/23.jpg)
Summary
It is indeed possible to automatically generate movement depictions from videos.
Not only the depiction can be treated as an art illustration, it also can serve the purpose of meaningful movement description.
Future work: finding the “optimal” movement depiction.
23
![Page 24: Human Movement Summarization and Depiction from Videos Hao Jiang Boston College Yijuan Lu Texas State University](https://reader035.vdocuments.us/reader035/viewer/2022062807/5697bfe91a28abf838cb6c61/html5/thumbnails/24.jpg)
Thank You!
24