lecture 4: video applications - m15/allgemeines€¦ · 7/21/2016 topological time series analysis...
TRANSCRIPT
![Page 1: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/1.jpg)
Lecture 4: Video Applications
Topological Time Series Analysis - Theory And Practice
Jose Perea, Michigan State University. Chris Tralie, Duke University
7/21/2016
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 2: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/2.jpg)
Table of Contents
I Sliding window video definition / examples
B Sliding window video formalism
B Natural video dynamics/geometry
B Memory efficiency / preprocessing
B Interactive Example
B KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 3: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/3.jpg)
Sliding Window Videos
Y[n]=
X[n]
.
.
.
Time
M
X[n+1]
X[n+2]
X[n+M-1]
X[n]
X[n+M-1]
http://www.ctralie.com/Research/SlidingWindowVideo-SOCG2016/
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 4: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/4.jpg)
Examples
Jumping jacks, heartbeat animation, my neck
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 5: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/5.jpg)
Table of Contents
B Sliding window video definition / examples
I Sliding window video formalism
B Natural video dynamics/geometry
B Memory efficiency / preprocessing
B Interactive Example
B KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 6: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/6.jpg)
Table of Contents
B Sliding window video definition / examples
B Sliding window video formalism
I Natural video dynamics/geometry
B Memory efficiency / preprocessing
B Interactive Example
B KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 7: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/7.jpg)
Pure Cosine Composition Model
On the board
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 8: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/8.jpg)
Oscillating Line Segment
0 20 40 60 80 100120140Pixel Index i
0
50
100
150
200
Tim
e t
Oscillating Line Segment Raw Embedding 2D PCA T-length Embedding 2D PCA
B Good Toy Example for Natural VideoB 2 questions
1. Why is it path-like without embedding?2. Why is it curved?
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 9: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/9.jpg)
Oscillating Line Segment
0 20 40 60 80 100120140Pixel Index i
0
50
100
150
200
Tim
e t
Oscillating Line Segment Raw Embedding 2D PCA T-length Embedding 2D PCA
B Good Toy Example for Natural VideoB 2 questions
1. Why is it path-like without embedding?
2. Why is it curved?
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 10: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/10.jpg)
Oscillating Line Segment
0 20 40 60 80 100120140Pixel Index i
0
50
100
150
200
Tim
e t
Oscillating Line Segment Raw Embedding 2D PCA T-length Embedding 2D PCA
B Good Toy Example for Natural VideoB 2 questions
1. Why is it path-like without embedding?2. Why is it curved?
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 11: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/11.jpg)
Jumping Jacks Example
Show PCA Videos
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 12: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/12.jpg)
Distance Matrix Interpretation
Oscillating Bar
Raw Embedding 2D PCA T-length Embedding 2D PCA
0 50 100 150Frame Number
0
50
100
150
Fram
e N
um
ber
0 20 40 60 80 100 120 140Frame Number
0
20
40
60
80
100
120
140
Fram
e N
um
ber
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 13: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/13.jpg)
Distance Matrix Interpretation
Convolving along diagonals (show video)
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 14: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/14.jpg)
Distance Matrix Interpretation
Jumping Jacks
0 2 4 6 8RHSV Number
0
20
40
60
80
100
Fram
e Nu
mbe
rRight Hand Singular Vectors 2D PCA
0 20 40 60 80 100Frame Number
0
20
40
60
80
100
Fram
e Nu
mbe
r
Self-Similarity Matrix
0 2 4 6 8RHSV Number
0
10
20
30
40
50
60
70
Fram
e Nu
mbe
r
Right Hand Singular Vectors 2D PCA
0 10 20 30 40 50 60 70Frame Number
0
10
20
30
40
50
60
70
Fram
e Nu
mbe
r
Self-Similarity Matrix
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 15: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/15.jpg)
Oscillating Line Segment
Why is it curved?B Eulerian pixel view
0 20 40 60 80 100 120 140 160Pixel Index i
0
50
100
150
200
250
Tim
e t
Oscillating Line Segment
0 50 100 150 2002.0
1.5
1.0
0.5
0.0
0.5
1.0
1.5
2.0Pixel Index 40
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 16: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/16.jpg)
Jumping Jacks Video Eulerian View
Show video
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 17: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/17.jpg)
Principal Component Videos
Show videos for heartbeat and jumping jacks
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 18: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/18.jpg)
Table of Contents
B Sliding window video definition / examples
B Sliding window video formalism
B Natural video dynamics/geometry
I Memory efficiency / preprocessing
B Interactive Example
B KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 19: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/19.jpg)
Naive Embedding
Naive video stacking blows up in memory!
B E.g. 400x400 video of length 300 frames, delay of 30frames
Solution?
B Do SVD
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 20: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/20.jpg)
Naive Embedding
Naive video stacking blows up in memory!
B E.g. 400x400 video of length 300 frames, delay of 30frames
Solution?
B Do SVD
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 21: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/21.jpg)
Naive Embedding
Naive video stacking blows up in memory!
B E.g. 400x400 video of length 300 frames, delay of 30frames
Solution?
B Do SVD
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 22: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/22.jpg)
Naive Embedding
Naive video stacking blows up in memory!
B E.g. 400x400 video of length 300 frames, delay of 30frames
Solution?
B Do SVD
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 23: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/23.jpg)
What About Drift?
e.g. X (t) = (cos(t), sin(t), t)Show video
B Take smoothed time derivative of each pixel
g(t) = te−t2/(2σ2)
B Can be interpreted as a bandpass filter
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 24: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/24.jpg)
What About Drift?
e.g. X (t) = (cos(t), sin(t), t)Show video
B Take smoothed time derivative of each pixel
g(t) = te−t2/(2σ2)
B Can be interpreted as a bandpass filter
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 25: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/25.jpg)
What About Drift?
e.g. X (t) = (cos(t), sin(t), t)Show video
B Take smoothed time derivative of each pixel
g(t) = te−t2/(2σ2)
B Can be interpreted as a bandpass filter
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 26: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/26.jpg)
1D Drift Example
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 27: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/27.jpg)
Table of Contents
B Sliding window video definition / examples
B Sliding window video formalism
B Natural video dynamics/geometry
B Memory efficiency / preprocessing
I Interactive Example
B KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 28: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/28.jpg)
Table of Contents
B Sliding window video definition / examples
B Sliding window video formalism
B Natural video dynamics/geometry
B Memory efficiency / preprocessing
B Interactive Example
I KTH Dataset
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 29: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/29.jpg)
KTH Dataset
http://www.nada.kth.se/cvap/actions/
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 30: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/30.jpg)
KTH Dataset
B We are not doing activity recognition!!Castrodad, Alexey, and Guillermo Sapiro. “Sparse modeling of human actionsfrom motion imagery.” International journal of computer vision 100.1 (2012):1-15.
B Instead, we will be ranking periodicity
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 31: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/31.jpg)
KTH Dataset: Experiment 1
Window length vs persistence
B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 32: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/32.jpg)
KTH Dataset: Experiment 1
Window length vs persistence
B Keep dimension fixed. For window length W , τ = d/W
B Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 33: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/33.jpg)
KTH Dataset: Experiment 1
Window length vs persistence
B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/N
B Take blocks of 160 frames
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 34: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/34.jpg)
KTH Dataset: Experiment 1
Window length vs persistence
B Keep dimension fixed. For window length W , τ = d/WB Keep number of points fixed. dT = (N − dτ)/NB Take blocks of 160 frames
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 35: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/35.jpg)
KTH Dataset: Experiment 2 (Interactive)
Rank 4 videos as a class
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications
![Page 36: Lecture 4: Video Applications - M15/Allgemeines€¦ · 7/21/2016 Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications. ... Theory And Practice Lecture](https://reader033.vdocuments.us/reader033/viewer/2022050200/5f53f585869cb93b844cc9ba/html5/thumbnails/36.jpg)
KTH Dataset: Experiment 3 (Interactive)
Rank all videos for a particular person
Topological Time Series Analysis - Theory And Practice Lecture 4: Video Applications