motion interchange patterns for action recognition in ... · motion interchange patterns (mip) -...
TRANSCRIPT
![Page 1: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/1.jpg)
Motion Interchange Patterns for Action Recognition
in Unconstrained Videos
Orit Kliper-Gross, Yaron Gurovich, Tal Hassner, Lior Wolf
Weizmann Institute of Science
The Open University of Israel
Tel Aviv University
http://www.openu.ac.il/home/hassner/projects/MIP/
![Page 2: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/2.jpg)
Controlled Sets
“In The Wild” Sets
What is an Unconstrained Video?
![Page 3: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/3.jpg)
Large variability
Challenges
Camera Motion
Action Ambiguity
Others
![Page 4: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/4.jpg)
• A new video descriptor:
• Dense Characterization of motion changes
• Captures shape of moving edges
• Built-in stabilization mechanism
Motion Interchange Patterns (MIP) - Highlights
State-of-the-art performance in the most recent and
challenging benchmarks (ASLAN, HMDB51, UCF50, …)
![Page 5: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/5.jpg)
Input Videos
Output STIP: On Space-Time Interest Points Laptev IJCV‘05 Local Descriptors: HOG, HOF and HNF Learning realistic human actions from movies Laptev et.al. CVPR’08
STIP + Local Desc
Action Recognition
Common Pipeline
![Page 6: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/6.jpg)
Output Global Descriptors Bag of Words (BOW)
Following: Learning realistic human actions from movies Laptev et.al. CVPR’08
STIP + Local Desc
Input Videos
Action Recognition
Common Pipeline
![Page 7: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/7.jpg)
Global Descriptors
Output SVM
classifier STIP +
Local Desc
Input Videos
Action Recognition
Common Pipeline
![Page 8: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/8.jpg)
STIP + Local Desc
LBP-based
Methods
New Video Descriptor Motion Interchange Patterns (MIP)
ECCV’12 New Video Descriptor
Global Descriptors
Input Video
SVM classifier
![Page 9: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/9.jpg)
• What:
- Low-level, dense, local representation
• How:
– Per-pixel encoding
– Uses binary/trinary digits 0 1 / -1 0 1
– The descriptor: frequencies of binary/trinary strings
• Very Successful: – Image textures [Ojala et al. ’96, Ojala et al. ’02, Heikkila et al. ‘06]
– Face recognition [Ahonen et al’ 06, Zhang et al. ’07, Wolf et al.’08]
– Facial expression [Zhao and Pietikainen ‘07]
– Action recognition [Yang et al. ‘07, kellokumpu et al. ‘08, Yeffet & Wolf ‘09]
Local Binary Patterns (LBP) -based representations
![Page 10: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/10.jpg)
t-1 t t+1
SSD( , ) SSD( , )
-1 1 0
ECCV’12
Motion Interchange Patterns (MIP)
x
α
i
j
![Page 11: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/11.jpg)
t-1 t t+1
SSD( , ) SSD( , )
-1 1 0
ECCV’12
Motion Interchange Patterns (MIP)
x
α = 0
α = 0
![Page 12: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/12.jpg)
Motion Interchange Patterns (MIP)
different α = different channels = diagonals
64-digits trinary code
![Page 13: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/13.jpg)
0
1
0
-1
0
1
-1
1
Per-pixel 64-digits trinary code
0-255 integer
0
1
0
0
0
1
0
1
0
0
0
-1
0
0
-1
0
0-255 integer
2 integers per-pixel Per Channel
Motion Interchange Patterns (MIP)
-1
1
0
-1
0
1
-1
1
0
1
0
-1
0
1
-1
1
0
1
0
-1
0
1
-1
1
0
1
0
-1
0
1
-1
1
Each α defines a channel 8 channels
![Page 14: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/14.jpg)
An example - one channel basic coding
Motion Interchange Patterns (MIP)
MIP captures: Motion, Motion Changes, and Shape
• Vote for next frame
• Vote for prev frame
• Static edges
![Page 15: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/15.jpg)
Suppression Mechanism
Switched Patch Suppression
Original Coding = 1
Switched Locations Coding = -1
2 ways to look at this: - No motion. - Contradicted motion voting. i.e. Original coding voted down Switched patches voted up
Suppress the code
Suppress background structure and noise
![Page 16: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/16.jpg)
MIP Suppression Mechanism An Example
Without Suppression
With Suppression
Original
![Page 17: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/17.jpg)
Effect of Camera Motion Motivating Example
Original Movie MIP Coding
![Page 18: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/18.jpg)
3. Use MIP silent pixels for global affine
MIP Stabilization Mechanism
Flow:
t-1 t t+1
Get Frames MIP Code
t-1 t t+1
Direct-MIP Wrapping
MIP Re-Code 1. Global translation on all pixels
2. Code MIP.
![Page 19: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/19.jpg)
MIP Stabilization Mechanism An Example
Without Stabilization
With Stabilization
Original
![Page 20: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/20.jpg)
Concatenate: 512-bins histogram
per Patch
Two 256-bins histograms
per Patch
Motion Interchange Patterns (MIP) Vectorization
16x16 patches
.
.
.
Vectorization: 512-dimensions code words
![Page 21: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/21.jpg)
Motion Interchange Patterns – pipeline
MIP Encoding Channel
code words
BOW Histogram
Representation
8 BOW concatenation
8 channels
Input movies
t-1 t t+1
x8
SVM Classifier
![Page 22: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/22.jpg)
47.9
20.2
65.3
15
25
35
45
55
65
75
85
95
Baseline
Performances
MIP on Most Challenging AR Datasets
: HOG/HOF
50 classes 51 classes Same/Not-Same
SotA on the most challenging Action Recognition DBs
![Page 23: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/23.jpg)
Motion Interchange Patterns (MIP)
Examples
![Page 24: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/24.jpg)
Results on ASLAN Same classified as Same (TP)
Jumping Jacks Pull Ups – 1 hand Moon Walk
![Page 25: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/25.jpg)
Results on ASLAN Same classified as Not-Same (FN)
Kissing Squat Talking on phone
![Page 26: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/26.jpg)
Results on ASLAN
Not-Same classified as Not-Same (TN)
![Page 27: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/27.jpg)
Results on ASLAN
Not-Same classified as Same (FP)
![Page 28: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/28.jpg)
Results on HMDB51 Brush hair success :
False positive, miss of ‘chew’:
![Page 29: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/29.jpg)
Results on HMDB51 Cartwheel success :
False positive, miss of ‘flic flac’:
![Page 30: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/30.jpg)
Results on UCF50 Basketball success:
False positive, miss of ‘Volleyball Spiking’:
![Page 31: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/31.jpg)
Results on UCF50 HighJump success :
False positive, miss of ‘Pole-vault’:
![Page 32: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/32.jpg)
Results on UCF50 Nunchucks success :
False positive, miss of ‘Pizza Tossing’:
![Page 33: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/33.jpg)
Summary
A New Video Descriptor:
• Efficient Low-level, dense, local representation
• Complete characterization of motion & motion changes
• Captures shape of moving edges
• Built-in suppression & stabilization mechanisms
![Page 34: Motion Interchange Patterns for Action Recognition in ... · Motion Interchange Patterns (MIP) - Highlights State-of-the-art performance in the most recent and challenging benchmarks](https://reader035.vdocuments.us/reader035/viewer/2022071214/6041ff24ddb6e056b25e2ad9/html5/thumbnails/34.jpg)
Thank You !
http://www.openu.ac.il/home/hassner/projects/MIP/