multi-view super vector for action recognition
DESCRIPTION
Multi-View Super Vector for Action Recognition. Shenzhen Institutes of Advanced Technology, CAS Chinese University of Hong Kong. Limin Wang. Zhuowei Cai. Xiaojiang Peng. Yu Qiao. Content. Motivation M-PCCA model & MVSV representation Experimental Results. Content. Motivation - PowerPoint PPT PresentationTRANSCRIPT
Multi-View Super Vector for Action Recognition
Shenzhen Institutes of Advanced Technology, CASChinese University of Hong Kong
Zhuowei Cai Yu QiaoXiaojiang PengLimin Wang
Content
• Motivation
• M-PCCA model & MVSV representation
• Experimental Results
Content
• Motivation
• M-PCCA model & MVSV representation
• Experimental Results
Actions in Video Clips can be captured by ...
HOG HOF MBHx/MBHy
*Video from chalearn looking at people chanllenge
static feature dynamic features
Feature Fusion - Concatenation
HOG HOF
Concatenation before GMM : HOG + HOF
Defect : features presumed to be strongly correlated
Feature Fusion - Kernel Average
HOG HOF
Concatenation after GMM : CodeHOG + CodeHOF
Defect : features presumed to be mutually independent
CodeHOG CodeHOF
Decomposition
HOG HOF
HOG-Specific HOF-SpecificHOG/HOF-Shared
HOG-Specific HOF-SpecificHOG/HOF-Shared
Merit : features are decomposed into relatively independent components
HOG-Specific HOF-SpecificHOG/HOF-Shared
M-PCCAMixture of Probabilistic Canonical Correlation Analyzers
Content
• Motivation
• M-PCCA model & MVSV representation
• Experimental Results
Mixture of Probabilistic Models
X = Wx Z + Zx
Y = Wy Z + ZyZ ~ N(0, I), Zx ~ N(μx, Φx), Zy ~ N(μy, Φy).
Latent Variable Models
V = W Z + Zv
Probabilistic Canonical Correlation Analyzer. *B. Francis, M. Jordan; K. Arto, S. Kaski
Z ~ N(0, I), Zv ~ N(μ, Φ)
Probabilistic Principal Component Analysis: Φ = σI. *M. Tipping, C. Bishop
Probabilistic Factor Analysis: Φ is diagonal.
Mixture Version: M-PPCA *M. Tipping, C. Bishop, M-FA *G. Zoubin, G. Hinton
M-PCCA
EM Learning Algorithm
M-PCCA
M-PCCA
X = Wx Z + Zx
Y = Wy Z + Zy
M-PCCA
Z2
Z1
Z3
= Shared Part11 , 1{ } {[ , ] ( )}K T T K
k k x y k i k i k ki
Z z W W v
M-PCCA
M-PCCA
gx
gy
= Private Part
1
( ) ( ){ , }Kkk k
x x
E L E L
1
( ) ( ){ , }Kkk k
y y
E L E L
M-PCCA
gx gy Private Part
Shared Part +
= Multi-View Super Vector
Content
• Motivation
• M-PCCA model & MVSV representation
• Experimental Results
Performance w.r.t number of Components Performance w.r.t Latent Dimension
MVSV with SVM classifier on HMDB51with various configurations
Results#components = 256, dimension = 45
HMDB51 FV VLAD MVSV
Fusion d-level k-level d-level k-level k-level
HOG+MBH 50.9% 50.4% 47.0% 48.5% 52.1%
HOG+HOF 47.0% 48.3% 44.4% 47.7% 48.9%
MBHx+MBHy 49.2% 49.1% 45.2% 47.0% 51.1%
Combine 52.4% 53.2% 51.5% 52.6% 55.9%
UCF101 FV VLAD MVSV
Fusion d-level k-level d-level k-level k-level
HOG+HOF 76.1% 77.7% 75.7% 77.5% 78.9%
MBHx+MBHy 78.9% 78.7% 75.6% 76.3% 80.9%
Combine 81.1% 81.9% 80.6% 81.0% 83.5%
X
Fusion
Y
X
Y
Descriptor-level
(linear) Kernel-level
GMM
GMM
Fusion
GMM
SVM Score
SVM Score
X
Y
MVSV
M-PCCA Fusion SVM Score
X
Y
Score-levelGMM
GMM
Fusion Score
SVM Score
SVM Score