mediaeval 2016 - ranking images and videos on visual interestingness by visual sentiment features
Post on 09-Jan-2017
25 Views
Preview:
TRANSCRIPT
1
Soheil Rayatdoost, Mohammad SoleymaniSwiss Center for Affective Sciences
University of GenevaSwitzerland
Ranking Images and Videos on Visual Interestingness by Visual Sentiment Features
What makes an video-image interesting• Novelty
• Uncertainty
• Conflict
• Complexity
• Comprehensibility
• Familiarity
• Emotional content
22/8
Automatic detection- Features:• Visual• Visual sentiment adjective noun-pairs• Deep learning (fc7)
• Audio:• The extended Geneva Minimalistic Acoustic Parameter(eGeMAPS)
- Regression models:• Linear regression• Support vector regression (SVR)• Sparse approximation weighted regression (SPARROW)
- A Principal Component Analysis (PCA) for dimensionality reduction
33/8
Our Method for video sub challenge
4
Shots …
MVSO
Feature Extraction
SVR-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐SPARROW
Regression
OpenSMILE
Video
Audio
Ranking
Frames
Visual sentiment descriptors
Deep learning features
eGeMAPS
4/8
Our Method for image sub challenge
5
Key frame
Linear Regression
MVSO
Visual sentiment descriptors
Deep learning features
Feature ExtractionRegression
Ranking
5/8
Results
6
Data Task Method Features MAP ↑Image LR MVSO+fc7 0.1710Video SPARROW MVSO+fc7 0.2617Video SPARROW Baseline 0.2414Video SPARROW eGeMAPS 0.1987Image LR MVSO+fc7 0.1704Video SPARROW MVSO+fc7 0.1710Video SPARROW Baseline 0.1497Video SPARROW eGeMAPS 0.1367
Dev. Set
Eval. Set
6/8
Conclusions and future works
• Mid-level visual descriptors Affective nature of interestingness
• Our features are all static and frame-based
• Temporal information The small size of the dataset
• Larger scale datasets Sophisticated methods
• Audio features
77/8
8
Thanks
top related