beyond correlation filters: learning continuous convolution operators...
TRANSCRIPT
![Page 1: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/1.jpg)
Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, Michael Felsberg
![Page 2: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/2.jpg)
Discriminative Correlation Filters (DCF)
Applications
• Object recognition
• Object detection
• Object tracking
– Among state-of-the-art since 2014
– KCF, DSST, HCF, SRDCF, Staple …
2Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 3: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/3.jpg)
Discriminative Correlation Filters (DCF)
3Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Single-resolution feature map
Limitations:Coarse output
scores
![Page 4: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/4.jpg)
DCF Limitations:1. Single-resolution feature map
• Why a problem?
– Combine convolutional layers of a CNN
• Shallow layers: low invariance – high resolution
• Deep layers: high invariance – low resolution
• How to solve?
– Explicit resampling?
• Artefacts, information loss, redundant data
– Independent DCFs with late fusion?
• Sub-optimal, correlations between layers
4Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 5: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/5.jpg)
DCF Limitations:2. Coarse output scores
• Why a problem?
– Accurate localization
• Sub-grid (e.g. HOG grid) or sub-pixel accuracy
• More accurate annotations=> less drift
• How to solve?
– Interpolation?
• Which interpolation strategy?
– Interweaving?
• Costly
5Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 6: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/6.jpg)
DCF Limitations:3. Coarse labels
• Why a problem?
– Accurate learning
• Sub-grid or sub-pixel supervision
• How to solve?
– Interweaving?
• Costly
– Explicit interpolation of features?
• Artefacts
6Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 7: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/7.jpg)
Our Approach: Overview
7Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Continuous filters
Continuous output
Multi-resolution features
![Page 8: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/8.jpg)
Multiresolution Features
8Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 9: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/9.jpg)
Interpolation Operator
9Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 10: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/10.jpg)
Interpolation Operator
10Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 11: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/11.jpg)
Convolution Operator
11Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 12: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/12.jpg)
Training Loss
12Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
[Danelljan et al., ICCV 2015]
![Page 13: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/13.jpg)
Training Loss – Fourier Domain
13Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 14: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/14.jpg)
Training Loss – Fourier Domain
14Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 15: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/15.jpg)
Localization
15Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 16: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/16.jpg)
Localization
16Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 17: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/17.jpg)
How to set and ?
• Use periodic summation of functions :
• Gaussian function for
• Cubic spline kernel for
• Fourier coefficients with Poisson’s summation formula:
17Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 18: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/18.jpg)
Object Tracking Framework: Features
• VGG network
– Pre-trained on ImageNet
– No fine-tuning on application specific data
18Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 19: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/19.jpg)
C-COT: Conjugate Gradient
Only need to evaluate
No sparse matrix handling
Finite memory
Warm starting: non-trivial
Tuning of pre-conditioners
SRDCF: Gauss-Seidel
Explicit computation of
Sparse matrix handling
“Infinite” memory
Warm starting: trivial
Object Tracking Framework: Optimization
Solving
19Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 20: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/20.jpg)
Object Tracking Framework: Pipeline
• Simple: … – Track – Train – Track – Train – …
• No thresholds
• No hidden “tricks”
20Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 21: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/21.jpg)
Experiments: Object Tracking
• 3 datasets: OTB-100, TempleColor, VOT2015
• Layer fusion on OTB:
• Compared to explicit resampling in DCF
– Performance gain: AUC
– Efficiency gain: data size
21Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 22: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/22.jpg)
Experiments: OTB (100 videos)
22Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 23: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/23.jpg)
Experiments: Temple-Color (128 videos)
23Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 24: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/24.jpg)
Experiments: VOT2016
24Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
[Matej et al., ECCV VOT workshop 2016]
![Page 25: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/25.jpg)
Object Tracking: Speed
• With CNN features: slow ~1 FPS (no GPU)
• With HOG features: ~ real time at SRDCF performance
25Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 26: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/26.jpg)
Feature Point Tracking Framework
• Grayscale pixel features,
• Uniform regularization,
26Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 27: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/27.jpg)
Experiments: Feature Point Tracking
• The Sintel dataset
27Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 28: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/28.jpg)
Feature Point Tracking: KITTI
28Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
C-COT
KLT
![Page 29: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/29.jpg)
Future Work
• Features
– Fine tuning
– Unsupervised learning
• Optimization
– Warm start in CG (theory and heuristics)
– Preconditioners
– Implementation aspects
– Alternative strategies or update rules
• Further explore of the continuous formulation
29Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 30: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/30.jpg)
Conclusions
• Continuous domain learning formulation
– Multi-resolution deep feature maps
– Sub-pixel accurate localization
– Sub-pixel supervision
• Superior results for two applications
– Object tracking
– Feature point tracking
30Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 31: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/31.jpg)
Oral and poster: O-4B-03
Friday afternoon(last session)
31Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
![Page 32: Beyond Correlation Filters: Learning Continuous Convolution Operators …data.votchallenge.net/vot2016/presentations/presentation... · 2017-05-18 · Discriminative Correlation Filters](https://reader035.vdocuments.us/reader035/viewer/2022070804/5f0354b97e708231d408af1c/html5/thumbnails/32.jpg)
www.liu.se
Martin Danelljan