fa-harris a fast and asynchronous corner detector for event … · 2019. 12. 1. · fa-harris: a...
TRANSCRIPT
![Page 1: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/1.jpg)
FA-Harris:A Fast and Asynchronous Corner Detector
for Event Cameras
Ruoxiang Li1, Dianxi Shi2,3, Yongjun Zhang2, Kaiyue Li1, Ruihao Li2,3
1 National University of Defense Technology, China
2Artificial Intelligence Research Center (AIRC), National Innovation Institute of Defense Technology (NIIDT), China
3Tianjin Artificial Intelligence Innovation Center (TAIIC), China
![Page 2: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/2.jpg)
Research Motivation
• The existing visual SLAM systems suffer from several challenges, like high speed
motion, high dynamic range, etc.
• The event camera, with the advantages of low latency, low energy consumption,
high temporal resolution and high dynamic range, responds to local, pixel-level
brightness changes instead of standard frames.
DAVIS 240C Dataset[1]
[1] Mueggler et al., The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM, IJRR 2017.
![Page 3: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/3.jpg)
Research Motivation
• The existing visual SLAM systems suffer from several challenges, like high speed
motion, high dynamic range, etc.
• The event camera, with the advantages of low latency, low energy consumption,
high temporal resolution and high dynamic range, responds to local, pixel-level
brightness changes instead of standard frames.
[1] Mueggler et al., The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM, IJRR 2017.
But standard vision algorithms can’t be applied directly.
DAVIS 240C Dataset[1]
![Page 4: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/4.jpg)
Research Motivation
• The existing visual SLAM systems suffer from several challenges, like high speed
motion, high dynamic range, etc.
• The event camera, with the advantages of low latency, low energy consumption,
high temporal resolution and high dynamic range, responds to local, pixel-level
brightness changes instead of standard frames.
[1] Mueggler et al., The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM, IJRR 2017.
But standard vision algorithms can’t be applied directly.
Feature DetectionDAVIS 240C Dataset[1]
![Page 5: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/5.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Corner Candidate
SelectionEvent Filter
Get Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
![Page 6: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/6.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Corner Candidate
SelectionEvent Filter
Get Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
• Surface of Active Events (SAE)
![Page 7: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/7.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Corner Candidate
SelectionEvent Filter
Get Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
• Surface of Active Events (SAE)
Local SAE
![Page 8: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/8.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Event FilterGet Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
(a) Local SAE
Corner Candidate
Selection
(b) Top view
![Page 9: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/9.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Event FilterGet Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
(a) Local SAE
Corner Candidate
Selection
1514
13
12
11
10
9
8 7 6
5
4
3
2
0
1
(b) Top view (c) Inner Circle
![Page 10: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/10.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Event FilterGet Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
(a) Local SAE
Corner Candidate
Selection
1514
13
12
11
(b) Top view (c) Inner Circle
![Page 11: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/11.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Event FilterGet Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
(a) Local SAE
Corner Candidate
Selection
1514
13
12
11
(b) Top view (c) Inner Circle
0
1
2
4
3
5
6
8
7
910
11
12
13
14
15
16
17
18 19
(d) Outer Circle
![Page 12: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/12.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Event FilterGet Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
(a) Local SAE
Corner Candidate
Selection
1514
13
12
11
(b) Top view (c) Inner Circle
14
15
16
17
18 19
(d) Outer Circle
![Page 13: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/13.jpg)
FA-Harris - Proposed Pipeline
Corner Candidate
Refinement
FA-Harris Detector
Corner Candidate
SelectionEvent Filter
Get Local Patch
Calculate Score
Global SAE
Maintenance
Local SAE ExtractionEvent-streams Event-corners
Global SAE Updating
Local SAE Local Patch
![Page 14: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/14.jpg)
Global SAE Construction and Updating
• Our method maintains a global SAE with the size width×height instead of
maintaining a local SAE for each pixel in the imaging plane[2].
• It greatly contributes to the speed-up of our algorithm.
[2] Vasco et al., Fast event-based Harris corner detection exploiting the advantages of event-driven cameras, IROS 2016.
![Page 15: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/15.jpg)
Global SAE Construction and Updating
[2] Vasco et al., Fast event-based Harris corner detection exploiting the advantages of event-driven cameras, IROS 2016.
Real-time performance of the methods with our proposed strategy or not.
• Our method maintains a global SAE with the size width×height instead of
maintaining a local SAE for each pixel in the imaging plane[2].
• It greatly contributes to the speed-up of our algorithm.
![Page 16: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/16.jpg)
Global SAE Construction and Updating
[2] Vasco et al., Fast event-based Harris corner detection exploiting the advantages of event-driven cameras, IROS 2016.
2× faster
• Our method maintains a global SAE with the size width×height instead of
maintaining a local SAE for each pixel in the imaging plane[2].
• It greatly contributes to the speed-up of our algorithm.
Real-time performance of the methods with our proposed strategy or not.
![Page 17: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/17.jpg)
• Our method employs a corner candidate selection and refinement strategy.
Corner Candidate Selection and Refinement
![Page 18: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/18.jpg)
• Our method employs a corner candidate selection and refinement strategy.
• Corner candidate selection method subsamples the original event-streams to enhance the
real-time performance of our algorithm.
Events Corner Candidates
Corner Candidate Selection and Refinement
![Page 19: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/19.jpg)
• Our method employs a corner candidate selection and refinement strategy.
• Corner candidate selection method subsamples the original event-streams to enhance the
real-time performance of our algorithm.
• Corner candidate refinement method helps to achieve better accuracy performance.
Events Corner Candidates Event-corners
Corner Candidate Selection and Refinement
![Page 20: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/20.jpg)
Event-corners in Spatio-temporal System
OursArc*[4]eFAST*[3]
eHarris* G-eHarris*Events
[3] Mueggler et al., Fast Event-based Corner Detection, BMVC 2017.
[4] Alzugaray et al., Asynchronous Corner Detection and Tracking for Event Cameras in Real Time, RAL 2018.
![Page 21: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/21.jpg)
Comparison
Ours
Our proposed method:
Events
0.3x
can detect the corners with large angles
0.3x
eHarris*
![Page 22: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/22.jpg)
Comparison
Ours
Our proposed method:
can reduce noise and wrong detections
Events
0.3x
can detect the corners with large angles
Arc*
![Page 23: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/23.jpg)
Evaluation
FPR
The False Positive Rate (%) of different
event-based corner detectors on different
scenes.
Ours
![Page 24: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/24.jpg)
Evaluation
Accuracy
The accuracy (%) of different event-based
corner detectors on different scenes.
0Ours
![Page 25: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/25.jpg)
Evaluation
Ours
Computational performance of different event-based corner detectors.
![Page 26: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/26.jpg)
Evaluation
Ours
Computational performance of different event-based corner detectors.
4˟8˟
![Page 27: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/27.jpg)
Conclusions
• Real-time: 8× faster than eHarris*
• Accuracy: Better performance compared to the previous methods
• Asynchronous: Detects event-corners on event-streams directly
![Page 28: FA-Harris A Fast and Asynchronous Corner Detector for Event … · 2019. 12. 1. · FA-Harris: A Fast and Asynchronous Corner Detector for Event Cameras Ruoxiang Li1, Dianxi Shi2,3,](https://reader033.vdocuments.us/reader033/viewer/2022061000/60af3b79f569e055de67a744/html5/thumbnails/28.jpg)
Thanks for your attention!