Upload File

amr suleiman, zhengdong zhang, luca carlone, sertac...

  • Home
  • Documents
  • Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac ...web.mit.edu/sze/www/navion/2018_vlsi_navion_poster.pdfAmr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, and Vivienne
1
Navion: A Fully Integrated Energy-Efficient Visual-Iner:al Odometry Accelerator for Autonomous Naviga:on of Nano Drones Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, and Vivienne Sze Demo: Trevor Henderson and Diana Wo@ Session/Paper No. : 13.1 Mo:va:on Virtual and augmented reality on energy-constrained devices Sponsors Want to know more? Visit navion.mit.edu Navion Results Key Challenges • Large frame memories for feature tracking and stereo computaFon • Large graph memory with irregular memory access • Slow linear solver Our Approach • Image compression • 4.4x smaller frame memory • 4.9x less power • Sparse feature tracks data structure • 5.4x smaller graph memory • Sparse linear solver • 5.2x smaller linear solver memory • 7.2x speed up Technology 65nm CMOS Chip area 4.0 x 5.0 mm 2 Logic gates 2,043 kgates SRAM 854 KB Frame size 480x752 Camera 28 – 171 fps Keyframe 16 – 90 fps Power 27 – 43.2 mW PlaUorm Xeon [2] ARM [2] FPGA [2] ASIC (This work) Accuracy* 0.20% 0.20% 0.24% 0.28% Camera max fps 21 5 20 171 Keyframe max fps 13 3 5 90 Power (W) 26.10 2.33 1.46 0.024 No GPS Loca:on (Trajectory) 3D Map Use images and inerFal data for state es0ma0on 5.0 mm 4.0 mm Feature Tracking Feature Detection Undistort & Rectify Undistort & Rectify Sparse Stereo Linear Solver IFE Shared Memory Graph Horizon States VFE Control Marginal References [1] C. Forster, L. Carlone, F. Dellaert, D. Scaramuzza. “On-manifold preintegraFon theory for fast and accurate visual-inerFal navigaFon,” IEEE Trans. RoboFcs, 2016 [2] Z. Zhang, A. Suleiman, L. Carlone, V. Sze, S. Karaman, “Visual-InerFal Odometry on Chip: An Algorithm-and-Hardware Co-design Approach,” RSS 2017. Vision Frontend (VFE) Backend (BE) Feature Detec:on (Iden:fy features) • Extract Harris corners (up to 200 per frame) • Search and distribute across the whole frame • Compute for every keyframe Feature Tracking (Es:mate movement of features) • Lucas-Kanade pyramidal opFcal flow • Supports up to 3 pyramid levels • Compute for all frames (measure movement between keyframes) 3D Stereo (Es:mate depth of features – a.k.a. landmarks) • Sparse block matching • Compute for every keyframe Fuses vision and IMU esFmates to refine the final state esFmates (x’) Memory size 3.5 MB 2.4 MB 1.6 MB 854 kB 1.5x 2.6x 4.1x Image Compression Two-stage Storage Sparse LS Memory Baseline Frame Graph Linear Solver Misc. IMU Frontend (IFE) Autonomous navigaFon of nano drones x’ 1 x’ 2 x’ 3 L k Solve with state-of-the-art factor graph algorithm Minimize inconsistencies between measurements across Fme Update state es0mates (x’) Over 4000+ factors in op0miza0on IMU Factors Vision Factors t=1 t=2 t=3 *Measured on EuRoc MAV dataset VIO: Visual-Iner:al Odometry Gyroscope: EsFmate RotaFon Accelerometer: EsFmate TranslaFon Pre-integra:on: Integrate IMU measurements into one per keyframe [1] Navion Chip Architecture Backend Control Data & Control Bus Build Graph Linear Solver Linearize Marginal Retract Graph Linear Solver Horizon States Shared Memory Floating Point Arithmetic Matrix Operations Cholesky Back Substitute Rodrigues Operations Feature Tracking (FT) Previous Frame Line Buffers Feature Detection (FD) Undistort & Rectify (UR) Undistort & Rectify (UR) Data & Control Bus Sparse Stereo (SS) Vision Frontend Control RANSAC Fixed Point Arithmetic Point Cloud Pre-Integration Floating Point Arithmetic IMU memory Current Frame Left Frame Right Frame Vision Frontend (VFE) IMU Frontend (IFE) Backend (BE) Register File Fully Integrated System (no off-chip processing or storage) VIO is a special instance of Simultaneous Localiza6on and Mapping (SLAM)

Upload: others

Post on 25-May-2020

3 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac ...web.mit.edu/sze/www/navion/2018_vlsi_navion_poster.pdfAmr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, and Vivienne

Navion:AFullyIntegratedEnergy-EfficientVisual-Iner:alOdometryAcceleratorforAutonomousNaviga:onofNanoDronesAmrSuleiman,ZhengdongZhang,LucaCarlone,SertacKaraman,andVivienneSze

Demo:TrevorHendersonandDianaWo@

Session/Paper No. : 13.1

Mo:va:on

Virtualandaugmentedrealityonenergy-constraineddevices

SponsorsWanttoknowmore?Visitnavion.mit.edu

NavionResults

KeyChallenges•  Large frame memories for feature

trackingandstereocomputaFon•  Large graph memory with irregular

memoryaccess•  Slowlinearsolver

OurApproach•  Imagecompression

•  4.4xsmallerframememory•  4.9xlesspower

•  Sparsefeaturetracksdatastructure•  5.4xsmallergraphmemory

•  Sparselinearsolver•  5.2xsmallerlinearsolvermemory•  7.2xspeedup

Technology 65nmCMOSChiparea 4.0x5.0mm2

Logicgates 2,043kgatesSRAM 854KBFramesize 480x752Camera 28–171fpsKeyframe 16–90fpsPower 27– 43.2mW

PlaUorm Xeon[2]

ARM[2]

FPGA[2]

ASIC(Thiswork)

Accuracy* 0.20% 0.20% 0.24% 0.28%Cameramaxfps 21 5 20 171Keyframemaxfps 13 3 5 90Power(W) 26.10 2.33 1.46 0.024

NoGPS

Loca:on(Trajectory)

3DMap

UseimagesandinerFaldataforstatees0ma0on

5.0 mm

4.0

mm

Feature Tracking

Feature Detection

Undistort& Rectify

Undistort& Rectify

Sparse StereoLinear Solver

IFE

SharedMemory

Graph

Horizon States

VFE Control

Marginal

References[1]C.Forster,L.Carlone,F.Dellaert,D.Scaramuzza.“On-manifoldpreintegraFontheoryforfastandaccuratevisual-inerFalnavigaFon,”IEEETrans.RoboFcs,2016[2]Z.Zhang,A.Suleiman,L.Carlone,V.Sze,S.Karaman,“Visual-InerFalOdometryonChip:AnAlgorithm-and-HardwareCo-designApproach,”RSS2017.

VisionFrontend(VFE) Backend(BE)FeatureDetec:on(Iden:fyfeatures)•  ExtractHarriscorners(upto200perframe)•  Searchanddistributeacrossthewholeframe•  Computeforeverykeyframe

FeatureTracking(Es:matemovementoffeatures)•  Lucas-KanadepyramidalopFcalflow•  Supportsupto3pyramidlevels•  Computeforallframes(measuremovement

betweenkeyframes)

3DStereo(Es:matedepthoffeatures–a.k.a.landmarks)•  Sparseblockmatching•  Computeforeverykeyframe

FusesvisionandIMUesFmatestorefinethefinalstateesFmates(x’)

Mem

ory

size

3.5 MB 2.4 MB 1.6 MB 854 kB

1.5x 2.6x 4.1x

Image Compression

Two-stage Storage

Sparse LS Memory

Baseline

Frame Graph Linear Solver

Misc.

IMUFrontend(IFE)

AutonomousnavigaFonofnanodrones

x’1

x’2

x’3

Lk

Solvewithstate-of-the-artfactorgraphalgorithm

MinimizeinconsistenciesbetweenmeasurementsacrossFme

Updatestatees0mates(x’)

Over4000+factorsinop0miza0on

IMUFactors VisionFactorst=1

t=2 t=3

*MeasuredonEuRocMAVdataset

VIO:Visual-Iner:alOdometry

Gyroscope: EsFmateRotaFonAccelerometer: EsFmateTranslaFon

Pre-integra:on: IntegrateIMUmeasurementsintooneperkeyframe[1]

NavionChipArchitectureBackend Control

Data & Control BusBuild Graph

Linear Solver

Linearize

Marginal

Retract

GraphLinear Solver

Horizon StatesShared

Memory

Floating Point

Arithmetic

Matrix Operations

Cholesky

Back Substitute

Rodrigues Operations

Feature Tracking

(FT)

Previous FrameLine Buffers

Feature Detection

(FD)

Undistort & Rectify

(UR)

Undistort & Rectify

(UR)

Data & Control Bus

Sparse Stereo (SS)

Vision Frontend Control

RANSAC Fixed Point Arithmetic Point Cloud Pre-IntegrationFloating Point

ArithmeticIMU

memory

Current Frame

Left Frame

Right Frame

Vision Frontend (VFE)

IMU Frontend (IFE)

Backend (BE)

Register File

FullyIntegratedSystem

(nooff-chipprocessingorstorage)

VIOisaspecialinstanceofSimultaneousLocaliza6onandMapping(SLAM)

Distributed Trajectory Estimation with Privacy and ...itzsid.github.io/itzsid.github.io/publications/Choudhary16icra.pdf · Siddharth Choudhary, Luca Carlone, Carlos Nieto, John Rogers,

Rajat Talak, Sertac Karaman, Eytan ModianoRajat Talak, Sertac Karaman, Eytan Modiano Abstract—Age of information (AoI) is a recently proposed metric for measuring information freshness

Gregoris Ioannou and Sertac Sonan - Home | etui

Unwrapping Low-rank Textures on Generalized Cylindrical Surfacesyima/matrix-rank/Files/iccv11_curv… · Unwrapping Low-rank Textures on Generalized Cylindrical Surfaces Zhengdong

Counterexample-Guided Safety Contracts for Autonomous Drivinglucasl/assets/files/WAFR_2018_paper_57.pdf · Russ Tedrake1; 2, Sertac Karaman , and Daniela Rus 1 Toyota Research Institute,

Deep Learning for NLP @Noah€¦ · Deep Learning for NLP @Noah: progress, challenges and opportunities Zhengdong Lu ... Deep Matching Models: • Models addressing different aspects

Fostering Patience in the Classroom: Results from ...home.ku.edu.tr/~sertac/Time_Intervention.pdfFostering Patience in the Classroom: Results from Randomized Educational Intervention

Carlos Montalvo Foy H. Moody High School; Corpus Christi ISD Faculty Mentor: Dr. Zhengdong Cheng Artie McFerrin Dept. of Chemical Engineering

Senior Member Update Results - 17 February 2018 SM Panel … · Barber Kenneth Bartell Alexandra Bastioli Simone Baumann Sabine Bauwelinck Johan BAYHAN Sertac BAZAWADA SURESH Bazzi

Rajat Talak, Sertac Karaman, Eytan Modianomodiano/papers/CV_C_212.pdf · Rajat Talak, Sertac Karaman, Eytan Modiano Abstract—Age of information (AoI) is a recently proposed metric

A STUDY OF THE•RENNE EFFECR T COSMO CARLONE

TEMPORAL ANALYSIS OF JET AND VORTEX ACTUATOR (JAVA ... · Hasan Gunes*1, Sertac Cadirci1, Franceso Baldani2, Bernd Peters3, Ulrich Rist3 1Department of Mechanical Engineering, Istanbul

Modeling and Optimization of Direct Chill Casting to ... · Sabau, Kozo Saito, Marwan Khraisheh, Mohamed H. Hassan, Kazunori Kuwana, Shridas Ningileri, Zhengdong Long, John Clark,

Anytime Motion Planning using the RRT$^*$people.csail.mit.edu/mwalter/papers/karaman11.pdf · Sertac Karaman Matthew R. Walter Alejandro Perez Emilio Frazzoli Seth Teller Abstract—The

Does Self-Relevance Affect Information Processing ...home.ku.edu.tr/~sertac/Bayesian.pdf · 2. Experimental Design and Procedures 2.1. Design The main treatment variable in the experiment

AAPM2012 MGuerrero Carlone - AMOS Onlineamos3.aapm.org/abstracts/pdf/68-19764-230349-86282.pdf · Marco Carlone, PhD Prince Margaret Hospital ... • LQ model matches other more sophisticated

Phase Transitions in Colloidal Suspensions of Disks Zhengdong Cheng, Texas Engineering Experiment Station, DMR 1006870 Despite their natural abundance

Rapidly-exploring Random Trees (RRTs) for Efficient Motion Planning RSS Lecture 10 Monday, 10 March 2014 Prof. Seth Teller (Thanks to Sertac Karaman for

Technical Writing and Presentation – Graduate English Copyright © 2003 CheMech. All Rights reserved Professor Zhengdong Wang, Ph.D. EAST CHINA UNIVERSITY

Sertac Mustafaoglu - Urban / Retail / Street Furniture Design Selection

Sertac Mustafaoglu - Selected Graphic Design Works

Heidi Carlone Curriculum Vitae (September 2015)Heidi Carlone CV Page 3 of 24 Education, 5(2), 477-493. 16. Carlone, H.B. & Johnson, A. (2007). Understanding the science experiences

Advice and Leader Selection Mechanisms in Group Decision ...home.ku.edu.tr/~sertac/PrefComm.pdf · Turkey (email: sertac@ku. edu.tr, phone: +90 (212) 3381880, fax: +90 (212) 338-1393)

Portable Dental Unit PC2635 · 2012. 7. 2. · Title: Portable Dental Unit PC2635 Author: Mike Carlone Created Date: 7/2/2012 5:24:07 PM

Author : N. Sertac Artan, Haowei Yuan, and H. Jonathan Chao Publisher/Conf : IEEE GLOBECOM 2008 Speaker : Chen Deyu Data : 2009.10.14

ORYX 2.0 - Worcester Polytechnic Institute (WPI) NASA’s Mars Rovers Carlone 2.1.2 Google Lunar X-Prize and Red Rover Carlone 2.2 Rover Mobility as the Common Problem Fagan 2.2.1

Object-oriented Neural Programming (OONP) for … Neural Programming (OONP) for Document Understanding Zhengdong Lu1 Haotian Cui 2* Xianggen Liu * Yukun Yan * Daqi Zheng1 1DeeplyCurious.ai

Zhengdong New District-problem and Draft Solution

Asymptotically-optimal Path Planning for Manipulation ...people.csail.mit.edu/mwalter/agile/publications/perez11.pdf · Alejandro Perez Sertac Karaman Alexander Shkolnik Emilio Frazzoli

Incremental Sampling-based Algorithms for Optimal Motion … · 2010. 5. 5. · Sertac Karaman Emilio Frazzoli Abstract—During the last decade, incremental sampling-based motion

Diana Wofk , Fangchang Ma , Tien-Ju Yang, Sertac Karaman, Vivienne Szefastdepth.mit.edu/2019_icra_fastdepth.pdf · 2019-03-01 · Diana Wofk , Fangchang Ma , Tien-Ju Yang, Sertac

Camera Calibration with Lens Distortion from Low …perception.csl.illinois.edu/matrix-rank/Files/...Camera Calibration with Lens Distortion from Low-rank Textures Zhengdong Zhang

Teachers’ Knowledge and Beliefs About English …...learners and their mathematics and science achievement (Carlone, Haun-Frank, & Webb, 2011; Lewis et al., 2012; Llosa et al., 2016)

Closed-loop Pallet Engagement in an Unstructured Environmentpeople.csail.mit.edu/mwalter/papers/karaman10.pdf · Closed-loop Pallet Engagement in an Unstructured Environment Sertac

D ESIGN E NGINEERING : C ELL E NCAPSULATION Presented by: Audrey Gonzalez and Carlos Montalvo (CCISD) Faculty mentor: Dr. Zhengdong Cheng of the Artie