autonomy incubator seminar series: neuromorphic solutions for autonomous land and aerial vehicles

Neuromorphic solutions

for autonomous land

and aerial vehicles Massimiliano Versace

CEO, Neurala Inc.

Director, Boston University Neuromorphics Lab

www.neurala.com nl.bu.edu versace@neurala.com

To produce virtual and robotic agents

that autonomously learn to perform

useful tasks

limited intelligence

1:1 ratio robot/

human operator

barriers to adoption

Why are not robots everywhere?

Three ingredients for success

Smart mind

Powerful brain Inexpensive body

The body

Low cost of sensors, new actuators

accelerometer

gyroscope

magnetometer

front and rear cameras

barometer

speaker

microphone

proximity

light sensor

Bluetooth

WiFi + cellular

humidity

temperature

navigation

vision

audition

Smart mind

Many robots, many minds

R&D $ R&D $$$

$$ensors

R&D $$

$ensors

One mind for autonomous robots?

learning

parallel processing

Why do we LOVE this mess??!!

Neuromorphic = of the form of neurons, brains Even in apparently simple tasks, many brain

functions work in synergy to help us solve

otherwise VERY HARD PROBLEMS

The brain does not work in silos of competences.

E.g., as you move around in the world, vision helps navigation (e.g., landmark

recognition, obstacle avoidance), navigation helps vision (what to expect in one

environment vs. another).

Ever thought why you can’t recognize the teller of your grocery at the movie theater?

“Stovepiped” sensemakig Rock detector Sky detector & cloud detector Dust devil detector

Onboard Autonomous Science Investigation System for Opportunistic Rover Science, Castano et al., 2007

Extremely un-stovepiped sensemaking!

object recognition

optic flow stereo & depth

environment map

with objects

segmentation

bias where to look

(data acquisition)

coarse

scenes help

define objects

object help

define scenes

map helps to find objects

objects help

orient

(e.g. Citgo)

to: obstacle

avoidance

scene recognition

Feedforward and feedback processing

Brain is a massively recurrent architecture

10X E.g. in the visual system, for each output

fiber from thalamus to the visual cortex, the

thalamus receives ~10 feedback fibers

What is this feedback for?

One key aspect of neuromorphic computing is feedback processing

Feedback happens all over the brain

Leave your desk.. ..walk to the conference room.. ..sit in the conference room

Feedforward information flow:

- leave the desk for the talk

- walk down the corridor to the conference room

- find the conference room, find a free spot as close to the exit as humanely

possible

Feedback information flow:

At every moment, you are MATCHING memory-based expectations with sensory

(and other cortical) INPUT

INPUT INPUT INPUT INPUT INPUT

New post-it (Visual)

Desk 0.5 cm lower

(Proprioceptive)

Chanel #5 smell

(Olfactory)

Different door knob (Tactile)

New speakers (Auditory)

EXPECTATION EXPECTATION EXPECTATION EXPECTATION EXPECTATION

Mark coming to the conference room

Feedback focuses on anomalous information

The president of the United States of America

Match memory with current input

Automatically and effortlessly focus on crucial differences

Predictive feedback is ubiquitous in the brain

eel is on the ...

The sentence “eel is on the ...” is preceded by white noise

Percept may be determined by the sound that one expects to hear in auditory

context based on previous language experiences

Expectations amplify consistent components of the white noise, while

suppressing inconsistent components

orange peel is on the orange

wagon wheel is on the wagon

shoe heel is on the shoe

Adaptive Resonance Theory

ATTENTIONAL

SUBSYSTEM

ORIENTING

SUBSYSTEM

MATCHING

CRITERION

ADAPTIVE RESONANCE THEORY

MATCH MISMATCH

Math behind neuromorphic systems?

𝑑𝑦𝑗

𝑑𝑡= −𝐴𝑦𝑗 + (𝐵 − 𝑦𝑗) 𝑤𝑖,𝑗𝑥𝑖 − (𝑦𝑗 + 𝐶) 𝑦𝑘

𝑘≠𝑗

𝑖=0

𝑑𝑤𝑖,𝑗

𝑑𝑡= −𝐷𝑦𝑗𝑤𝑖,𝑗 + 𝐸𝑥𝑖𝑦𝑗

NASA STTR Phase II

With Mark Motter, NASA Langley

One algorithm for all robots

Sensory

Motivation

Navigation

Classification

Prefrontal

Cortex

Binding What

& Where

Unclear

Localization

Multiple Areas

Position-

Invariant Cells

Complex

Oriented

Saliency

Posterior PC

Luminance

Processing

Retina

Opponency

Curiosity

Unclear

Selection Map

Unclear

Reward

Location Map

Unclear

Current

Location

Enthorinal Ctx

Active Goal

Location

Prefrontal Ctx

Reward

Signal

Basal Ganglia

Forward

Spread Trigger

Dentate Gyrus

Reverse

Spread

Prefrontal Ctx

Forward

Spread

Desired

Next Location

Motor Turn

Command

Motor Cortex

Motor Run

Command

Motor Cortex

Lack of

Comfort Drive

Unclear

Path-Finding

Medial Septum

Desired

Heading

Cingulate Ctx

Virtual Environment or Robotic Platform

~32M neurons, ~13B synapses

Example application: MoNETA

Morris water maze

Familiarity

Selection

MoNETA’s visual input

Brain diagram

Reverse

spread

Obstacle

map 1st run 4th run 6th run 8th run

One algorithm for all robots

Simulated Mars Rover

fusion

Vestibular

gyro &

accel.

Visual

optic flow

velocity

(1) (2)

Feedback improves heading estimation

Source Average error (deg) Avg. error rate (deg/hr) Vestibular 62.8 1.63 Visual 35.9 0.916 Motor 29.9 0.983 Integrated ring 14.9 0.111

How should robots “see”?

Biological vision is:

Multimodal (focus vision on speaker)

• computational savings (up to 30x documented vs. uniformly

sampled images)

• conformal mapping

• some invariance to changes in scale

and rotation Traver and

Bernardino (2010)

Selective (non-uniform sampling)

Predictive (look where the ball is going)

Artificial visual system

exogenous

spatial attention

Visual System

Preprocessing

surface

representations

learning

control

object/scene

cognitive

biases

movements

Learning to classify Martian rocks

Expanding the visual system

Temporal continuity:

“how do you know that

the thing you are

looking at now is the

same you look at 1

second ago?”

Upcoming development under the STTR Phase II program:

1) Depth processing, optic-flow aided segmentation, scene “gists”

2) Implementation on robotic platforms

Robotic applications

One brain for all robots: UAVs?

Video courtesy of Mark Motter

Optic flow as a collision clue

Sense and Avoid from Flow Expansion

SAFE model

with Mark Motter, NASA Langley, CIF 2012-13

past environment (s) - action (a) pair

learning rate

reward/punishment value current state/action

random action

Obstacle avoidance via optic flow learning

Cost of crashing ~= cost of deviating course

Cost of crashing >> cost of deviating course

Each point = MAvg over 200 trials

Collision rate with other UAVs in collision path

Rate at which the UAV turns off-course in the

absence of a colliding UAV

Rate at which the UAV turns off-course

after a UAV has passed

Identifying other UAVs: virtual environment

Identifying other UAVs: real video

How do pilots look at things?

1a) Develop navigation and space variant vision on land robot;

1b) Develop collision avoidance for UAV;

2) Merge 1 & 2 on UAV;

NASA STTR Phase II timeline

Smart mind

The brain

100 billion neurons

250 trillion synapses

Brain power

Brains, GPUs, multicore CPUs

Videogames and smartphones

Graphic Processing Units

(GPUs), now in mobile

Parallel vs. serial

2012 2017

8M 0.87V

24M 0.65V

12M 0.78V

digital

Computing power roadmap

“fancy”

2012 2017

Mouse brain

2012 2017

Human brain

2015 35 m2

The Human Brain Project

$100M/yr.

2013-23

BRAIN Initiative,

projected $300M/yr.

10 yrs.

2012 2017 2023 2017

Faster than we think?

Not only academia!

The NPU will

"not only mimic human-like perception

but also have the ability

to learn how biological brains do”

* See also IBM, Intel

Towards autonomous machines

Smart mind

Outreach: Boston Museum of Science

1.5 million visitors: the most visited cultural attraction in Boston

NASA?...no, MASA!

~1800 visitors used the app Snow inches by day - 2014

Neurala Massimiliano Versace

CEO, Neurala Inc.

www.neurala.com

versace@neurala.com

in partnership with

To What

Pathway

Hot Spot (log-polar)

Hot Spot (Linear)

Eye mov. (Linear)

Eye/head

(Linear)

Working

memory (head cent.)

Working

memory (retinal)

Input (log-polar)

Diffusion

Contours

Working

memory kernel

Shroud (log-polar)

Reset Fit Shroud (log-polar)

Early Vision Processing

Attention Shroud

Working Memory

Eye/Head Movement

Where pathway

log-polar input representation

hot-spot working memory new hot spot

(where the eye will look next)

reset signal (different object)

shroud forming (bottom)

and fully formed (top)

Where pathway

Boundary/ completed

Shroud from Where

pathway

View Category

Object Category

Name Category

Teacher Where reset

Associative learning

ART learning

Recurrent excitation

Inhibition Spatial shroud exists Top-down priming

From “where”

pathway

From external

source

Core of “what”

pathway

Generates a name for the foveated object

(unsupervised). Can also take top down

name assigned by external teacher

(supervised)

Groups multiple views of a single object

into an object category. Reset of object

category is gated by Where signal

Groups similar log-polar views

of same object into view categories

What pathway

Confidential

Neurala – AFRL site visit,

November 14, 2012

bottom-up

activated name

autonomy incubator seminar series: neuromorphic solutions for autonomous land and aerial vehicles

Technology

an astrocyte-modulated neuromorphic central pattern

autonomy and autonomous systems at nasa larc’s autonomy...

neuromorphic enginerring- subthrshold design

emerging technologies - neuromorphic engineering / computing

f09/f10 neuromorphic computing - physikalisches … ·...

neuromorphic electronics - uio

kic innoenergy at the ecosummit berlin 2015: incubator and...

spintronics enabled efficient neuromorphic computing

finding a roadmap to achieve large neuromorphic hardware...

neuromorphic sensing and computing 2019 › ... ·...

neuromorphic computing: from materials to...

neuromorphic computing based processors - isqed ·...

quadrocopter stabilization using neuromorphic embedded...

neuromorphic image processing

applications of neuromorphic computing

neuromorphic computing slides - intel newsroom

neuromorphic microchips

neuromorphic fringe projection proﬁlometry

executionmodel: neuromorphic computation

photonic neuromorphic information processing and reservoir...