autonomy incubator seminar series: neuromorphic solutions for autonomous land and aerial vehicles

Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 1

Neuromorphic solutions

for autonomous land

and aerial vehicles Massimiliano Versace

CEO, Neurala Inc.

Director, Boston University Neuromorphics Lab

www.neurala.com nl.bu.edu [email protected]

http://www.neurala.com/

http://nl.bu.edu/

mailto:[email protected]


To produce virtual and robotic agents

that autonomously learn to perform

useful tasks


limited intelligence

1:1 ratio robot/

human operator

barriers to adoption

Why are not robots everywhere?


Three ingredients for success

Smart mind

Powerful brain Inexpensive body


The body


Low cost of sensors, new actuators

accelerometer

gyroscope

magnetometer

front and rear cameras

NFC

barometer

speaker

microphone

proximity

light sensor

Bluetooth

GPS

WiFi + cellular

humidity

temperature

navigation

vision

audition



Smart mind



Many robots, many minds

R&D $ R&D $$$

$$ensors

R&D $$

$ensors


One mind for autonomous robots?

learning

parallel processing


Why do we LOVE this mess??!!

Neuromorphic = of the form of neurons, brains Even in apparently simple tasks, many brain

functions work in synergy to help us solve

otherwise VERY HARD PROBLEMS

The brain does not work in silos of competences.

E.g., as you move around in the world, vision helps navigation (e.g., landmark

recognition, obstacle avoidance), navigation helps vision (what to expect in one

environment vs. another).

Ever thought why you can’t recognize the teller of your grocery at the movie theater?


“Stovepiped” sensemakig Rock detector Sky detector & cloud detector Dust devil detector

Onboard Autonomous Science Investigation System for Opportunistic Rover Science, Castano et al., 2007


Extremely un-stovepiped sensemaking!

object recognition

optic flow stereo & depth

environment map

with objects

segmentation

bias where to look

(data acquisition)

coarse

view

scenes help

define objects

object help

define scenes

map helps to find objects

objects help

orient

(e.g. Citgo)

to: obstacle

avoidance

scene recognition


Feedforward and feedback processing

Brain is a massively recurrent architecture

10X E.g. in the visual system, for each output

fiber from thalamus to the visual cortex, the

thalamus receives ~10 feedback fibers

What is this feedback for?

One key aspect of neuromorphic computing is feedback processing

Feedback happens all over the brain


Leave your desk.. ..walk to the conference room.. ..sit in the conference room

Feedforward information flow:

- leave the desk for the talk

- walk down the corridor to the conference room

- find the conference room, find a free spot as close to the exit as humanely

possible

Feedback information flow:

At every moment, you are MATCHING memory-based expectations with sensory

(and other cortical) INPUT

INPUT INPUT INPUT INPUT INPUT

New post-it (Visual)

Desk 0.5 cm lower

(Proprioceptive)

Chanel #5 smell

(Olfactory)

Different door knob (Tactile)

New speakers (Auditory)

EXPECTATION EXPECTATION EXPECTATION EXPECTATION EXPECTATION

Mark coming to the conference room


Feedback focuses on anomalous information

The president of the United States of America

?

& ?

Match memory with current input

Automatically and effortlessly focus on crucial differences


Predictive feedback is ubiquitous in the brain

eel is on the ...

The sentence “eel is on the ...” is preceded by white noise

Percept may be determined by the sound that one expects to hear in auditory

context based on previous language experiences

Expectations amplify consistent components of the white noise, while

suppressing inconsistent components

orange peel is on the orange

wagon wheel is on the wagon

shoe heel is on the shoe


Adaptive Resonance Theory

F2

+ +

- F1

X

Y

+

RESET

ATTENTIONAL

SUBSYSTEM

ORIENTING

SUBSYSTEM

Σ -

+ +

MATCHING

CRITERION

ADAPTIVE RESONANCE THEORY

(ART)

MATCH MISMATCH

Σ Σ

Math behind neuromorphic systems?

x

y

w

𝑑𝑦𝑗

𝑑𝑡= −𝐴𝑦𝑗 + (𝐵 − 𝑦𝑗) 𝑤𝑖,𝑗𝑥𝑖 − (𝑦𝑗 + 𝐶) 𝑦𝑘

𝑛

𝑘≠𝑗

𝑛

𝑖=0

𝑑𝑤𝑖,𝑗

𝑑𝑡= −𝐷𝑦𝑗𝑤𝑖,𝑗 + 𝐸𝑥𝑖𝑦𝑗


NASA STTR Phase II

With Mark Motter, NASA Langley


One algorithm for all robots

Sensory

Motivation

Navigation

Classification

Prefrontal

Cortex

Binding What

& Where

Unclear

Self-

Localization

Multiple Areas

Position-

Invariant Cells

IT

Complex

Cells

V2

Oriented

Edges

V1

Saliency

Map

Posterior PC

Luminance

LGN

Pre-

Processing

Retina

Color

Opponency

LGN?

Curiosity

Drive

Unclear

Goal

Selection Map

Unclear

Reward

Location Map

Unclear

Current

Location

Enthorinal Ctx

Active Goal

Location

Prefrontal Ctx

Reward

Signal

Basal Ganglia

Forward

Spread Trigger

Dentate Gyrus

Reverse

Spread

Prefrontal Ctx

Forward

Spread

CA3

Desired

Next Location

CA1

Motor Turn

Command

Motor Cortex

Motor Run

Command

Motor Cortex

Lack of

Comfort Drive

Unclear

Path-Finding

Reset

Medial Septum

Desired

Heading

Cingulate Ctx

Virtual Environment or Robotic Platform

~32M neurons, ~13B synapses


Example application: MoNETA

Morris water maze

Familiarity

map

Goal

Selection

map

MoNETA’s visual input

Brain diagram

Reverse

spread

Obstacle

map 1st run 4th run 6th run 8th run


One algorithm for all robots


Simulated Mars Rover


fusion

Vestibular

gyro &

accel.

Visual

optic flow

Motor

wheel

velocity

(1) (2)

Feedback improves heading estimation

Source Average error (deg) Avg. error rate (deg/hr) Vestibular 62.8 1.63 Visual 35.9 0.916 Motor 29.9 0.983 Integrated ring 14.9 0.111


How should robots “see”?


Biological vision is:

Multimodal (focus vision on speaker)

• computational savings (up to 30x documented vs. uniformly

sampled images)

• conformal mapping

• some invariance to changes in scale

and rotation Traver and

Bernardino (2010)

Selective (non-uniform sampling)

Predictive (look where the ball is going)


Artificial visual system

exogenous

spatial attention

WHERE

Visual System

Preprocessing

surface

representations

learning

control

object/scene

input

image

cognitive

biases

label

where

eye

movements

WHAT

what


Learning to classify Martian rocks


Expanding the visual system

Temporal continuity:

“how do you know that

the thing you are

looking at now is the

same you look at 1

second ago?”

Upcoming development under the STTR Phase II program:

1) Depth processing, optic-flow aided segmentation, scene “gists”

2) Implementation on robotic platforms


Robotic applications


One brain for all robots: UAVs?

Video courtesy of Mark Motter


Optic flow as a collision clue


Sense and Avoid from Flow Expansion

SAFE model

with Mark Motter, NASA Langley, CIF 2012-13

past environment (s) - action (a) pair

learning rate

reward/punishment value current state/action

random action


Obstacle avoidance via optic flow learning

Cost of crashing ~= cost of deviating course

Cost of crashing >> cost of deviating course

Each point = MAvg over 200 trials

Collision rate with other UAVs in collision path




Rate at which the UAV turns off-course in the

absence of a colliding UAV






Rate at which the UAV turns off-course

after a UAV has passed




Identifying other UAVs: virtual environment


Identifying other UAVs: real video


How do pilots look at things?

1a) Develop navigation and space variant vision on land robot;

1b) Develop collision avoidance for UAV;

2) Merge 1 & 2 on UAV;

NASA STTR Phase II timeline



Smart mind



The brain

100 billion neurons

250 trillion synapses

+ =


Brain power

+


Brains, GPUs, multicore CPUs


Videogames and smartphones

+ =

Graphic Processing Units

(GPUs), now in mobile


Parallel vs. serial

CPU

GPU


2012 2017

# n

eu

ron

s/c

m2, p

ow

er s

up

ply

2023

2012

2015

2017

8M 0.87V

2013

24M 0.65V

2018

12M 0.78V

digital

GF

LO

PS

pe

r W

att

(G

PU

) &

# m

ob

ile

CP

U c

ore

s

6

4

2014

12?

2016

25?

48?

Computing power roadmap

“fancy”


2012 2017

# n

eu

ron

s/c

m2, p

ow

er s

up

ply

2023

Mouse brain

2017

2015

2018

6 cm2

28 V

3 cm2

5 V


2012 2017

# n

eu

ron

s/c

m2, p

ow

er s

up

ply

2023

Human brain

2017

2015 35 m2

2 kV

2018

70 m2

5 kV


The Human Brain Project

$100M/yr.

2013-23

BRAIN Initiative,

projected $300M/yr.

10 yrs.

2012 2017 2023 2017

Faster than we think?


Not only academia!

The NPU will

"not only mimic human-like perception

but also have the ability

to learn how biological brains do”

* See also IBM, Intel


Towards autonomous machines

Smart mind



Outreach: Boston Museum of Science

1.5 million visitors: the most visited cultural attraction in Boston


NASA?...no, MASA!


~1800 visitors used the app Snow inches by day - 2014

0

0.2

0.4

0.6

0.8

1

1.2


Neurala Massimiliano Versace

CEO, Neurala Inc.

www.neurala.com

[email protected]

in partnership with

http://www.neurala.com/


56

To What

Pathway

Hot Spot (log-polar)

Hot Spot (Linear)

Eye mov. (Linear)

Eye/head

mov.

(Linear)

Working

memory (head cent.)

Working

memory (retinal)

Input (log-polar)

Edges

Diffusion

Contours

Working

memory kernel

Shroud (log-polar)

Reset Fit Shroud (log-polar)

Early Vision Processing

Attention Shroud

Working Memory

Eye/Head Movement

Where pathway


57

log-polar input representation

hot-spot working memory new hot spot

(where the eye will look next)

reset signal (different object)

shroud forming (bottom)

and fully formed (top)

Where pathway


Boundary/ completed

Shroud from Where

pathway

View Category

Object Category

Name Category

Teacher Where reset

Associative learning

ART learning

Recurrent excitation

Inhibition Spatial shroud exists Top-down priming

From “where”

pathway

From external

source

Core of “what”

pathway

Generates a name for the foveated object

(unsupervised). Can also take top down

name assigned by external teacher

(supervised)

Groups multiple views of a single object

into an object category. Reset of object

category is gated by Where signal

Groups similar log-polar views

of same object into view categories

What pathway


59

Confidential

Neurala – AFRL site visit,

November 14, 2012

59

bottom-up

activated name

category

top-down

teaching signal

current input

view-

specific

neurons

view-specific neuron weights

learning multiple views

of the same digit

What pathway


Semantic influence on data collection

5

learning

invariance

new instance

(ambiguous view)

?

0

1

2

3

4

5

6

7

8

9

Few presentations

“heat maps”

… 0 1 2 5

initial

hypothesis

“heat maps”

0 5

Where to look next

to maximize

discrimination




Collision rate with otherwise “missed” UAVs



autonomy incubator seminar series: neuromorphic solutions for autonomous land and aerial vehicles

Technology