neural computing: what scale and complexity is...

Photos placed in horizontal position

with even amount of white space

between photos and header

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin

Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2011-XXXXP

Neural Computing:

What Scale and Complexity is Needed? Neuromorphic Computing Workshop, Oak Ridge, TN 2016

1

Brad Aimone, Kris Carlson, Fred Rothganger

Sandia National Laboratories

Neuromorphic computing at SNL

leverages a legacy of research efforts

NeuroXyce

Neural

algorithms Computational

Neuroscience

Biological

Neuroscience

Cyber data

analysis

Human

Dimension

Memory

technology

Non-von Neumann

architectures Micro-sensors

Neural Theory

Applied Neural

Systems

Neural-enabling

hardware

Deployable National

Security Applications

• Cyber Defenses

• Embedded Pattern

Recognition Systems

• Smart Sensor Technologies

Enabling Advanced

Scientific and Data-driven

Computing

• Neural-inspired

communication paradigms

• Adaptive memory

management

• Robust machine learning

3

HAANA is taking a broad approach to leverage

neural algorithms in applications

Applications – Cyber,

Remote imaging

Behavioral

Trajectory Analysis

Response to

Concept Drift

Unsupervised

Feature Extraction

Temporal

Coding

Sparse

Coding

Adaptive

Coding

DPUs ReRAM

Crossbar

Future HAANA

Architecture

FPGA

Device

Modeling

Circuit

Modeling

Materials

Chemistry

Filament

Design

Algorithms

Core

Architecture

Core

Learning

Hardware

Core

Applications

Core

Hybrid

4

HAANA is taking a broad approach to leverage

neural algorithms in applications

Applications – Cyber,

Remote imaging

Behavioral

Trajectory Analysis

Response to

Concept Drift

Unsupervised

Feature Extraction

Temporal

Coding

Sparse

Coding

Adaptive

Coding

DPUs ReRAM

Crossbar

Future HAANA

Architecture

FPGA

Device

Modeling

Circuit

Modeling

Materials

Chemistry

Filament

Design

Algorithms

Core

Architecture

Core

Learning

Hardware

Core

Applications

Core

Hybrid

Guiding principles to our approach:

1. Bring rigor and formality to neural computing algorithms and

hardware

2. Be deliberate in how we use neural inspiration

3. Be open minded about where impact may lie

5

HAANA is taking a broad approach to leverage

neural algorithms in applications

Applications – Cyber,

Remote imaging

Behavioral

Trajectory Analysis

Response to

Concept Drift

Unsupervised

Feature Extraction

Temporal

Coding

Sparse

Coding

Adaptive

Coding

DPUs ReRAM

Crossbar

Future HAANA

Architecture

FPGA

Device

Modeling

Circuit

Modeling

Materials

Chemistry

Filament

Design

Algorithms

Core

Architecture

Core

Learning

Hardware

Core

Applications

Core

Hybrid

Guiding principles to our approach:

1. Bring rigor and formality to neural computing algorithms and

hardware

2. Be deliberate in how we use neural inspiration

3. Be open minded about where impact may lie

Neuromorphic research often confuses neural scope with desired computational abstraction

Scope

Adult neurogenesis

Continuous formation of granule

cell neurons throughout life

Scope

Dentate Gyrus

Millions of primarily GCs

but many different

varieties

Scope

Hippocampus

“Three layer” cortical like

region responsible for

memory and spatial

processing

Scope

Brain

Proteins, DNA, and

cellular structures

Layers

Neurons

Regions

A good approach to look at is the scope that provides a useful function…

Brain

Hippocampus

Dentate Gyrus

Neurogenesis

Na+ Channels

Identify functions that are both clear and have value

Cognition

One shot learning, associative memory

Pattern separation, conjunctive encoding

Novel information encoding

Spiking dynamics

Brain

Hippocampus

Dentate Gyrus

Neurogenesis

Na+ Channels

Desired computational function should determine scope

Cognition

One shot learning, associative memory

Pattern separation, conjunctive encoding

Novel information encoding

Spiking dynamics

Brain

Hippocampus

Dentate Gyrus

Neurogenesis

Na+ Channels

Level of neural abstraction should be determined by functional need

Dentate Gyrus CA3 CA1

Pattern separation &

conjunctive encoding

Auto-association &

Recursive Dynamics

Temporal associations &

Cortical-Hippocampal

Comparisons

Produce high level model and provide increased resolution as needed

Entorhinal

Cortex

Dentate

Gyrus

CA3

CA1

Hippocampus

Systematic abstraction becomes necessary with high complexity

Computational Dentate Gyrus

Goal of proving larger model with clear function

Which components are necessary to produce

desired pattern separation function?

? ?

?

• What scale?

• What types of plasticity?

• What resolution?

• What types of neurons?

Realistic simulations can guide abstraction for computing

What neural scale?

What cell types?

What plasticity?

Neurogenesis?

Etc…

DG model with realistic neuron types and numbers,

anatomy-guided connectivity, and detailed

neurogenesis

Realistic simulations can guide abstraction for computing

Measure “compressiblility” of activity Measure magnitude of activity

Reduced scale simulations are qualitatively different than realistic scales

0 0.5 1 1.5 2 2.5

x 104

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Neuron Number

Tim

e s

tep

0 0.5 1 1.5 2 2.5

x 104

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Neuron Number

Tim

e s

tep

25,000 Granule Cells 300,000 Granule Cells

Neurogenesis is critical for encoding and separation, but depends on scale

0 50 100 150 200 250 30010

-3

10-2

10-1

100

Number of GCs

Fir

ing

Ra

te (

Hz)

0 50 100 150 200 250 3000.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Number of GCs

Co

mp

res

sib

ilit

y

0 0.5 1 1.5 2 2.5

x 104

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Neuron Number

Tim

e s

tep

0 0.5 1 1.5 2 2.5

x 104

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Neuron Number

Tim

e s

tep

25,000 Granule Cells 300,000 Granule Cells

More

information

encoded

Improved

pattern

separation

Can use formal UQ / SA techniques to inform path to abstraction

Important*: Scale is critical

Neurogenesis is necessary form of plasticity to encode novel information

Less important*: Interneuron diversity

Specific neuron dynamics

* For these specific functions

Carlson, in preparation

Abstracted concepts can become computing-friendly

Formalize dentate gyrus /

neurogenesis sparse

coding algorithm

Severa et al., submitted

Identify critical aspects

of computation

(expansive scale;

sparse activation;

neurogenesis)

Simulate at high

level of neural

fidelity

Can we systematically use this approach?

Rothganger et al., Frontiers in Neural Circuits 2014

N2A model framework and

software

• compositionally construct

dynamical neural models

• compiles simulation codes

• potential “front-end” to neural

hardware

Can we systematically use this approach?

Rothganger et al., Frontiers in Neural Circuits 2014

Thanks!