neural computing: what scale and complexity is...
TRANSCRIPT
Photos placed in horizontal position
with even amount of white space
between photos and header
Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin
Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2011-XXXXP
Neural Computing:
What Scale and Complexity is Needed? Neuromorphic Computing Workshop, Oak Ridge, TN 2016
1
Brad Aimone, Kris Carlson, Fred Rothganger
Sandia National Laboratories
Neuromorphic computing at SNL
leverages a legacy of research efforts
NeuroXyce
Neural
algorithms Computational
Neuroscience
Biological
Neuroscience
Cyber data
analysis
Human
Dimension
Memory
technology
Non-von Neumann
architectures Micro-sensors
Neural Theory
Applied Neural
Systems
Neural-enabling
hardware
Deployable National
Security Applications
• Cyber Defenses
• Embedded Pattern
Recognition Systems
• Smart Sensor Technologies
Enabling Advanced
Scientific and Data-driven
Computing
• Neural-inspired
communication paradigms
• Adaptive memory
management
• Robust machine learning
3
HAANA is taking a broad approach to leverage
neural algorithms in applications
Applications – Cyber,
Remote imaging
Behavioral
Trajectory Analysis
Response to
Concept Drift
Unsupervised
Feature Extraction
Temporal
Coding
Sparse
Coding
Adaptive
Coding
DPUs ReRAM
Crossbar
Future HAANA
Architecture
FPGA
Device
Modeling
Circuit
Modeling
Materials
Chemistry
Filament
Design
Algorithms
Core
Architecture
Core
Learning
Hardware
Core
Applications
Core
Hybrid
4
HAANA is taking a broad approach to leverage
neural algorithms in applications
Applications – Cyber,
Remote imaging
Behavioral
Trajectory Analysis
Response to
Concept Drift
Unsupervised
Feature Extraction
Temporal
Coding
Sparse
Coding
Adaptive
Coding
DPUs ReRAM
Crossbar
Future HAANA
Architecture
FPGA
Device
Modeling
Circuit
Modeling
Materials
Chemistry
Filament
Design
Algorithms
Core
Architecture
Core
Learning
Hardware
Core
Applications
Core
Hybrid
Guiding principles to our approach:
1. Bring rigor and formality to neural computing algorithms and
hardware
2. Be deliberate in how we use neural inspiration
3. Be open minded about where impact may lie
5
HAANA is taking a broad approach to leverage
neural algorithms in applications
Applications – Cyber,
Remote imaging
Behavioral
Trajectory Analysis
Response to
Concept Drift
Unsupervised
Feature Extraction
Temporal
Coding
Sparse
Coding
Adaptive
Coding
DPUs ReRAM
Crossbar
Future HAANA
Architecture
FPGA
Device
Modeling
Circuit
Modeling
Materials
Chemistry
Filament
Design
Algorithms
Core
Architecture
Core
Learning
Hardware
Core
Applications
Core
Hybrid
Guiding principles to our approach:
1. Bring rigor and formality to neural computing algorithms and
hardware
2. Be deliberate in how we use neural inspiration
3. Be open minded about where impact may lie
Neuromorphic research often confuses neural scope with desired computational abstraction
Scope
Adult neurogenesis
Continuous formation of granule
cell neurons throughout life
Scope
Dentate Gyrus
Millions of primarily GCs
but many different
varieties
Scope
Hippocampus
“Three layer” cortical like
region responsible for
memory and spatial
processing
Scope
Brain
Proteins, DNA, and
cellular structures
Layers
Neurons
Regions
A good approach to look at is the scope that provides a useful function…
Brain
Hippocampus
Dentate Gyrus
Neurogenesis
Na+ Channels
Identify functions that are both clear and have value
Cognition
One shot learning, associative memory
Pattern separation, conjunctive encoding
Novel information encoding
Spiking dynamics
Brain
Hippocampus
Dentate Gyrus
Neurogenesis
Na+ Channels
Desired computational function should determine scope
Cognition
One shot learning, associative memory
Pattern separation, conjunctive encoding
Novel information encoding
Spiking dynamics
Brain
Hippocampus
Dentate Gyrus
Neurogenesis
Na+ Channels
Level of neural abstraction should be determined by functional need
Dentate Gyrus CA3 CA1
Pattern separation &
conjunctive encoding
Auto-association &
Recursive Dynamics
Temporal associations &
Cortical-Hippocampal
Comparisons
Produce high level model and provide increased resolution as needed
Entorhinal
Cortex
Dentate
Gyrus
CA3
CA1
Hippocampus
Systematic abstraction becomes necessary with high complexity
Computational Dentate Gyrus
Goal of proving larger model with clear function
Which components are necessary to produce
desired pattern separation function?
? ?
?
• What scale?
• What types of plasticity?
• What resolution?
• What types of neurons?
Realistic simulations can guide abstraction for computing
What neural scale?
What cell types?
What plasticity?
Neurogenesis?
Etc…
DG model with realistic neuron types and numbers,
anatomy-guided connectivity, and detailed
neurogenesis
Realistic simulations can guide abstraction for computing
Measure “compressiblility” of activity Measure magnitude of activity
Reduced scale simulations are qualitatively different than realistic scales
0 0.5 1 1.5 2 2.5
x 104
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Neuron Number
Tim
e s
tep
0 0.5 1 1.5 2 2.5
x 104
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Neuron Number
Tim
e s
tep
25,000 Granule Cells 300,000 Granule Cells
Neurogenesis is critical for encoding and separation, but depends on scale
0 50 100 150 200 250 30010
-3
10-2
10-1
100
Number of GCs
Fir
ing
Ra
te (
Hz)
0 50 100 150 200 250 3000.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Number of GCs
Co
mp
res
sib
ilit
y
0 0.5 1 1.5 2 2.5
x 104
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Neuron Number
Tim
e s
tep
0 0.5 1 1.5 2 2.5
x 104
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Neuron Number
Tim
e s
tep
25,000 Granule Cells 300,000 Granule Cells
More
information
encoded
Improved
pattern
separation
Can use formal UQ / SA techniques to inform path to abstraction
Important*: Scale is critical
Neurogenesis is necessary form of plasticity to encode novel information
Less important*: Interneuron diversity
Specific neuron dynamics
* For these specific functions
Carlson, in preparation
Abstracted concepts can become computing-friendly
Formalize dentate gyrus /
neurogenesis sparse
coding algorithm
Severa et al., submitted
Identify critical aspects
of computation
(expansive scale;
sparse activation;
neurogenesis)
Simulate at high
level of neural
fidelity
Can we systematically use this approach?
Rothganger et al., Frontiers in Neural Circuits 2014
N2A model framework and
software
• compositionally construct
dynamical neural models
• compiles simulation codes
• potential “front-end” to neural
hardware
Can we systematically use this approach?
Rothganger et al., Frontiers in Neural Circuits 2014
Thanks!