general mechanisms of neocortical memory jeff hawkins director redwood neuroscience institute june...
DESCRIPTION
“I conclude that cytoarchitectural difference between areas of neocortex reflect differences in their patterns of extrinsic connections. The traditional or usual ‘functions’ of different areas also reflect these differences in extrinsic connections. They provide no evidence whatsoever for differences in intrinsic structure or function..” “Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.” Vernon Mountcastle, 1978TRANSCRIPT
General mechanisms of Neocortical memory
Jeff HawkinsDirectorRedwood Neuroscience InstituteJune 12, 2003 MIT
Outline
Top down analysis:nature of problem and solutionrepresentationtime and prediction
Bottom up example:auditory memory task
- deduce necessary algorithms- unique map to anatomy
“I conclude that cytoarchitectural difference between areas of neocortex reflect differences in their patterns of extrinsic connections. The traditional or usual ‘functions’ of different areas also reflect these differences in extrinsic connections. They provide no evidence whatsoever for differences in intrinsic structure or function..”
“Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.”
Vernon Mountcastle, 1978
motor touch audition vision
spatiallyspecific
spatiallyinvariant
temporallyspecific (fast)
temporallyinvariant
Neocortical connectivity
motor touch audition vision
spatiallyspecific
spatiallyinvariant
temporallyspecific (fast)
temporallyinvariant
motor touch audition vision
spatiallyspecific
spatiallyinvariant
temporallyspecific (fast)
temporallyinvariant
motor touch audition vision
spatiallyspecific
spatiallyinvariant
temporallySpecific (fast)
temporallyinvariant
Prediction(spatially and temporally specific)
MacKay, Mumford, Softky, Rao & Ballard
motor touch audition vision
spatiallyspecific
spatiallyinvariant
temporallyfast
temporallyinvariant
Prediction(spatially and temporally specific)
Q1. Why make predictions?Q2. How do we make predictions?Q3. How do we form invariant representations?
Q1. Why make predictions
Non-mammalianbrain
Sophisticatedsenses
Complexbehavior
Posterior Neocortex: sensory prediction
Predictions allow brain to react prior to events, to “see” into the future.
Sophisticatedsenses
Complexbehavior
Mammalianposterior neocortex
Anterior Neocortex: motor sequences
Sophisticatedsenses
Complexbehavior
Mammalianposterior neocortex
Humananterior neocortex
Q2. How do we make predictions?
- Store sequence of patterns: allows prediction of future events- Invariant representations cannot make specific predictions
invariantrepresentations
specificafferents
… … …
time
Q2. How do you make predictions?
- Store sequence of patterns: allows prediction of future events- Invariant representations cannot make specific predictions- invariant prediction + input[t-1] = specific prediction[t]
invariantrepresentations
specificafferents
… … …
+
time
Q3. How do we form invariant representations?
Spatially invariant representations require- convergence of features that constitute object- divergence to unite objects that although different represent the same thing
(x1⋂x2⋂x3 …) ⋃ (x4⋂x5⋂x6 …) ⋃ (x7⋂x8⋂x9 …) …
Top down summary
Every cortical region:- Forms representations by convergence of features- Forms invariant representations by divergence- Stores and recalls sequences of invariant representations
sequence memory- Recalls pattern sequences auto-associatively- Combines recalled patterns with input to:
make predictions of sensory afferentsdrive motor efferents
Top down summary
Every cortical region:- Forms representations by convergence of features L4,
Thalamus
- Forms invariant representations by divergence L2,3 horiz
- Stores and recalls sequences of invariant representations L1,2,3sequence memory
- Recalls pattern sequences auto-associatively- Combines recalled patterns with input to: L5,6
make predictions of sensory afferentsdrive motor efferents
Bottom up example:
Auditory memory (melodies)- Representations are invariant to pitch
recognized and recalled in any pitch- Stored as sequences of associated patterns
have repeated elements (ggge- fffd ggge- aaag)
each note has a stored duration- Prediction: we “hear” notes prior to occurrence- Hierarchical representation, e.g. AABA structure
(temporal invariance/reduction)
A1
L freq H
Thalamus
C D E F G A B C1 D1 E1 A1
A2C-C’ D-D’ E-E’ F-F’ G-G’A-A’ B-B’
octave
(x1⋂x2⋂x3 …) ⋃ (x4⋂x5⋂x6 …) ⋃ (x7⋂x8⋂x9 …) …(C⋂C’) ⋃ (D⋂D’) ⋃ (E⋂E’) …
frequency
intervals
Pitch invariance = interval representation
A2
L freq H
A1
L freq H
Thalamus
A2
L freq H
A1
L freq H
L
H
Thalamus
Intersecting inputs in layer 4define all possible intervals
A2
L freq H
A1
L freq H
L
H
Thalamus
Iso-interval bandsupdown
A2
L freq H
A1
L freq H
L
H
Thalamus
Freq invariant interval bandsupdown
L2,3L4
- Intersecting inputs to L4- Spread of activation in L2,3
How do we store the sequence of interval activations?How do we represent unique intervals in unique songs?
GGGE- FFFD GGGE- AAAG
How do we store and recall the precise time duration ofeach unique interval?
L2,3
L1
L4
L5
L6
Layer 2,3 cellsDense and smallHigh local mutual excitationHigh local mutual inhibitionLong distance excitatory coll.Dendrites in L1Axon synapses in L5
L2,3
L1
L4
L5
L6
Layer 2,3 is sparsely activeMutual excitation drives allStrong inhibition prevents most
cells from firingLayer1 plays role in deciding
who is active
L2,3
L1
L4
L5
L6
Layer 1 is context1. Context from higher areas2. Local context from L2,33. Input from matrix thalamus
(time)
L2,3
L1
L4
L5
L6
Layer 1context
Layer 2,3unique representations of
freq invariant intervals
There is a unique sparse L2,3 activation pattern for each instance of this interval ever learned. Each unique pattern represents a particular interval in a particular melody.
Layer 4Freq specific intervals
Converging inputs form object representations
L freq HL
H
Layer 2,3Freq invariant intervals
Horizontal connections joinobjects to form spatially invariant representations
Layer 1State: time & location
L1 axons link representations in sequence.Unique representations link to unique representations
Song is represented as a sequence of freq invariant interval bands. Each invariant interval has a unique representation and is associatively linked to its predecessor.
Representing “class” and “individuality”
Activation area defines object class
Unique activation pattern defines individual object
How do we store and recall the precise time duration ofeach unique interval?- Actual duration vs. relative duration (actual)- Duration must be stored in-situ with interval
Proposal …- Matrix thalamic nuclei emits a clock pattern to L1- Part of L1 changes on each clock tick- L5 cell resets clock on L4 transition or L1 match
L2,3
L1
L4
L5
L6
New input arrives at L4, causes L5 cell to burst, inhibition shuts down L4L5 burst teaches L5 cell to fire when exact pattern in L1 is seen in futureL5 burst also sets matrix thalamic nuclei to a deterministic state (resets clock) causing
interval state transitionL5 cells encode duration of a particular state (note in song): when the elapsed time of
a particular state occurs, they burst fire
MatrixThalamus
How do you predict next note in proper key?
invariant prediction + input[t-1] = specific prediction[t]
invariantrepresentations
specificafferents
… … …
+
time
L2,3
L1
L4
L6a
L6b
A1(t-1)Th(t)
freq
Pattern from A1
L2,3
L1
L4
L6a
L6b
A1(t-1)Th(t)
freq
Th(t)
freq
Pattern from A1
Simple interval
L2,3
L1
L4
L6a
L6b
A1(t-1)Th(t)
freq
Th(t)
freq
Pattern from A1
Simple interval
Invariant unique interval
L2,3
L1
L4
L6a
L6b
A1(t-1)Th(t)
freq
Th(t)
freq
Pattern from A1
Simple interval
Invariant unique interval
Associative spread
L2,3
L1
L4
L6a
L6b
A1(t)
freq
freq
Pattern from A1(t)
Predicted next interval
L2,3
L1
L4
L6a
L6b
A1(t)
freq
freq
A1(t) + predicted interval
Predicted next interval
L2,3
L1
L4
L6a
L6b
freq
freq
A1(t) + predicted interval
Predicted next interval
Next predicted noteback to Thalamus
L2,3
L1
L4
L6a
L6b
freq
freq
A1(t) + predicted interval
Predicted next interval
Horizontal projectionsfrom stored previous richpattern to apical dendritesof predicted pattern copiesrich attributes
Hierarchical representation
words / melodies
phrases / songs
sentences
Hierarchical representation
words / melodies
phrases / songs
sentences
ProblemThe number of state transitions
must decrease as you ascend the hierarchy.
However L2,3 projects to upper areas and it changes on every event.
Hierarchical representation
SolutionSome cells in L2,3 learn to be
stable over repeated patterns.
Hierarchical representation
SolutionSome cells in L2,3 learn to be
stable over repeated patterns.
Therefore we should see L2,3 cells that stay active over longer periods of time. Only these cells should project to next higher cortical area.
How generic is this model?
Performs a non-trivial memory processing function- invariant, rich predicting, branching, hierarchical, sequence memory
Aligns well with top down constraints
Accounts for much of known cortical anatomy- involves all layers, excitatory and inhibitory spread- how could other areas of cortex be fundamentally different?
Other cortical areas are likely variations on this themeOther principles are likely in use as well
A2 as I have drawn it A2 as it might appear- limited to octave intervals- appearance of tonotopy
Redrawing A2
Compares input from two ears- inter-aural delay accentuated subcortically- predicts location of sounds in body space
Possible interpretation of A1
Narrowly tuned
Broader tuned, sweep
Broader tuned, sweep
low freq high
Summary
1) Converging L4 inputs define objects2) Horizontal connections in L2,3 create spatially invariant
representations2) Sparse activation in Layers 2,3 encodes unique instances of
invariant representations3) L1 mediates memory of sequences4) L5 thalamo-cortical loops encode duration of events5) Sustained activity in some L2,3 cells establishes basis for
temporal invariance6) L6 cells make specific predictions from L2,3 and afferents
Summary
1) Converging L4 inputs define objects2) Horizontal connections in L2,3 create spatially invariant
representations2) Sparse activation in Layers 2,3 encodes unique instances of
invariant representations3) L1 mediates memory of sequences4) L5 thalamo-cortical loops encode duration of events5) Sustained activity in some L2,3 cells establishes basis for
temporal invariance6) L6 cells make specific predictions from L2,3 and afferents
Testable - buildable - a start
Thank - - -
“It is not that most neurobiologists do not have some general concept of what is going on. The trouble is that the concept is not precisely formulated. Touch it and it crumbles. What is conspicuously lacking is a broad framework of ideas within which to interpret these different approaches.”
Francis Crick 1979
There is “no evidence whatsoever for differences in intrinsic structure or function. This suggests that the necortex is everywhere functionally much more uniform than hitherto supposed and that its avalanching enlargement in mammals and particularly in primates has been accomplished by replication of a basic neural module without the appearance of wholly new neuron types or qualitatively different modes of intrinsic organization.”
“Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.”
Vernon Mountcastle, 1978
All cortical regions