MVPAOpening a new window on
the mind via fMRI
Rebecca Saxe Summer Course
20151
Image removed due to copyright restrictions. Please see the video.
AccidentalIntentional
THINKING ABOUTHow much blame? THOUGHT
1
2
3
4
5
6
7
Dies Fine Dies Fine
© University of Illinois Press. All rights reserved. This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.Source: Heider, F., & Simmel, M. (1944) "An experimental study in apparent behavior."The American Journal of Psychology, 57, 243-259.
This image is in the public domain. Source: Photographs of the IMF 2007 Annual Meetings,International Monetary Fund Photograph by Stephen Jaffe. 2
THEORY OF MINDThe False Belief Task
3
THEORY OF MINDThe False Belief Task
4
THINKING ABOUT THOUGHT%
Sig
nal c
hang
e
-0.5
0.0
0.5
1.0
Time (s)-4-20 2 4 6 8 12 16 20 24 28
PhysicalBelief
A volcano erupted on this Caribbean island three months ago. Barren lava
rock is all the remains. Satellite photos show the island as it was
before the eruption.
The photo shows the island as...
Group average
Anne made lasagna in the blue dish. After Anne left, Ian came home. He
threw out the lasagna and made spaghetti in the blue dish and replaced it back in the fridge.
Anne thinks the blue dish contains...
Individual participants
‐1
‐0.5
0
0.5
1
1 2 3 4 5 6 7 8 9
Considered thoughts
TPJ
beta
-val
ueR
% S
igna
l cha
nge
-0.3
0
0.3
0.6
Time (s) 0 4 8 12 16 20 24 28 32 36
Body Mental Physical
5
THINKING ABOUT THOUGHT%
Sig
nal c
hang
e
-0.5
0.0
0.5
1.0
Time (s)-4-20 2 4 6 8 12 16 20 24 28
Individual participants
Considered thoughts
TPJ
beta
-val
ueR
% S
igna
l cha
nge
-0.3
0
0.3
0.6
Time (s) 0 4 8 12 16 20 24 28 32 36
Body Mental Physical
1
2
3
4
5
6
7
Dies Fine Dies Fine
TMS to RTPJ
Causal role
AccidentalIntentional
6
THINKING ABOUT THOUGHTIndividual participants
y M
uch
None
...
V
er
1
2
3
4
5
6
7
Dies Fine Dies Fine
Causal roleTMS to RTPJ
Hypothesis:RTPJ selectively “involved in” ToM
% S
igna
l cha
nge
-0.5
0.0
0.5
1.0
Time (s)-4-20 2 4 6 8 12 16 20 24 28
AccidentalIntentional
7
FMRI & COGNITIONBeyond “involvement”
Aver
age
BOLD
Activity shows: both stories describe thoughts Theory of Mind: Who thinks what? Why? (i.e. what reasons? what motivations?) With what consequences?
Stor
ies
Albert really wants this ski trip to be a success. Though
the ice looks quite thin at points, Albert thinks the
pond is sufficiently frozen over to support a person’s
weight. He tells his girlfriend to walk out on the ice.
During a trip Grace is irritated by her friend’s
constant whining. Grace sees a container labeled “toxic poison”, so she thinks the
powder is poison. She puts the powder in her friend’s
coffee.
Representations
* inspired by real stimuli
Population codes of features/ dimensions
Computations Transformation
8
Beyond “involvement”FMRI & COGNITION
Traditional analysis “MVPA” analysis
Univariate Multivariate avg magnitude across voxels relative magnitude across voxels
“Forward” / “Encoding” direction “Reverse” / “Decoding” direction Region scale Sub-region scale Stimulus “Type” Within type features
9
Koster-Hale et al (2013)
Your family is over for dinner. You wish to show off your culinary skills. For one of the dishes, adding peanuts will really bring out
the flavor.
You grind up some peanuts, add them to that dish, and serve everyone.
Your cousin, one of your dinner guests, is severely allergic to peanuts.
You had absolutely no idea about your cousin's peanut allergy when you added the
peanuts.
How much blame should you get?
VERSION #1Haxby style correlations
Experiment 1A Methods:
Minimal pair 4s; 2-4 words changed
You knew about your cousin's peanut allergy when you added the peanuts.
* real stimuli
cent
Sig
nal
Chan
gePe
r
-0.5
0.0
0.5
1.0
0 4 8 12 16 20 24
KnowingUnknowing
10
Across
Across
Koster-Hale et al (2013)
You had absolutely no idea about your
cousin's allergy when you added the
peanuts.
You could see that your classmate was standing too close
but you kicked anyway.
Generalize across heterogenous items: (NB every item is unique)
Knowing Unknowing
The essay was typed, so you completely didn’t realize who
had written it.
Based on what the manager said, you
definitely realized the chute was faulty.
With
in
With
in
VERSION #1Haxby style correlations
With
in
With
in
oss
Acr oss
Acr
Even
Odd
11
Koster-Hale et al (2013)
You had absolutely no idea about your
cousin's allergy when you added the
peanuts.
You could see that your classmate was standing too close
but you kicked anyway.
Knowing Unknowing
Experiment a
AcrossWithin
∗
Knowing vs Unknowing Harm
corre
latio
n (Z
-sco
re)
0
.5
1
1.5
Generalize across heterogenous items: (NB every item is unique)
(n=23)
The essay was typed, so you completely didn’t realize who
had written it.
Based on what the manager said, you
definitely realized the chute was faulty.
With
in
With
in
AcrossAcross
VERSION #1Haxby style correlations
12
Koster-Hale et al (2013)
You knew about your cousin's peanut
allergy when you added the peanuts.
Knowing Unknowing
∗
Knowing vs Unknowing Harm
elat
ion
(Z-s
core
) co
rr 0
.5
1
1.5
(n=23)
∗
Experiment b(n=16)
Experiment c
∗
(n=14)Experiment 1A
LTPJ
elat
ion
(Z-s
core
) co
rr 0
.5
1
1.5
PC DMPFC(n=39) (n=39) (n=39)
Experiment a
AcrossWithin
You had absolutely no idea about your
cousin's allergy when you added the
peanuts.
VERSION #1Haxby style correlations
Same distinction, new implementation
The container is labeled “sugar”, so Grace believes that the white powder is
regular sugar.
The container is labeled “toxic”, so Grace believes that
the white powder is a toxic substance.
Experiment 1B&C
True Belief False Belief
*old data
13
Koster-Hale et al (2013)
You knew about your cousin's peanut
allergy when you added the peanuts.
Knowing Unknowing
∗
corre
latio
n (Z
-sco
re)
0
.5
1
1.5
(n=23)
∗
Experiment b(n=16)
Experiment c
∗
(n=14)
n (Z
)RT
PJ P
atte
r
-0.8
-0.4
0
0.4
0.8
Intentional > Accidental (Z)
0 1 2
R² = 0.361
Experiment 1A
The container is labeled “sugar”, so Grace believes that the white powder is
regular sugar.
The container is labeled “toxic”, so Grace believes that
the white powder is a toxic substance.
Experiment 1B&C
Knowing vs Unknowing
Experiment a
AcrossWithin
You had absolutely no idea about your
cousin's allergy when you added the
peanuts.
VERSION #1Haxby style correlations
One measurement per individual
True Belief False Belief
14
FMRI & COGNITIONBeyond “involvement”
Haxby-style correlations: - robust but simple measure - sensitive to minimal manipulation - generalises across heterogenous stimuli - stable in participant (relates to ID) - different across regions
15
Koster-Hale, Bedny, Saxe
Justbeforeheleaveshishouse,Quentinhearsamessagefromhismotheronhisphone.Themessagesaysthatshehas
badnewstotellhim.HeWesleyseeshimselfinamirr
Aftertheinterview,
seesstthaainthisdowshirtnthe
hafrsontabigcoff
or.
.ee
Whhis@iaenh
ncéestegetst
aondingwtherestaurant,Ericsees
Herfacelooksveryithherpa
happy.rents.
Whilecleaningoutherdormroom,Abigailhearsfootstepscomingdownthehallway.Thefootstepssoundlikeher
belovedboyfriend’s.
VERSION #1AA few more Haxby style correlations
Two orthogonal differences:
n=13; 24 stories/3 conditions, counterbalanced 16
Koster-Hale, Bedny, Saxe
Seeing vs Hearing
VERSION #1AA few more Haxby style correlations
ela
tion
ed
corr
Z-s
cor
Two orthogonal differences:
17
hearsamessagefrJustbeforeheleaves
omhismotheronhishishouse,Quentin
phone.Themessagesabadnewstotellhim.
ysthatshehasHeWesleyseeshimselfinamirr
Aftertheinterview,
seesstthaainthisdowshirtnthe
hafrsontabigcoff
or.
.ee
Whhis@iaenh
ncéestegetst
aondingwtherestaurant,Ericsees
Herfacelooksveryithherpa
happy.rents. Abig
Whilecleaningoutherdormrailhearsfootstepscomingdo
oom,
hallway.Thefbelov
ootedbostepssoundly
ikeherwnthe
friend’s.
VERSION #1AA few more Haxby style correlations
Two orthogonal differences: Negative
PositiveKoster-Hale, Bedny, Saxe n=13; 24 stories/3 conditions, counterbalanced 18
Koster-Hale, Bedny, Saxe
Seeing vs Hearingela
tion
ed
corr
Z-s
cor
Good vs Bad
VERSION #1AA few more Haxby style correlations
Two orthogonal differences:
19
FMRI & COGNITIONBeyond “involvement”
Haxby-style correlations: - robust but simple measure - sensitive to minimal manipulation - generalises across heterogenous stimuli - stable in participant (relates to ID) - different across regions - multiple orthogonal distinctions
BUT - binary, no info about ‘why’
20
FMRI & COGNITIONBeyond “involvement”
More general idea: Response pattern -> Vector -> Point in voxel space
- Train classification - typically linear
- Independent test trials
21
VERSION #2Classifying single trials
More general idea: Response pattern -> Vector -> Point in voxel space
- Train classification - typically linear
- Independent test trials
DV: classification accuracy
22
VERSION #2
Bella poured the sleeping potion into Ardwin's soup and went into the next room, where her sister, Jen, was waiting.
They held their breaths while Ardwin started to eat.
Bella stared through the
secret peep hole and waited. In the bright light,
Bella saw his eyes close and his head droop.
Bella tried to peer through a crack in the door. In the very dim light, Bella
squinted to see his eyes close.
Bella grinned from ear to ear. "The potion worked!" she exclaimed.
Bella pressed her ear against the
door and waited. In the sudden
quiet, Bella heard the spoon drop
and a soft snore.
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality
Classifying single trials
Not binary:
Modality Quality
23
VERSION #2
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality
Classifying single trials
Not binary Not redundant
Bella poured the sleeping potion into Ardwin's soup and went into the next room, where her sister, Jen, was waiting.
They held their breaths while Ardwin started to eat.
Bella stared through the
secret peep hole and waited. In the bright light,
Bella saw his eyes close and his head droop.
Bella tried to peer through a crack in the door. In the very dim light, Bella
squinted to see his eyes close.
Bella grinned from ear to ear. "The potion worked!" she exclaimed.
Bella pressed her ear against the
door and waited. In the sudden
quiet, Bella heard the spoon drop
and a soft snore.
Modality Quality
24
VERSION #2
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality
Classifying single trials
Not binary Not redundant
0.5
0.55
0.6
0.65
Modality Quality
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality
Distinct information across regions
0.5
0.55
0.6
0.65
Modality Quality ValenceKoster-Hale et al submitted25
VERSION #2
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality Valence
Classifying single trials
Not binary Not redundant
0.5
0.55
0.6
0.65
Modality Quality Valence
Clas
sifica
tion
Accu
racy
0.5
0.55
0.6
0.65
Modality Quality Valence
Distinct information across regions
0.5
0.55
0.6
0.65
Modality Quality ValenceKoster-Hale et al submitted26
FMRI & COGNITIONBeyond “involvement”
Classification analyses - sensitive to minimal manipulation - generalises across heterogenous stimuli - different across regions - multiple orthogonal distinctions - item-specific, continuous (not binary)
BUT - tests hypotheses / features sequentially
27
VERSION #3Representational (dis)similarity matrices
After an 18 hour flight, Alice arrived at her vacation destination to learn that her baggage (including necessary camping gear for her trip) hadn't made the flight. After waiting at the airport for 2 nights, Alice was informed that the airline had
lost her luggage altogether and wouldn't provide any compensation.
20 AFCJealous
Disappointed Devastated
Embarrassed Disgusted
Guilty Annoyed
Apprehensive Terrified Furious Lonely
Surprised Nostalgic Content
Impressed Proud
Excited Hopeful Joyful
Grateful
200 unique stories, 80 positive and 120 negative
Sarah swore to her roommates that she would keep her new diet. Later, she was in the kitchen getting a glass of water, and took a bite of a cake she had bought for
their dinner party the following evening. Sarah’s roommates arrived home to find that she had eaten half the cake and broken her diet.
For the months before her marathon, Dianne trained even harder than usual, running extra miles and adding strenuous weight sessions at the gym. Dianne
hoped to shave at least 10 minutes off of her previous best of 3:14. On race day, she came in 23rd in her age group with a new personal record of 2:46.
Brenda was texting while driving. She went through a red light and hit a boy on a bike. She jumped out of the car to see if the boy was okay. He had a couple
scrapes, but, somehow, was otherwise okay. Brenda put away her phone and vowed to never text while driving again.
Skerry and Saxe in prep 28
3. NDE confusion matrix and group neural confusion matrix??
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
i
nten
ded
emot
ion
judged emotion
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Confusion Matrix (Behavioral) Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
DMPFC
ToM Network
A. B. MMPFC
RTPJ ToM Network
65%
Sig. Class
n=22 FWE p<.05, k>25
n=139
Whole brain searchlight
VERSION #3Representational (dis)similarity matrices
Skerry and Saxe in press 29
3. NDE confusion matrix and group neural confusion matrix??
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
i
nten
ded
emot
ion
judged emotion
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Confusion Matrix (Behavioral) Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
DMPFC
ToM Network
A. B. MMPFC
RTPJ ToM Network
65%
n=139
Whole brain searchlight
Sig. Class B>P
Overlap
n=22 FWE p<.05, k>25
Accu
racy
00.050.1
0.150.2
0.25
RTPJ MPFC LTPJ PC
** **
All
**
VERSION #3Representational (dis)similarity matrices
Skerry and Saxe in press 30
3. NDE confusion matrix and group neural confusion matrix??
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
i
nten
ded
emot
ion
judged emotion
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Confusion Matrix (Behavioral) Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
DMPFC
ToM Network
A. B. MMPFC
RTPJ ToM Network
VERSION #3Representational (dis)similarity matrices
Representation
Skerry and Saxe in press
After an 18 hour flight, Alice arrived at her vacation destination to learn that her baggage
(including necessary camping gear for her trip) hadn't made the flight. After waiting at the
airport for 2 nights, Alice was informed that the airline had lost her luggage altogether and
wouldn't provide any compensation.
Event features
Was this situation caused by a person or some other external force?
Was this situation caused by Alice herself? Does the situation refer to something in her past?
Was Alice interacting with people? Did this situation affect her relationships with
other people? …
31
3. NDE confusion matrix and group neural confusion matrix??
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
i
nten
ded
emot
ion
judged emotion
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Confusion Matrix (Behavioral) Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
DMPFC
ToM Network
A. B. MMPFC
RTPJ ToM Network
VERSION #3Representational (dis)similarity matrices
Skerry and Saxe in press
Agen
t Cau
seSe
lf Ca
use
Rela
tions
hip
38 appraisal features
Past
Inte
ract
ing
32
3. NDE confusion matrix and group neural confusion matrix??
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
i
nten
ded
emot
ion
judged emotion
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Confusion Matrix (Behavioral) Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Jealous Disappointed
Devastated Embarrassed
Disgusted Guilty
Annoyed Apprehensive
Terrified Furious Lonely
Surprised Nostalgic
Content Impressed
Proud Excited Hopeful
Joyful Grateful
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
Gra
tefu
l Jo
yful
H
opef
ul
Exc
ited
Pro
ud
Impr
esse
d C
onte
nt
Nos
talg
ic
Sur
pris
ed
Lone
ly
Furio
us
Terri
fied
App
rehe
nsiv
e A
nnoy
ed
Gui
lty
Dis
gust
ed
Em
barra
ssed
D
isap
poin
ted
Jeal
ous
DMPFC
ToM Network
A. B. MMPFC
RTPJ ToM Network
VERSION #3Representational (dis)similarity matrices
Skerry and Saxe in press
Representational dissimilarity
Accu
racy
00.250.5
0.751
Observers Event Features
Classification of test stories
33
VERSION #3Representational (dis)similarity matrices
Skerry and Saxe in press
Representational dissimilarity
Accu
racy
00.250.5
0.751
Observers Event Features
Classification of test stories
Kend
all’s
tau
0
0.05
0.1
0.15
Event F Valence Observers
Correlation to neural RDM
34
FMRI & COGNITIONBeyond “involvement”
RDM analyses - parameter free fit - models of different complexity - sensitive to overall “structure” of representation - direct comparison of multiple hypotheses
BUT - less info about specific features
35
Beyond “involvement”FMRI & COGNITION
Traditional analysis “MVPA” analysis
Univariate Multivariate avg magnitude across voxels relative magnitude across voxels
“Forward” / “Encoding” direction “Reverse” / “Decoding” direction Region scale Sub-region scale Stimulus “Type” Within type features
Key problems:Null resultsTheory of concepts
36
Beyond “involvement”FMRI & COGNITION
Traditional analysis “MVPA” analysis
Univariate Multivariate avg magnitude across voxels relative magnitude across voxels
“Forward” / “Encoding” direction “Reverse” / “Decoding” direction Region scale Sub-region scale Stimulus “Type” Within type features
Future ApplicationsConceptual change in childrenCombine with dynamics
37
Funding
THANKS
Packard Foundation John Merck Fellows Program Ellison Medical Foundation Simons Foundation ONR NSF CAREER NIH RO1 DARPA
Martinos Imaging Center Judith Medeiros-Adams and COUHES staff Participants and their families
38
Image removed due to copyright restrictions. Please see the video.
MIT OpenCourseWarehttps://ocw.mit.edu
Resource: Brains, Minds and Machines Summer CourseTomaso Poggio and Gabriel Kreiman
The following may not correspond to a p articular c ourse o n MIT OpenCourseWare, but has beenprovided by the author as an individual learning resource.
For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.