premotor cortex, action control, and...
TRANSCRIPT
Premotor cortex, action control, and language
Arthur M. Glenberg
University of Wisconsin-Madison
Ago, ergo Cogito
Action control is amazinghitting a tennis ball playing piano walking down steps
Ago, ergo Cogito
hitting a tennis ball, playing piano, walking down steps
coordination among muscles - hierarchical controlover serial order
continuous recalibration
exquisitely precise timingexquisitely precise timing
slow sensory feedback and irrelevant feedback
Evolution and the brain have done a wonderful job solving a set ofvery tricky problems.
Might those solutions be used to solve other tricky problems?
Today’s talk
Ago, ergo CogitoA solution to action control problems: controller/predictor models of action control, Wolpert’s HMOSAIC t ti i tHMOSAIC as a starting point.
Relating HMOSAIC to the brain: Canonical and Mirror Neurons
A marvelous confluence: The HMOSAIC model of action control,with minimal changes, can be a model of language learning,
d ti d h iproduction, and comprehension.
Data from several research projects testing the claim that languagp j g g gcomprehension is intimately related to action control.
Wolpert’s MOSAIC
feedbackAgo, ergo Cogito
lift 150 g
socialenvironmentpredictorlift 500 g
lift 350 g
GainControl
Predictions &probabilities
environment
body/perceptual
predictorsensory/envpredictor
perceptualstates
h i l
controllercontroller +
physicalenvironment
efference copypypredictor
faster than feedbackselects relevant feedback (attention)
feedback & errorlearning predictor selects relevant feedback (attention)
helps learning controllerenhances perception
glearning controllerlearning responsibilitie
drink
feedback: move eyesfeedback: extendHMOSAIC
Ago, ergo Cogitosocialenvironment
sensory/envpredictor
drink
Gain<post<post.
body/perceptualcontroller
predictor
Predictions &probabilities
GainControleyes,…>
i
extend,…>
perceptualstates
controller <prioreyes,…><priorextend,…>
physicalenvironment
+efference copyeyesgraspextend
efference copy
t d grasp lifteyes extendbasic motor repertoire; motor cortex:
feedback
HMOSAIC and language
Ago, ergo Cogito
i ldi t
drink
feedback
socialenvironment
predictor
Predictions &
GainControl
body/perceptual
states
controller
Predictions &probabilities
physicalenvironment
+grasp
efference copy
canonical neuronsmirror neurons
canonical neurons
Ago, ergo Cogitomirror neurons
purposeaction recognitionintent/theory of mindenhance perceptionenhance perceptioncommunication?
Human datafMRI, TMS,
behaviorBroca’s & handBroca s & handBuccino, Tettamanti,Aziz-Zadeh, more
Fadiga et al., speech MNFrom, Rizzolatti & Arbib, 1998
feedback
HMOASIC and languageBroca’s area: speech and hand; Fadiga et al. speech MN
Ago, ergo Cogito
i ldi t
drink
feedback
/ “drink”
socialenvironment
predictor
Predictions &
GainControl
speech
body/perceptual
states
controller
Predictions &probabilities
hBroca’s area
physical
speech
bi directional assoc
control over serialorder in speech
environment
+grasp /dr/bi-directional assocof speech and actio
gesture
efference copy
canonical neuronsmirror neurons
grounding of meaning
feedbackLearning nouns
socialenvironment
predictor
bottle / “bottle”Ago, ergo Cogito
environment
body/
speechPredictions &probabilities
GainControl
body/perceptual
states
controller
speech
p
Assume child knows how to interact with a
physicalenvironment
how to interact with a bottle: move eyes, grasp
Joint attention: eyes+ Speech mirror neuron
graspeyes
Canonical neurons
efference copyphonology & meaning ->
a construction
feedback
Learning verbs
Ago, ergo Cogito
i ldi t
drink / “drink”
socialenvironment
predictor
speechPredictions &
GainControl
body/perceptual
states
controller
h
Predictions &probabilities
physical
speech
Assume childknows how to drinkenvironment
+Auditory mirror&
grasp /dr/
knows how to drink
efference copy
&Action mirror
Dad says, “Give Dada the cup.”
feedback
Ago, ergo Cogito
di t
give cup / “give”social
predictor
speechPredictions &
GainControl
environment
controller
h
probabilities body/perceptual
statesAssume child knows
“cup” ->
speech
physicalenvironment
how to give a cup
peyesgrasp
Action
releaseextendgrasp +Action
efference copy
give5/“give”i 4/“ i ” 3 “Give Lavinnia the candy ”
5 “Give Dada the spoon”4 “Give Dada the candy”
In other words: Behavior consistent with having learned an abstract, double-object construction emerges from integrating (by responsibilities)give4/“give”
give3/“give”give2/“give”
Ago, ergo Cogito2 “Give Momma the bottle.”3 Give Lavinnia the candy.
1 “Give Dada the cup.”feedback
emerges from integrating (by responsibilities) actions learned for specific, concrete situations. Syntax (and compositionality) emerges from
ti t lgive2/ give
give1/“give”Learning speech/action paramepredictor
feedback
socialenvironment
action control.
Attend with eyes to speaker
Attend with eyes to object,controller
speech Predictions &probabilities
GainControl body/
perceptualstates
Attend with eyes to speaker
Attend with eyes to object, set kinematics, graspspeech
states
physicalenvironment
+
Now “Give Dada the bottle”
y pspeaker
object
graspExtend arm to speaker
+
controller
predictorspeech
bottle / “bottle”
Now, Give Dada the bottlespeaker
act
controllerspeech
graspeyes
The Data!
Ago, ergo CogitoLanguage acquisitionAngrave & Glenberg (2007)
Action systems affect language comprehensionGlenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg & Cattaneo (in preparation)
Language comprehension affects action systemsBorghi, Glenberg, & Scorolli (in preparation)
A & Gl b (2007)
Does the child know about bottles before learning “bottle?”
Ago, ergo Cogito
Angrave & Glenberg (2007)Analysis of MacArthur-Bates CDI for gesture & speech
• A01 armshow Look • C14 pushtoycarPush• A01. armshow Look • A02. reachout Look • A03. points See
A09 requestarm Get
• C14. pushtoycarPush • C15. throwball Throw • E01. sweep Sweep
E02 keyindoor Put• A09. requestarm Get • A12. Shrugsallgone Finish • B02. pattycake Clap
B05 sing Sing
• E02. keyindoor Put • E03. Poundhammer Hit • E06. pretendread Read
E09 pla m sinstr Pla• B05. sing Sing • B06. dance Dance • C01. Eatspoon Eat
C02 d i k D i k
• E09. playmusinstr Play • E10. pretenddrive Drive • E13. writepen Write
E15 t l P t• C02. drinkcup Drink • C05. wipeownface Clean
• E15. putonglasses Put
28
A12Verb onset month = .61 x Action onset month + 17 monthsp < .001
26E01 E02
E13E15
pwrite
mon
th
24
tesnhtnom
A01A02
A09B02
B05
B06C05
C14C15 E03
E10clap throw
sweep
rb o
nset
m
22
breVO
A03
E06E09
drinkread
Ve
20
C01
C02 read
4 6 8 10 12 14 16Action Onset monthAction onset month
feedback
Why the lag between comprehension and production?
Ago, ergo Cogito
i ldi t
drink / “drink”
socialenvironment
predictor
speechPredictions &
GainControl
body/perceptual
states
controller
h
Predictions &probabilities
physical
speech
environment
+grasp /dr/
efference copy
Controller and predictor for Art’s knees
Ago, ergo CogitoFatigue muscles systemsfeedback
Bean task!
socialenvironmentpredictor
step
G i
/“step”
environment
body/perceptual
statescontroller
Predictions &probabilities
GainControl
speech
states
physicalenvironment
+step /st/
speech
efference copy
+step /st/
efference copy
2500
Con Sent PNCon Sent Tow (Art gave you the pen)You see the coins in the fountain
John see the coins in his hand.
2250
mse
c)
Con Sent Far
Abs Sent Away (You told Liz the story)
Abs Sent Tow (Liz told you the story)
Con Sent Away (You gave Art the pen)You see the coins in the fountain.
2000
men
t Tim
e (m
y ( y)
1750
Judg
m
Con Sent YouCon Sent Near
You see the coins in your hand.You see the coins in your hand.
1500
12501250 Tow Away Tow Away Tow Away Tow Away Tow Away
beans lexical decision beans beans beansExp 1 Exp 2 Exp 3 Exp 3 Exp 3
Conclusions
In designing primate action systems, evolution figured Ago, ergo Cogito
g g p y , gout solutions to difficult problems of timing, learning, serial order, and hierarchical control.
Many of those same problems occur in language, and it appearsthat evolution adapted mechanisms of action control, in Broca’s at lto language.
Data supporting these claims come frompp glanguage acquisitionaffects of action systems on language comprehensionaffects of language comprehension on use of action systemaffects of language comprehension on use of action system
Language can be used as a tool because linguistic meaning i d t i d b d t d t b dil ti i th ldis determined by and connected to bodily action in the world.
Bologna collaborators: Anna Borghi, Claudia Scorolli, Annalisa SettiLeipzig collaborators: Shirley-Ann Rüschemeyer, Wolfgang Prinz, Simone Bosbach
Many Thanks to:
The Wisconsin Laboratory for Embodied Cognition:
Ago, ergo Cogito
Parma collaborators: Mark Sato, Luigi Cattaneo, Daniele Palumbo, Lucia Riggio, Givoanni Buccino, Vittorio Gallese, Giacomo Rizzolatti
The Wisconsin Laboratory for Embodied Cognition:Doctoral Students: Post-doc:
Megan Brown Lawrence AngraveDavid HavasDavid Havas
Erica Kesin RichmondLisa Lindeman
Undergraduate Students:gJenna Dittmar Laura KleinTiana Gutierrez Stephen MlinazLauren Hazen Josh MoonMaia Jacobs Emily MouilsoBeth Jaworski Michal Rischall Scott Johnson
Funding: National Science Foundation, University of Wisconsin, Max Plank Institute for Human Cognitive and Brain Sciences, & University of Parma, Department of Neuroscience
Iacoboni (2005)
Ago, ergo Cogito
= predictor= controller controller
lenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (2007)
Ago, ergo Cogito
Trans-cranial magnetic stimulation (TMS)
TMS DesignTransfer vs No-Transfer vs NonsenseConcrete vs AbstractPulse at Verb or End
record MEP from OPrecord MEP from OPNB: NO ARM MOVEMENT FOR RESPONSE
Ago, ergo Cogito
0.4 TransferNo Trans
0.2
0
-0.2
Concrete Abstract
-0.4Verb End Verb End
Pulse LocationPulse timePulse time
Language comprehension affects motor systems
Hamilton, Wolpert, & Frith (2004) “Your own action
Ago, ergo Cogito
influences how you perceive another person’s action”lift 15 kg
lift 150 gfeedback from vision
lift 150 g
Predictions &
socialenvironmentpredictor
/lift 500 g
lift 350 g
GainPredictions &probabilities
body/perceptual
statescontroller
sensory/envpredictor
+
GainControl
states
physicalenvironment
controller +
efference copy
Borghi, Glenberg, & Scorolli (in preparation)
Videos from Bosbach, Cole, Prinz, & Knoblich
Ago, ergo CogitoLarge box weights:3 6 12 18 kg
, ,(2005)
3, 6, 12, 18 kgSmall box weights.05, .3, .6, .9 kg
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.Heavy lift: Move the bowling ball from the cart to the tablefrom the cart to the table.Light lift : Move the wine glass from the cart to the table.Descriptive: The wine glass is on thetable.Is the object on the table breakable?
Ago, ergo Cogito
1on
0.75
ion
Hea Video
Cor
rela
tio 0.5
Cor
rela
ti
Light Video
Heavy Video
C
0.25
lift lift
0Descriptive Heavy Light
lift liftSentence Type
Rüschemeyer, Glenberg, Kaschak, & Friederici
Ago, ergo CogitoTable 1Examples of the sentences with English translations
Towards Away StationaryDas Auto fährt auf Dich zu Das Auto Fährt von Dir weg Das Auto sieht großDas Auto fährt auf Dich zu.Das Auto Fährt von Dir weg. Das Auto sieht großThe car drives towards you. The car drives away from you. The car looks bigDer Pfeil fliegt auf Dich zu. Der Pfeil fliegt von Dir weg. Der Pfeil sieht klien aus.Th fli t d Th fli f Th l kThe arrow flies towards you. The arrow flies away from you. The arrow looks Der Ball rollt auf Dich zu. Der Ball rollt von Dir weg. Der Ball sieht groß aus.The ball rolls towards you. The ball rolls away from you. The ball looks big.
Ago, ergo Cogito
“Give momma the bottle.”“Give momma the bottle.”Dad says,
Ago, ergo Cogito
di t
give / “give”feedback
socialpredictor
speechPredictions &
GainControl
environment
“momma” ->eyes
controller
h
probabilities body/perceptual
states
“bottle” ->
reachspeech
physicalenvironment
eyesgraspreleaseextendgrasp
eyes to mommaeyes to bottlegain control gain control+
gain controlact!
efference copyreleaseextendgrasp
Language comprehension & simulation“The girl gives the horse an apple.”
give3/“give”give2/“give”
Ago, ergo Cogito
speech activates motor systemgive1 /“give”
feedback
p y
gain control set
“give”-> speech MN & controller
predictor
speech
give1 / give
P di ti &Gain
Control
socialenvironment
give -> speech MN & controllerpredictor generates
expectations of a recipiencontroller
h
speech Predictions &probabilities
Controlbody/
perceptualstates
speech
physicalenvironment+
lh
“the horse”-> speech MN & conpredictor generates
sensory feedback of horse
Efference copy -> A mental modelwith perspective
applehorse sensory feedback of horse
with perspectiveEmergence of working memory
eyes grasp
“Transfer X the Y.”
Ago, ergo Cogito
Learning kinematicspredictorhand/“hand”
feedback
socialenvironment g
Hand X the Y. (hand priors)Th X th Y (th i )controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual
Throw X the Y. (throw priors)controller
speech
perceptualstates
physicalenvironment
Result: Easily learned action schemas for verbs of concrete
gainlocate X
locate Y
environment+
schemas for verbs of concrete transfer.
locate Y
kinematics
gain Syntactic/semantic bootstrappingusing the motor system!
“Transfer X the Y.”
Ago, ergo Cogito
Learning kinematicspredictortell/“tell”
feedback
socialenvironment g
Tell X the Y. (tell priors)Radio X the Y (radio priors)controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual Radio X the Y.(radio priors)
Delegate X the Y. (delegate prioetc.
controller
speechstates
physicalenvironment
Results: Easily learned action schema for
gainlocate X
locate Y
+
Easily learned action schema for verbs of abstract transfer.
locate Y
kinematics
gain
1400Ago, ergo Cogito
1300
1400No Transfer
Away Toward
ACE interaction
1200
1300 ACE interaction& main effect of C t1200 Concreteness
1100
Concrete Abstract
Motor system forunderstanding
or1000
Response Location
Yes-near Yes-farYes-near Yes-farMotor imagery &responding
Response Location
Ago, ergo Cogito
The Plan:
Replicate Glenberg & Kaschak using Italian
Use TMS to determine if action effect is during understanding or responding.
Collaborators: Marc Sato Luigi Cattaneo LuciaCollaborators: Marc Sato, Luigi Cattaneo, Lucia Riggio, Daniele Palumbo, and Giovanni Buccino
Dipartimento di Neuroscienze, Università di Parma
Triad of Concrete sentences
Toward: Marco/ ti/ da' /le carte. (Marco gives you the papers.)
Ago, ergo CogitoAway: Tu/ dai /le carte/ a Marco. (You give the papers to Marco.)
No Transfer: Tu/ leggi/ le carte/ con Marco. (You read the papers with Marco.)
Triad of Abstract sentences
Toward: Anna/ ti/ delega/ le responsabilità. (Anna delegates the responsibilities to
you )you.)
Away: Tu/ deleghi/ le responsabilità/ ad Anna. (You delegate the responsibilities
to Anna.)
No Transfer: Tu/ discuti/ delle responsabilità/ con Anna. (You discuss the
responsibilities with Anna.)
Nonsense Sentences
Giovanni/ canta/ le carte/ con te. (Giovanni sings the papers with you.)
Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)
Note: The slashes indicate the segments used in Experiment 2.
Simulate temporal course of giving (away) vs receiving
t fi t l tAgo, ergo Cogito
at first laterAway: grasp open
Toward: open grasp
Pulse at EndPulse at Verb
AbstractConcreteAbstractConcrete0 5
0.75
p g p
AbstractConcreteAbstractConcrete
0.25
0.5
0
0 5
-0.25
-0.5Toward Away No Trans Toward Away No Trans Toward AwayNo Trans Toward AwayNo Trans
Sentence Direction
Action and the embodiment of concrete language
Ago, ergo CogitoHauk, Johnsrude, & Pulvermüller (2004)
Tettamanti et al (2005)Tettamanti et al. (2005)
Aziz-Zadeh et al. (2006)
What about the embodiment of abstract language?Space:Space:Richardson, Spivey, Barsalou, & McRae (2003) Boroditsky and Ramscar (2002)
Action: Gl b & K h k (2002)Glenberg & Kaschak (2002)
Other Predictions
Ago, ergo Cogito
eye movements and language
development of gesture & languagep g & g g
signed languages
eye movements and languageTanenhaus visual world paradigm
Ago, ergo Cogito
Tanenhaus visual world paradigmAltmann & Kamide
Richardson & colleagues
P dd t l (i )Ago, ergo Cogito
Padden et al. (in press)
In Al-Sayyid Bedouin Sign Language (ABSL), signs for both yy g g g ( ) gconcrete and abstract verbs of transfer involve arm movemeof away from the body.
( )
Other signed languages?
Karen Emmorey (personal communication) “it is true for ASL and probably for many signed languages”
Conclusions
Ago, ergo CogitoThe data strongly imply that concrete language (about
transfer) is grounded in the motor system
The data suggest that abstract language (about transfer)is also grounded in the motor systemis also grounded in the motor system
Models of language built on models of motor controlModels of language built on models of motor controlprovide a unique understanding of languagephonology, syntax, and semantics as arising f hi hi l h i f t t lfrom hierarchical mechanisms of motor control
Why might abstract language understanding involvemotor system? A developmental hypothesis:
Ago, ergo Cogitolearning nounsattention (move eyes); affordances
i l ?
learning verbs of transfer: parameters of actionsource object kinematics (force direction mode)
canonical neurons?
source, object, kinematics (force, direction, mode)mirror neurons?
generalization of parametersgeneralization of parameterssource, objects, kinematics --> transfer schema
syntactic/sematic boot-strapping using the motor system!y pp g g yYou tell Liz the story.
“Give me the cup.”“Give momma the bottle.”“Give Lavinnia the candy ”
Ago, ergo Cogito
Give Lavinnia the candy.
Learning action parameters:predictor
give/“give”
feedback
socialenvironment g p
Gain control (move only eyes)controller
predictor
speech Predictions &probabilities
GainControl
environment
body/perceptual
Attend with eyes to destination{momma, me, Lavinnia}
controller
speech
perceptualstates
physicalenvironment
gainlocate X
locate Y
Attend with eyes, set kinematic{bottle, cup, candy}
environment+
locate Y
kinematics
gainGain control (ready to move)
Result: A double object action schema (verb argument construction) for giveAct Now!
Linda Smith (2005) “Action alters shape categories”
Ago, ergo Cogito
Action
0.9
1
0 5
0.6
0.7
0.8
Sy
0.2
0.3
0.4
0.5As
0
0.1
S1 S2 S3 A1 A2 A3
Similarity of Test Object to Exemplar
Thus, perceived similarity, categorization, and namingare influenced by action.
Glenberg & Kaschak 2002
Action systems affect language comprehension
Ago, ergo Cogito
Glenberg & Kaschak, 2002
Task: Judge if a sentence is sensibleIV 1: Implied Direction: Toward vs. AwayIV 2: Sentence type
ImperativeOpen the drawer. / Close the drawer.
Concrete TransferConcrete TransferCourtney handed you the notebook. / You handed Courtney the
notebook.Andy delivered the pizza to you. / You delivered the Pizza to Andy.
Ab t t T f tAbstract Transfer sentences Liz told you the story. / You told Liz the story.The policeman radioed the message to you. / You radioed the
message to the policeman.g pNonsense
Open the plate./You drank the house to Joe.
IV 3: Actual Response Direction: Yes-is-near vs Yes-is-IV 3: Actual Response Direction: Yes is near vs. Yes isfar
e
1400Imperative
1850
Abstract Transfer
Concrete Transfer
1850A
n R
eadi
ng T
ime
13501800
1750
1800
AwayToward
AwayToward
AwayToward
Tri
mm
ed M
ean
13001700
17501750
1700
Toward
You told Liz the stor
T
1250Yes-is-near Yes-is-far
1650Yes-is-near Yes-is-far
1650Yes-is-near Yes-is-far
Response Direction
Is the effect caused by comprehension processes or response pro