Brain related semantics

CSCTR – Session 8Dana Retová

Rejects the standard view that amodal symbols represent knowledge in semantic memory

Cognition shares the same mechanisms with perception, action and introspection

Simulation◦ A core form of computation in the brain◦ Reenactment of perceptual, motor and introspective

states acquired during experience

◦ As experience occurs, the brain captures the states across modalities and integrates them with a multimodal representation stored in memory

Grounded cognition

Modal representation and imagery representing knowledge◦ Epicurus, Kant, Reid

Behaviorists◦ Imagery not sufficiently scientific

Cognitivists◦ Amodal representation (feature lists, semantic

networks, frames) Elegant and powerful formalisms for representing

knowledge Could be implemented in AI

Mental imagery vs. amodal symbols

No evidence supports the presence of amodal symbols in cognition

Grounding problem◦ Traditional theories fail to explain how cognition

interfaces with perception and action Problem where the brain stores amodal

symbols◦ How is it consistent with neural principles of

computation?

Problem with traditional AI

Simulation Situated action Bodily states

◦ Modal representations are central to knowledge

Grounded cognition theories

Cognitive Linguistics Theories◦ Lakoff & Johnson (1980, 1999)

Abstract concepts are grounded metaphorically in embodied and situated knowledge

Theories of situated action◦ Gibson (1979)

Role of environment in shaping cognitive mechanisms Coupling of perception and action during goal achievement Social interaction

◦ Research in robotics◦ Dynamic systems as preferred architecture

Fixed representations do not exist

Theories of Grounded Cognition

Memory theories◦ Glenberg (1997)

Memory is not just passive storage of information Perception of relevant objects triggers affordances for

action stored in memory Reasoning about future actions relies on remembering

affordances while suppressing perception of the environment

Social simulation theories◦ How we represent the mental states of other people

We use simulations of our own minds To feel someone else’s pain we simulate our own pain Mirror neurons Empathy, imitation, social coordination

Cognitive Simulation Theories

Perceptual Symbol Systems◦ Synthetic approach

Implements standard symbolic functions Type-token binding, inference, productivity, recursion, propositions

◦ A single multimodal representation system in the brain supports diverse forms of simulation across different cognitive processes High-level perception Working memory long-term memory conceptual knowledge

Convergence zone architecture (Damasio 1989, Simmons & Barsalou 2003) Single representation system controlled by multiple simulation

mechanisms

Cognitive Simulation Theories

ensemble of neurons within which many feedforward/feedback loops make contact.

It 1) receives forward projections from cortical regions located in the connectional level immediately below

2) Sends reciprocal backward projections to the originating cortices

3) Sends forward projections to cortical regions in the next connectional level; and

4) Receives projections from heterarchically placed cortices and from subcortical nuclei in thalamus, basal forebrain, and brainstem.

Convergence zone

Perceptual Inference Perception-action coordination Perception of space Memory

◦ Implicit memory◦ Explicit memory◦ Working memory

Conceptual processing

Empirical Evidence for Grounded Theories

Vision and motion◦ Goldstone (1995)

Association between shape and color◦ Hansen et al. (2006)

Object’s natural color distort achromatic perception of the object toward the opponent color

◦ Motion (Freyd 1987, Shiffrar & Freyd 1990,1993) Subjects simulate the visual trajectory beyond its actual

trajectory Also during apparent motion, simulation of possible

action shapes the perception of motion◦ Speech (Warren 1970) : http://www.youtube.com/watch?

v=UlJs24j3i8E Lexical knowledge produces simulation in speech

perception – missing phoneme simulation

Perceptual inference

Simulations of potential actions◦ Viewing an object grasped with a precision or power grip (grape

vs. hammer) produces a simulation of the appropriate action (Tucker & Ellis 1998) This is affected by

object’s orientation (Symes et al. 2007) Size (Glover et al. 2004)

◦ Simulations of both grasping and functional actions (Bub et al 2007)

◦ Also name triggers simulation (Tucker & Ellis 2004)◦ Hearing a word activates the articulatory action associated with

producing it (Pulvermuller 2006)◦ Perceived effort affects visual perception (Proffitt 2006)

Being tired from a run makes a hill look steeper Carrying a heavy pack makes a path look longer

Perception-action coordination

Motor simulations◦ Motor system constructs a feed-forward

simulation of the action to guide and correct it (Grush 2004, Wolpert et al. 1999)

◦ Generating visual inferences about the anticipated actions of perceived agents (Wilson & Knoblich 2005)

Perception-action coordination

The perception of space is shaped by the body and it’s relation to the environment◦ Locating objects has various difficulty along

different axes Vertical

easiest Front-back Left-right

Most difficult – bodily cues are lacking

◦ Perception of near space extends with arm length (Longo & Laurenco 2007)

Perception of space

Results form simulation of perceptual memories

Repetition priming is strongest when the modalities of the memory and stimulus match (e.g. auditory) (Kirsner et al., 1989)

Repetition priming is strongest when perceptual details of the memory and stimulus match (e.g. orientation, size,…) (Jacoby&Hayman, 1987)

Imagining produces repetition priming similar to actual perception (Roediger&Blaxton, 1987)

Implicit memory

Multimodal simulations of previous episodes◦ Important for constructing future events

The retrieval of a word stimulates the modal operations performed at encoding (Wheeler et al. 2000)◦ Visual areas become active during retrieval

following visual study while auditory areas become active following auditory study

Greater activation in modal areas when remembering something that really occurred than false memories (Slotnick & Schacter 2004)

Explicit memory

Absent stimulus is stored in working memory (Levy & GoldmanRakic 2000)◦ To maintain working memory, neurons in the frontal lobes

maintain a simulation of the absent stimulus in the modal system that processed it originally. Some frontal regions maintain working memories of

objects, other spatial locations, motion, textures, etc. They are highly selective for the specific features

Visual imagery in working memory simulates visual processing (Finke 1989, Kosslyn 1980,…)◦ Analogously, motor imagery, auditory imagery, etc.

Mental rotation of visual objects -> motor simulations of making them turn (Richter et al. 2000)

Working memory

Behavioral evidence◦ When asked whether an property belongs to an objects

subjects simulate properties to verify them (Solomon & Barsalou 2004)

Lesion evidence◦ Lesions in one modality – losing categories that rely on it

for processing (Damasio 1994, …) E.g. damage to visual areas – losing of ability to

categorize animals (visual processing is dominant) Damage to motor areas – categorization of tools

Neuroimaging evidence (Martin 2001, 2007)◦ When processing conceptual knowledge, brain areas

representing properties are active

Conceptual processing

Perceptual simulation Motor simulation Affective simulation

Language comprehension

Situation models◦ Evidence of modal representations in language

comprehension Spatial representation (Bower & Morrow 1990) People confused pictures with text (Intraub &

Hoffman 1992) Replacing words with pictures did not disrupt

sentence processing (Potter 1986)

Language comprehension

Subjects read a sentence and then processed a picture that either matched or mismatched something implied by the sentence◦ “The ranger saw the eagle in the sky”◦ Picture of an eagle – wings outstretched or folded

Visual irrelevant information interferes with spatial inferences (Fincher-Keifer 2001)

Perceptual simulation

Verbs for head, arm and leg actions produce head, arm and leg simulation in the respective areas of the motor system (Pulvermuller 2005)

When action to make a response is consistent with text meaning, the response is quicker (Glenberg & Kaschak 2003)

Subjects simulate corresponding motion through space (Richardson et al. 2003)

Positive/negative valence (Meier & Robinson)

High/low power (Schubert 2005)

Motor simulation

Subjects’ faces configured according to sentences with emotional content (Havas 2007)

When facial emotion matched the content comprehension was better

Gesture◦ Producing gestures helps speakers retrieve words

whose meaning are related to the gestures (Krauss 1998)

◦ Also help listeners comprehend what speaker says◦ Children can gesture before speaking

Affective simulation

Physical reasoning◦ Gear, pulleys◦ Driven by spatial simulation◦ Sketchy, not holistic and detailed

Abstract reasoning◦ Content effects◦ Reasoning about time using space domain -

metaphors

Thought

Embodiment effects◦ Activating elderly stereotype causes people to

walk slowly and to perform lexical decision slowly (Dijksterhuis & Bargh 2001)

◦ Engaging the smiling musculature produces positive affect (Strack et al. 1988)

Social mirroring◦ Individual differences in the ability to simulate

other people’s mental states correlate with rated empathy (Jackson et al. 2005)

Development◦ Mirroring, object permanence…

Social Cognition

Questions?