Slide 1

Remembering what you saw is not the same as remembering what you heard.

Ray BeckerBielefeld University

Monica Gonzalez-MarquezCornell University & Bielefeld University

Todd FerrettiWilfrid Laurier UniversityEmpirical Evidence for EvidentialityJanuary 10, 20141

Empirical Methods in Cognitive Linguistics (2015)Local organizer:Jan Edson LeiteLinguistics Professor at UFPBJoo Pessoa, Paraba, Brazil

2Embodied cognitionPerceptual symbol systems (Barsalou, 1999)

Simulation (Barsalou, 1999; Glenberg, 1997)She heard/saw the man speak at the school.

Are there modality-specific differences?3Cross-modal switching costsPecher, Zeelenberg, and Barsalou, 2003Property verification taskleaves rustling then bird chirpingleaves rustling then raspberry tart

4Modality specific differences?Pecher, Zeelenberg, and Barsalou, 2003No main effect (visual-property verification was not slower than auditory-property verification)

What about at the discourse level?Would the brains response differ?

5N400 component

Federmeier, K.D. and Kutas, M. (1999) A rose by any other name: long-term memory structure and sentence processing. J. Mem. Lang.41, 4694956P600 componentHagoort, P. & Brown, C. M. (1999). Gender electrified: ERP evidence on the syntactic nature of gender processing. Journal of Psycholinguistic Research, 28, (6) 715-728.

7P600 componentHagoort, P. & Brown, C. M. (1999). Gender electrified: ERP evidence on the syntactic nature of gender processing. Journal of Psycholinguistic Research, 28, (6) 715-728.

8Neuroscience dataBurkhardt, P. (2006)Context A: BRIDGED DP Tobias visited a concert in Berlin.

Context B: GIVEN DP Tobias visited a conductor in Berlin.

Context C: NEW DP Tobias talked to Nina.

Target sentence (following Context AC)He said that the conductor was very impressive.

9Main ERP component of interestThe Late Positivity component (P600/LPC)

Reflects integration difficulty

Decay of the referent in memory leads to the concept needing to be re-instantiated into the current situation model.

10Burkhardts findingsMore positively-leading (P600/LPC) for referents that followed Bridged and New contexts than Given contexts

11The current studyCould modality-specific differences also effect brainwave patterns (N400 or P600)

She saw the man speak at the school. (Complex)She heard the man speak at the school. (Less rich)

If the simulation is complex then it should be more salient in the current situation model and be more available for later reference.

12MaterialsCecile was a high school student in Waterloo, Ontario.Yesterday a motivational speaker was coming to the auditorium.She went to class and then it was time for the speaker.She heard/saw the man speak at the school.The auditorium was packed.She thought about the man and how inspired she felt.After school she went to the soccer field for practice.The inspiration drove her to take some extra penalty shots.It felt good to get in some extra practice.

The man was speaking at the school. (yes)13Procedure52 WLU undergraduate participants read short stories.

There were 20 experimental stories mixed in to another unrelated experiment. (124 stories total).

EEG was recorded from 64 electrodes.

Analysis was time locked to the onset of the target (e.g., man).14ResultsGreater positive deflection (P600) for heard versus seenSeen TargetHeard Target

Mean Amplitude (V)8 V1 s-4 V

Seen targets are easier to integrate

15DiscussionDescriptions of seen versus heard events has a quantitative effect on later reference.

Simulating the visual descriptions of an event may lead to greater saliency for that event in the current situation model than auditory descriptions.16Next stepsHow could interdisciplinary collaboration further our understanding of evidentiality?

Are there qualitative differences in eyewitness testimony depending on the sensory description?17Factive VerbsCompare sentences such as:

I heard/saw the man speak at the school.I knew/guessed that the man18Eyewitness testimonyIs this testimony more veridical than if the witness had heard the event?

19Eyewitness testimonyCompare the number of Google hits for I saw versus I felt.20Eyewitness testimonyNow compare those numbers for I saw and I felt to I heard.21Into the numbersI saw

I saw the tail of a large airliner. ... It plowed right into the Pentagon. Flight 77 (9/11)

I saw many people being shot down as they were trying to run away. Nairobi mall

and up the hill I saw a man running toward the monument and I started running over there. JFK assassination22Into the numbersI heard

Shortly after watching the second tragedy, I heard jet engines pass our building. Flight 77 (9/11)

I heard a rain of hundreds of bullets in a span of less than a minute. Nairobi mall

Just after the President's car passed, I heard three shots come from up toward Houston and Elm. JFK assassination

I heard him say he saw this happen.23Into the numbersI felt

I was cutting the grass and it came in screaming over my head. I felt the impact. Flight 77 (9/11)

I felt I'm almost going to lose consciousness. Syria alleged chemical attack

My check was torn, I discovered as I felt it gingerly. Hiroshima (1945)You can see the variety of different felt experiences.24Thank youMonica Gonzalez-MarquezTodd FerrettiCatherine CravenAlannah GuzakJenna HarwoodRebecca McKerron

25