readers routinely represent implied object rotation: the role of visual experience
DESCRIPTION
Readers routinely represent implied object rotation: The role of visual experience. Wassenberg & Zwaan , in press, QJEP. Brennan Payne Psych 525 10.27.10. Theories of discourse comprehension. Construction-Integration (Kintsch & van Dijk , 1978; Kintsch, 1998) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/1.jpg)
Readers routinely represent implied object rotation: The role of visual experience
Wassenberg & Zwaan, in press, QJEP
Brennan PaynePsych 525
10.27.10
![Page 2: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/2.jpg)
Theories of discourse comprehension
•Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998)
•Structure-Building (Gernsbacher, 1990, 1997)
•Event-Indexing Model (Zwaan et al., 1995)
•Resonance Model (O’Brien et al., 1995, 1998)
•Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.
![Page 3: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/3.jpg)
Theories of discourse comprehension
•Construction-Integration (Kintsch & van Dijk, 1978; Kintsch, 1998)
•Structure-Building (Gernsbacher, 1990, 1997)
•Event-Indexing Model (Zwaan et al., 1995)
•Resonance Model (O’Brien et al., 1993, 1995, 1998)
•Shared assumption that discourse comprehension can be modeled as the integration of abstract and amodal representations.
![Page 4: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/4.jpg)
Construction-Integration• Computational model
• Different levels of representation: • Surface form of the text• Text base: propositional information from the text• Situation model: representation of situation implied with the text;
derived from propositional text base• Proposition: “idea unit”; smallest unit of knowledge. Follows predicate
argument form: PREDICATE(ARGUMENT1, ARGUMENT2)
2a. The carpenter pounded the nail into the wall.
2b. The carpenter pounded the nail into the floor.[POUNDED(CARPENTER, NAIL)], [INTO (NAIL, WALL)]
[POUNDED(CARPENTER, NAIL)], [INTO(NAIL, FLOOR)]
1a. The ranger saw the eagle in the sky.[SAW(RANGER, EAGLE)], [IN (EAGLE, SKY)]
1b. The ranger saw the eagle in its nest.[SAW(RANGER, EAGLE)], [IN (EAGLE, NEST)]
![Page 5: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/5.jpg)
Alternative Account1. The carpenter pounded the nail into the wall.
2. The carpenter pounded the nail into the floor.
[POUNDED (CARPENTER, NAIL)], [IN (NAIL, WALL)]
[POUNDED (CARPENTER, NAIL)], [IN (NAIL, FLOOR)]
Proposition Account: Highly Identical, only difference is N specifying orientation
Perceptual Symbol Account (Barsalou, 1999a,b; Stanfield & Zwaan, 2001; Zwaan et al., 2002): Sentences are very different in perceptual representation that is implied.
![Page 6: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/6.jpg)
Do readers represent perceptual information?
John put the pencil in the cup.
• Sentence-picture verification task Was this mentioned in the previous
sentence?
Significant differences in RT latencies when objects matched vs. mismatched.
838 (331)
882 (329)
*
Stanfield & Zwaan (2001); Psych. Sci.
![Page 7: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/7.jpg)
Perceptual Traces (Zwaan & Kaschak, 2008)• Orientation (Stanfield & Zwaan, 2001)• Shape (Zwaan et al., 2002; Dijkstra et al., 2004)• Size (Taylor & Zwaan, 2010)• Movement and Motion (Kaschak et al., 2005; 2006) • Color (Richter et al., 2009; Therriault et al., 2009)
Previous Research: Nature of language representation
Sentence Picture
Can this perceptual information affect online language processing?
Picture Sentence
Do readers represent perceptual information?
![Page 8: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/8.jpg)
Current StudyDoes a recent visual exposure to an object in a specific orientation affect later language comprehension?
Wassenburg & Zwaan (in press); QJEP.
A 3-phase “visual memory” paradigm (Zwann et al., 2010)1. Word-picture verification task
-Experimental items shown in vertical or horizontal orientation2. 15- minute filler task3. Eye-tracking session
Three phases are presented as unrelated experiments to deter some kind of strategy use.
Predict a match/ mismatch effect: fixation times on the prepositional phrase (into the wall/ in the cup) that implies the object orientation should be sensitive to the orientation of the previously seen image.
![Page 9: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/9.jpg)
MethodParticipants:•N = 34; N= 28 after track loss/errors. 50%Female •Age= 20.3 (18-24). •Native Dutch Speakers
MaterialsP1• 80 word-picture items • 60 fillers; 20 critical items• Each critical item formed a match with its word• Orientation (H;V) counterbalanced across participants over 2 lists
P2• ??? Maybe the flag test
P3 • Tobii 2150 eye tracker• 40 Dutch Sentences• 20 filler; 20 critical sentences using critical words from P1 • Half of each orientation matched and half mismatched
![Page 10: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/10.jpg)
Method
Aunt Karen finally found the toothbrush in the sink of the bathroom.
Phase 3:
ToothbrushPhase 1:
Phase 2: VSP rotation filler taskApprox. 15 min.
1 2 3 4* 5
Procedure
![Page 11: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/11.jpg)
Results*
†
†p = .06
*p < .05
![Page 12: Readers routinely represent implied object rotation: The role of visual experience](https://reader036.vdocuments.us/reader036/viewer/2022070423/56816720550346895ddba4f5/html5/thumbnails/12.jpg)
Much in the way that language affects later visual processing (Stanfield & Zwaan, 2001), visual memory also influences language processing.
Prior exposure to a picture of an object in a particular orientation affects later reading times for phrases that imply the orientation of that object.
Match/mismatch effects occur on first pass measures on the disambiguating PP and diminish quickly, suggesting that these effects are both early and immediate.
Reading comprehension may be multimodal, not only using linguistic representations, but sensory/perceptual representations as well.
Conclusion