Perceiving, Remembering & Knowing Perceiving, Remembering & Knowing

in Scene Cognition: in Scene Cognition:

Where are the Divisions?Where are the Divisions?

Helene IntraubHelene Intraub

University of DelawareUniversity of Delaware

Framework for Scene CognitionFramework for Scene Cognition

Visual construct Visual construct Spatial constructSpatial construct

““Dividing lines” between perception and memoryDividing lines” between perception and memory

Visual Visual Starting point: Starting point:

representation of therepresentation of the

picturepicture Very short-term storesVery short-term stores

• IconIcon• Transsaccadic memoryTranssaccadic memory• CSTMCSTM• VSTMVSTM

What is transferred?What is transferred? Lost?Lost? Added?Added? Locus of effect?Locus of effect?

Very short termstores

STMVisual Working

Mem

LTM

Intraub & Richardson (1989) JEP:LMC

LTM [minutes – 2 days] (Koriat et al., 2000; Annual Review) nonlinguistic version

RSVP trios, 1 sec masked interval-Intraub, Gottesman, Willey & Zuk (1996), -Bertamini, Jones, Spooner & Hecht (2005)

Boundary Extension (BE)Boundary Extension (BE)

Roediger (1996) Roediger (1996) “… the phenomenon falls at “… the phenomenon falls at the boundary between perceiving and the boundary between perceiving and remembering”remembering”

Spatial Spatial Starting point: Starting point:

spatial representation of thespatial representation of the

scenescene the pic only partially the pic only partially

revealsreveals

Neighborhood

Outdoors

Sky

Street surface

Spatial CognitionSpatial Cognition Starting point: Starting point:

spatial representationspatial representation

below

right left

behindbehind

above

In frontIn front

Graded representation Graded representation

Imagine, response to object imagine in Imagine, response to object imagine in front, faster than in back front, faster than in back

““Resolution” better than front than back Resolution” better than front than back

(Franklin, N., Henkel, L., & Zangas, T., 1995; (Franklin, N., Henkel, L., & Zangas, T., 1995; Franklin & Tversky, 1990)Franklin & Tversky, 1990)

Perception vs. Memory?

Visual: BE is the result of this active process

Spatial: scene understanding is a graded 3-D representation

The region that will constitute the BE error is already there.

BE is the result of a decision – which part of the entire scene representation had a sensory source – was seen before?

When does BE occur?When does BE occur?

Visual construct: Visual construct: • No a priori predictionNo a priori prediction• Early no BE, over some pd of time it emergesEarly no BE, over some pd of time it emerges• Which early visual memory system?Which early visual memory system?

Spatial approachSpatial approach• Predicts – when sensory info is gone Predicts – when sensory info is gone • Conscious memory includes surrounding Conscious memory includes surrounding

swath of spaceswath of space

Visual constructVisual construct

Diff possibilitiesDiff possibilities

Either wayEither way

Either wayEither way

May or may notMay or may not

Spatial ConstructSpatial Construct

When sensory When sensory input is goneinput is gone

Weaker record, Weaker record, more BEmore BE

Weaker weaker, Weaker weaker, more BEmore BE

Same spatial Same spatial scene repscene rep

Timing?Timing?

Retention intervalRetention interval

Stimulus durationStimulus duration

Divided attention?Divided attention?

Haptics?Haptics?

Early time-course experiments Early time-course experiments

are in collaboration with: are in collaboration with:

Christopher A. DickinsonChristopher A. Dickinson

Remains on Screen

______

BoundaryRating

Scene (250 ms)

Interrupt42 ms

Exp 136 Ss 36 Trials (4 types)

CC WW

0

+2

-1

-2

+1

Much moreMuch moreWide-angleWide-angle

Slightly moreSlightly moreWide-angleWide-angle

Much moreMuch moreClose-upClose-up

Slightly moreSlightly moreClose-upClose-up

Identical viewIdentical view::

Trial Type:Stimulus and Test Identical

CC

Trial Type:Stimulus and Test Identical

WW

0

+2

-1

-2

+1

Much moreMuch moreWide-angleWide-angle

Slightly moreSlightly moreWide-angleWide-angle

Much moreMuch moreClose-upClose-up

Slightly moreSlightly moreClose-upClose-up

Identical viewIdentical view::

Close-ups Wide-angle

CC WW-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0M

ean

Rat

ing

Test Type

Exp 1_CC_WW_42 ms

42 ms Interruption

Identical View during Presentation and TEST

CC vs. WW: t(35) = 3.29, p = .002

CW WC

0

+2

-1

-2

+1

Much moreMuch moreWide-angleWide-angle

Slightly moreSlightly moreWide-angleWide-angle

Much moreMuch moreClose-upClose-up

Slightly moreSlightly moreClose-upClose-up

Identical viewIdentical view::

Trial Type:Stimulus and Test Differ

CW

Trial Type:Stimulus and Test Differ

WC

0

+2

-1

-2

+1

Much moreMuch moreWide-angleWide-angle

Slightly moreSlightly moreWide-angleWide-angle

Much moreMuch moreClose-upClose-up

Slightly moreSlightly moreClose-upClose-up

Identical viewIdentical view::

Close-Wide Wide-Close

Park, Intraub, Yi, Widders & Chun; fMRI/PPA/RSC

Slightly Different View during Presentation and TEST

CW WC-1.2-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0M

ean

Rat

ing

Test Type

Exp 1_CW_WC_42 ms

42 ms Interruption

CW vs. WC (deviation from same – asymmetry): t(35) = 6.64, p < .001

SummarySummary

BE occurred when memory ought to have been BE occurred when memory ought to have been excellent!excellent!

Memory Load: 1 pictureMemory Load: 1 picture Task is known & repeated across 36 trialsTask is known & repeated across 36 trials Target location was fully predictableTarget location was fully predictable Extremely brief interruption of sensory input: Extremely brief interruption of sensory input:

• 42 ms (commensurate w/saccade)42 ms (commensurate w/saccade)• Eye tracking, also found across actual Eye tracking, also found across actual

saccadesaccade (103 ms) (103 ms)

Graded Representation

Effect of Presentation time?

100 ms

250 ms

500 ms

1.0

0

-1.0

-.50

.50

100 250 500 Duration (ms)

Intraub, Daniels, Horowitz & Wolfe (submitted)

Divided Attention = Greater BE

750 ms

BE occurs following haptic exploration of space. Blindfolded-signted, and anEarly deaf and blind woman (Lebers Syndrome) – Intraub (2004) Cognition

Mono Large (3 cm)Mono Small (.6 cm)

Quinn & Intraub (2007)Quinn & Intraub (2007) Child DevelopmentChild Development

3-4 month old infants3-4 month old infants(and 6-7 month old infants)(and 6-7 month old infants)

Scene Understanding: Spatial ConstructScene Understanding: Spatial Construct

Visual – BE develops (pic rep – then fill out)Visual – BE develops (pic rep – then fill out)• Perception/memory issuePerception/memory issue

Spatial – BE area Spatial – BE area is already thereis already there• Which part of the full spatial representation was Which part of the full spatial representation was

originally sensory information?originally sensory information?• Reality monitoring (Johnson & Raye, 1981)Reality monitoring (Johnson & Raye, 1981)

• Usually LTMUsually LTM

• But may apply to any scene memory decisionBut may apply to any scene memory decision

Perception v. memory dividing line? Not an issuePerception v. memory dividing line? Not an issue Graded representation –prior sensory, amodal, Graded representation –prior sensory, amodal,

knowledgeknowledge

World is continuous… World is continuous… But sensory input is not!But sensory input is not!

Brain’s goal: Represent the scene Brain’s goal: Represent the scene Don’t retain spurious boundariesDon’t retain spurious boundariesSensory input is gone – mental rep remains Sensory input is gone – mental rep remains Regions are not tagged as “high acuity”, Regions are not tagged as “high acuity”,

“low acuity”, amodally “filled in” or “filled out” “low acuity”, amodally “filled in” or “filled out” On-line, layout projection facilitate view On-line, layout projection facilitate view

integration integration

Supporting a coherent representation that is Supporting a coherent representation that is always based on discrete successive always based on discrete successive views.views.

AcknowledgmentsAcknowledgments

Kristin O. Michod (RA)Kristin O. Michod (RA)

Undergrad RAsUndergrad RAs Dan Bensonoff Dan Bensonoff Dave DrowosDave Drowos Jason Rand Jason Rand Kari SchweifelKari Schweifel Mattie WilsonMattie Wilson

Erika FurlongErika Furlong We thank NIMH We thank NIMH David GriecoDavid Grieco

Kristin’s latest research, “Conceptual Masking:  Is it really all about the concept or does layout matter?” Submitted to VSS ’07

Special thanks to..Special thanks to..

James E. HoffmanJames E. Hoffman Anna PapafragouAnna Papafragou Neil IntraubNeil Intraub

For generously spending time listening, commenting, questioning…For generously spending time listening, commenting, questioning…

New RM-related research:New RM-related research:

w Anna and Ozge w Anna and Ozge (Turkish evidentials and ReMon)(Turkish evidentials and ReMon)

w Jessica Hughes (BE and ReMon)w Jessica Hughes (BE and ReMon)

Visual CognitionVisual Cognition BE developed over the BE developed over the

interruptinterrupt

Spatial CognitionSpatial Cognition The content that we The content that we

ultimately define as BE ultimately define as BE was part of the was part of the representation.representation.

Once sensory input is Once sensory input is gone…gone…

Cannot distinguish Cannot distinguish peripheral from amodal…peripheral from amodal…

Can distinguish from less Can distinguish from less constrained knowledgeconstrained knowledge

C:\Documents and Settings\Administrator\My Documents\My Files\Presentations\Psychonomics 06 use\program folder\demo script 2 program.exe

Park, Intraub, Yi, Widders, & Chun (under review)

PPA

Face changes every 150 ms

Face changes every 150 ms

Spatial LayoutSpatial Layout

Boundary Extension:Boundary Extension: Infants as young a 3-4 months Infants as young a 3-4 months

Quinn & Intraub Quinn & Intraub (in press).(in press). Child Child Dev.Dev.

Children 6–80 y.o. AdultsChildren 6–80 y.o. Adults

Seamons et al., (2002)Seamons et al., (2002) Adult haptic exploration Adult haptic exploration

• Deaf & blind adult (Lebers Syndrome)Deaf & blind adult (Lebers Syndrome)

• Blindfolded adultsBlindfolded adults

Intraub (2004). Cognition.Intraub (2004). Cognition.

Labeling Cognitive ProcessesLabeling Cognitive Processes

Heuristic – Perceiving, remembering Heuristic – Perceiving, remembering IssuesIssues““New Look” psychologist 1940’s/1950’sNew Look” psychologist 1940’s/1950’sBeyond the stimulus/or Perception as CategorizationBeyond the stimulus/or Perception as Categorization

Jerome Bruner ( )Jerome Bruner ( )Hochberg (1968, 1978)– or one of the chaptersHochberg (1968, 1978)– or one of the chaptersQuote about schemaQuote about schema““Levels of processing” 1970’sLevels of processing” 1970’sMemory a byproduct of perceptual processesMemory a byproduct of perceptual processes

Craik & Lockhart ( 1972)Craik & Lockhart ( 1972)Situated Perception (or Embodied) 90’sSituated Perception (or Embodied) 90’sBarsalou and othersBarsalou and others

Exp 2: BE across a SaccadeExp 2: BE across a Saccade

EyeLink II (sampling at 500 hz; chinrest) EyeLink II (sampling at 500 hz; chinrest) N = 24 N = 24 36 trials36 trials Stimulus and test on opposite sides of screenStimulus and test on opposite sides of screen Stimulus locations held constant for each SStimulus locations held constant for each S

• Stimulus right; Test on left for ½ SsStimulus right; Test on left for ½ Ss• Stimulus left ; Test on right for ½ SsStimulus left ; Test on right for ½ Ss

Exp 2:Across a Saccade?Exp 2:Across a Saccade?

Launch SaccadeTo this location

250 ms

Boundary Rating

Exp 2:Across a Saccade?Exp 2:Across a Saccade?

LaunchSaccade

250 ms

Boundary Rating

Mean total time = 298 ms Initiate saccade = 195Saccade duration = 103 (approx. 11° amplitude)Full screen is 28° x 21°

Target onset is GAZE Contingent (EyeLink II, chinrest)

When eyes land:Identical View Different View

CC WW-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Mea

n R

atin

g

Test Type

Exp 2_EyeLink_CC_WW_GC

Target onset is GAZE Contingent (EyeLink II, chinrest)

When eyes land:Identical View Different View

CC WW-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Mea

n R

atin

g

Test Type

Exp 2_EyeLink_CC_WW_GC

CW WC-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Mea

n R

atin

g

Test Type

Exp 2_EyeLink_CW_WC_GC

59% Single Fixation; 41% two fixations

Target onset is GAZE Contingent (EyeLink II, chinrest)

When eyes land:Identical View Different View

CC vs. WW: t(23) = 2.11, p = .046CW vs. WC (absolute values): t(23) = 3.23, p = .004

Filtered DATA – Single saccade only

When eyes land:Identical View Different View

Single Saccade ONLY trials

CC WW-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Mea

n R

atin

g

Test Type

Exp 2_EyelinkCC_WW_GC_ Single Saccade only

CW WC-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Exp 2_EyelinkCW_WC_GC_ Single Saccade only

Mea

n R

atin

gs

Test TypeFiltered DATA – Single saccade only

When eyes land:Identical View Different View

Single Saccade ONLY trials

Visual Cognition Spatial Cognition

Johnson & Raye (1981) Reality MonitoringJohnson & Raye (1981) Reality Monitoring Johnson et al. (1993) Source MonitoringJohnson et al. (1993) Source Monitoring

Spatial RepresentationSpatial Representation Enter room blindfoldedEnter room blindfolded Spatial sense (in front, above, left, behind…)Spatial sense (in front, above, left, behind…) Remove blindfold -- Discrete samplesRemove blindfold -- Discrete samples

- 3 Fixations/sec- 3 Fixations/sec

- Saccade 30-50ms (Rayner, 1998)- Saccade 30-50ms (Rayner, 1998)

– – saccadic suppressionsaccadic suppression

Transaccadic memory (Irwin, 1992)Transaccadic memory (Irwin, 1992)

Is Is BEBE available?available?

Perception vs. Memory?

Graded representation

Perception vs. Memory?

Visual: extrapolation occurs around the edges

BE is the result of this active process

Spatial: scene understanding is a graded 3-D representation

The region that will constitute the BE error is already there.

BE is the result of a decision – which part of the entire scene representation had a sensory source – was seen before?

Representation of surrounding space is present immediately.

So what determines BE?

Job: distinguish what portion of this large, mostly vague/abstract representation, was derived from sensory experience.

Reality Monitoring

(Johnson & Raye, 1981)

Quality of detail/resolution

Starting with stimulus – BE develops some time after an initial representation is available

Starting with the spatial representation – the information necessary for BE is already there, so is part of the representation as soon as there is no sensory input – can’t distinguish.

Which part of the rep initially was delivered via sensory info?Can’t distinguish from nearby highly constrained part of amodal rep.

Don’t confuse with general knowledge, but do confuse with constrained portion.

Like reality monitoring … (Johnson and Raye, 1981).In fact we know that reading can lead to memory confused with sensory input about scenes that were never seen!

But that always associates with LTM – here it applied immediately following stim offset (Skidmore group and amodal completion of objects).