Lecture 4. Representing Space II: Shortcomings of the inner space

proposals and a possible alternative

The problem of nonconceptual representation of space and scenario content

The internalized space options Internalizing constraints: Marr & Shepard Functional Space Literal (Neural) space

The empirical evidence for a special form of reconstructed spatial representation: Some imagery demonstrations and experimental findings

A quick summary of some empirical findings involving mental imagery: orWhat people mean when they say images are ‘spatial’

Two general kinds of studies

1. Studies showing that imagery ‘involves’ the visual system I will briefly mention these because most of them

lead to the ‘images are spatial’ view

2. Studies showing that image representations have spatial properties

Summary of some empirical findings on mental imagery and vision

Imagery and Vision Early studies of interference between vision and

imagery (Brooks) Activation of visual cortex during imagery

Why is this relevant? Because V1 is retinotopic!

Image-vision superposition studies

Most of these studies relating imagery and vision are of interest to us only because they were thought to support a picture-theory of images in whch images are laid out in space

Is mental imagery is carried out by the early (modular) visual system?

Suppose that mental imagery involved the visual system, what would that tell us about how space is represented? Before drawing conclusions, consider:

Virtually all mental imagery phenomena (scanning, mental rotation, etc) are exhibited by congenitally blind people.

Evidence for the involvement of primary visual cortex is not univocal. The majority of studies report that many cortical areas are involved in mental imagery

There is clear evidence of spared visual imagery with severely damaged visual perception and vice versa (this applies to achromatopsia, agnosia, visual neglect and cortical blindness)

There is good evidence that information in imagery is not reperceived through vision but only inferred

Summary of some empirical findings concerning the spatial

character of mental images.

What makes images “spatial”?

Why do we think images are spatial? The strong intuition that things in an image have a location, in

some relevant sense – e.g., you can point to them! How could a “symbolic” format have scenario content? Directions?

Spatial interaction between location in an image and location in a stimulus Spatial Interference (Brooks) Attentional enhancements (e.g., Podgorny & Shepard) Illusions in combined image-perception displays

Effects of image distances (e.g., scanning) The connection with the motor system

Eye movements during mental imagery (Brandt &Stark, 1997) Visual-motor adaptation (Finke) The Simon effect (S-R compatibility with images)

Brooks’ spatial interference study

Respond by pointing to symbols in a table or by saying the words left or right

Shepard & Podgorny experiment

Both when the displays are seen and when the F is imagined, RT to say whether the dot was on the F was fastest when the dot was at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

Visual illusions with projected images

Bernbaum & Chung. (1981)

Studies of ‘mental scanning’Do they show that images have metrical properties?

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Relative distance on image

La

ten

cy

(s

ec

s)

Other reasons for thinking that images are “spatial”: The connections among vision,

imagery and motor-control Eye movements during imagery are similar to those during

visual perception of the same figure Visual-motor adaptation occurs with imagined hand position

that is similar to adaptation with prism glasses (Finke, 1979) Stimulus-response compatibility effects are observed between

location in an image and the side of the response Patients with visual neglect often show imaginal neglect

Famous 1978 Milan Duomo experiment of Eduardo Bisiach A patient with post-operative tunnel vision also showed

“tunnel imagery”(But by the time she was tested, the patient know how things looked to her)

It takes longer to report details from a “small” image!

If images are spatial, where is the space?

Is it in the head?Is it an abstract representation?

Functional space? Analogue space? Tacit (conceptual) knowledge of space?

Is it in the world that is concurrently perceived and in which objects are indexed?

Are images displayed in a functional space?

Can we explain the spatial character of images by appealing to a “functional space”?

What properties does a functional space have?In virtue of what (natural laws, principles,

architectural properties of mind) does the functional space have the properties it does?

The picture theory and “Functional Space”

“The space in which the points appear need not be physical…, but can be like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is “read” and processed that it comes to function as if it were arranged into an array (with some points being close, some far, some falling along a diagonal, and so on).”

(Kossyln, 1994)

Is there any coherent sense of “space” that is neither literal (real, physical) space nor merely knowledge of spatial properties, with no commitment about the format in which this knowledge is represented?

In what way is a matrix data structure spatial?

Must the cells be accessed in any particular order? Which cells are “between” two

given cells? Which cells are collinear? What makes that so?

In what way is a matrix data structure spatial?

Must the cells be accessed in any particular order? Which cells are “between” two

given cells? Which cells are collinear? What makes that so?

In what way is a matrix data structure spatial?

Must the cells be accessed in any particular order? Which cells are “between” two

given cells? Which cells are collinear? What makes that so?

Questions about the Functional Space option

Where does the “spatial” property of “functional space” come from? Why does a matrix seem to have the spatial properties assumed in picture-theories of mental imagery?

Must Pythagoras’s theorem hold of the matrix? Why?Why does attention have to pass through “intermediate” cells in

getting from A to B. What makes these cells “intermediate”? In what sense are they

“between” A and B? Why does it take longer to move attention through more cells?

Why does adding point C not change the relation between points A and B?

What makes it the case that when C is added, then the location of B is between A and C?

Spatial properties of functional space

There are two possibile reasons why spatial properties apparently hold of functional (matrix) space:The principled alternative. These are principled

(lawlike, inherent) properties because the matrix is (tacitly) assumed to be just a simulation of real space, or

The ad-hoc alternative. The properties are not inherent properties of the functional space. Rather, they are just a way of depicting the properties that were observed in the data (this is just like listing the findings without explaining them).

Ways of explaining the scanning effect:

1. If images have spatial extent, then the explanation is straightforward:

Time = Distance Speed

2. If images represent distance, then

Time = Representation-of (distance) ? Representation-of (speed)

Only (1) constitutes an explanation!

Aside: Is image space represented in an analogue form?

An analog representation is not subject to the arguments presented against “functional space”

This is a much more difficult proposal than generally recognized – which is why nobody has seriously proposed an analogue representation of space other than space itself

It would require a representation that not only results in Time = Rep(distance) Rep (speed), but the properties that represent distance and speed would also have to meet many other requirements – e.g., they would have to model Euclid’s Axioms, and much of basic contact physics (since these are obeyed in imagery).

This would still not do because most imagery phenomena are Cognitively Penetrable so they do not arise from properties of the architecture

A puzzle …Why do we find it natural to assume that brain processes

underlying mental imagery share certain properties with what is represented, but not others? Do the brain processes involved in representing a certain event

take the same amount of time as the duration of the imagined event? A monotonic function of real time?

Does the size of the brain state corresponding to a large image reflect the relative size of what is imagined?

Are distances in the neural representation of a certain scene a monotonic function of distances in the scene?

Does the color (volume, mass, temperature) of the brain state vary with the equivalent represented property?

The Intentional Fallacy strikes again!

The literal space option

This was not taken serously until the recent findings that:

The early visual cortex (V1) is spatially mapped, and

Visual cortex is active (as determined by PET and fMRI) when people examine a visual image

Activity on visual cortex of primates is retinotopic

Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218(4575), 902-904.

Does the retinotopic mapping of visual cortex support the view that images are

displayed in real space in V1?

The “display” in visual cortex is retinotopic, which means it is narrow in scope, moves with eye movements, and is inherently 2-dimensional!

But scenario content is panoramic and 3-dimensional! Mental images are experienced as being in allocentric

coordinates and are panoramic or even cycloramic Every imagery finding applies equally in 3D as in 2D Image size does not map onto size in V1, so the “hard to

see” explanation of small images does not work

The topographical structure of the visual cortex could not support mental images

The 2D mapping of retinal activity in V1 cannot be identified with the mental image which is panoramic, 3-dimensional, dynamic and has many other properties that could not be mapped onto V1, so we would need a different theory for them.

…and there are many other differences …

Image content is not visually reinterpreted: it does not have the signature properties of visual interpretation (e.g., amodal completion, spontaneous 3D interpretation, visual ambiguity, Emmert’s law, etc etc)

Image content cannot be accessed in free order

Image content is whatever we want it to be! i.e., it is cognitively penetrable because it is your image!

e.g., our mental scanning findings

Exactly the same applies to the effect of image size, visual angle of the mind’s eye, the oblique effect, etc

Imagine two parallelograms

Imagine two parallelograms

Imagine two parallelograms

Amodal completion in imagery?

Amodal completion in imagery?

Slezak figures

Pick one of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

Rotated Slezak figures

No subject was able to recognize the mentally rotated figure

Subjects remembered the figures well enough so if they drew it they could recognize the rotated figure

PS. There have been differing claims about such results!

…and there are many other differences … Image content is not visually reinterpreted: it does not

have the signature properties of visual interpretation (e.g., amodal completion, spontaneous 3D interpretation, visual ambiguity, Emmert’s law, etc etc)

Image content cannot be accessed in free order

Image content is whatever we want it to be! i.e., it is cognitively penetrable because it is your image!

e.g., image size?

e.g., our mental scanning findings

Exactly the same applies to the visual angle of the mind’s eye, the oblique effect, and dozens of other findings

Studies of ‘mental scanning’Do they show that images have metrical properties?

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Relative distance on image

La

ten

cy

(s

ec

s)

scan image

imagine lights

show direction

(Pylyshyn & Bannon. See Pylyshyn, 1981)

An alternative view of the source of spatial properties of images

There is something special about images that are ‘projected’ onto a perceived scene Can phenomena involving projected

images be explained in terms of indexing of objects and binding them to imagined things?

Shepard & Podgorny experiment

Both when the displays are seen and when the F is imagined, RT to say whether the dot was on the F was fastest when the dot was at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

Visual illusions with projected images

Bernbaum & Chung. (1981)

Visual adaptation and aftereffects through images (Ron Finke)

Instead of seeing their hand in the wrong location (due to wedge prisms) subjects were told to imagine that there hand (which was actually under a screen) was at some shifted location

After moving their hands a few times and each time imagining them to be at the false location, subjects were allowed to see their hand reaching and it did show a negative aftereffect

But the conditions for adaptation are known to be (1) a mismatch between felt and seen position and (2) a conviction that the hand is actually at the given location. The same result can be obtained by a light held by the hand. No actual visual properties of the hand are required

S-R Compatibility effect with a visual display

S-R Compatibility effect with a mental image

What happens when you use imagery with your eyes closed?

The existence of a very accurate spatial sense suggests that there are mechanisms similar to visual indexes in other modalities

We don’t need to have a representation of empty places so long as we have coordinate transformation operations linking gestures and representations of filled locations (tactile perceptions are of tactile objects)

Can indexing convert an apparently mental-space phenomenon to a perceived-space phenomenon?

When the imagined stimulus is projected onto a visually perceived background (as in the S-R compatibility example) it is easy to see how it can become a visual phenomenon – subjects must respond to a place selected in the actual display (or near it)

Can we apply this approach when there is no visual input to which to bind imagined objects?

This is where the idea of spatial sense plays a role. We can convert between sensed modalities with coordinate transformation processes.

A spatial sense is very different from a mental picture

Exercise of the spatial sense need not be accompanied by a perceptual experience. It is amodal and typically free of any conscious content

It does not require that any properties of imagined objects be represented – just their identity (indicated by binding them to sensory objects)

The spatial sense exploits visual, auditory, kinesthetic and proprioceptive information

The sense of space does not need an internal spatial medium. It arrises purely from the capacity to bind object files (possibly empty) to sensed objects.

We don’t need a spatial display in our head if we have a way to pick out and keep track of a few

sensory objects in real space

1. None of the experiments that suggest a spatial display in visual cortex need to appeal to anything more than a small number of imagined locations (e.g., Podgorny Shepherd , Finke, Tlauka, …).All spatial imagery phenomena have been found in the blind

2. If we can pick out a small number of occupied locations in real space we can use these to allocate attention or to target eye movements. This is likely what goes on in the Shepard & Podgorny, Bernbaum & Chung and other cases involving combined vision and imagery.

We don’t need a spatial display in our head...

3. If these selected perceived objects are also bound to objects of thought (e.g., “Object Files”) it can result in the objects of our thought having persisting spatial locations and spatial relations (e.g., obeying metrical Euclidean axioms).

4. This would also explain why some patients with visual neglect sometimes show neglect to the same side of their image (Bisiach & Luzzatti, 1978). Patients may have projected imagined objects onto real space in the

course of their imagining, and their visual neglect resulted in failure to orient to the indexed objects in the neglected field. Neglect appears to involve a failure to orient attention (Bartolomeo & Chokron, 2002)

The patient whose surgery resulted in tunnel vision and tunnel imagery may simply have been showing she knew how things looked to her post-surgery

Should you try to rescue the picture theory?

Summary I have presented a view of how conceptual representations

might be grounded in causal connections to the world, through a perceptual mechanism that enables individuation, selection, and reference to individual sensory objects

I have proposed that this mechanism picks out a limited number of sensory individuals in the world – individuals that usually turn out to be physical objects, although they are picked out nonconceptually and therefore not as objects

While identifying and tracking individuals in general requires a conceptual apparatus for individuation and identity, perceptual systems are equipped to provide this function in a way that almost always works in our kind of world. Without such a mechanism, grounding of concepts in experience would not be possible

SummaryThe theory of Indexing (FINST theory), developed for quite

different reasons, postulates that there is a primitive capacity in the modular perceptual architecture for identifying and keeping track of individuals in a nonconceptual manner

This nonconceptual “picking out” is closely related to focal attention and is able to independently select several (~4-5) causally efficacious entities. These are usually physical objects, as opposed to empty places, since empty places do not have index-capturing causal powers

This view has consequences for a theory of sentience and for any theory of symbol-grounding (or concept-grounding). It provides a different foundation for sentience than that of feature-placing, adopted by Strawson, Clark and others.

SummaryThis account assumes a spatial sense, viewed as a

perceptual-motor skill, whose exercise requires referring to causally efficacious indexed things in the perceived world and computing coordinate transformations only among the selected individuals and only as needed.

The theory shuns the assumption of a nonconceptual mental representation with rich scenario content, such as that which we consciously experience, and denies the existence of picture-like representations in either vision or imagery. Spatial properties are assumed to be conceptually encoded from the world itself, rather than from a nonconceptual picture-like mediating representation. Since such property encoding occurs only for indexed objects, this also provides a solution to the binding problem

Summary & End

The FINST index theory claims that certain spatial properties of mental images are inherited from the world by a process of associating objects of thought with indexed objects in several modalities. Since properties of indexed objects can be visually evaluated, this makes it possible to carry out certain spatial inferences without the need for a rich nonconceptual spatial representation beyond that provided by the selection of individuals through FINST indexes.

More information can be found at:

http://ruccs.rutgers.edu/faculty/pylyshyn.html