imagery slides
DESCRIPTION
Imagery slides. Imagery and Memory. Memory Examples: Dual Code Theory To recall Y you must first recall X Windows, doorknob, glasses, other facial features, global-to-local But: Something like the same thing happens in recall of alphabet letters and many other memorized lists - PowerPoint PPT PresentationTRANSCRIPT
Imagery slides
Imagery and Memory
● Memory Examples: Dual Code Theory To recall Y you must first recall X Windows, doorknob, glasses, other facial features,
global-to-localBut: Something like the same thing happens in recall of
alphabet letters and many other memorized lists● Imageability rating are more effective than
frequency of occurrence or frequency of co-occurrence in paired-associates learning.
Vision is clearly involved when images are superimposed onto vision
Many experiments show that when you project an image onto a display the image acts very much like a superimposed display• Shepard & Podgorny (paper folding task…)• Interference effects (Brooks)
Controvercial Perky effect: Perception or response bias?
Project an image onto a perceived form
Brooks’ spatial interference study
Respond by pointing to symbols in a table or by saying the words left or right
Perception or attention effects?
● Many impressive imagery effects can be plausibly attributed to attention
● Bisiach widely-cited finding on visual neglect Bartolomeo, P., & Chokron, S. (2002). Orienting of attention in left unilateral
neglect. Neuroscience and Biobehavioral Reviews, 26(2), 217-234. Dulin, D., Hatwell, Y., Pylyshyn, Z. W., & Chokron, S. (2008). Effects of
peripheral and central visual impairment on mental imagery capacity. Neuroscience and Biobehavioral Reviews, 32(8), 1396-1408.
Does neglect require vision? Chokron, S., Colliot, P., & Bartolomeo, P. (2004). The role of vision in spatial
representations. Cortex, 40, 281-290.
We can to some extent control our attended region
Farah, M. J. (1989). Mechanisms of imagery-perception interaction. Journal of Experimental Psychology: Human Perception and Performance, 15, 203-211.
Is an image being projected onto a percept, or just a selective attention?
Shepard & Podgorny experiment
Both when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.
Similarities between perception of visual scenes and ‘perception’ of mental images
● Judgments from mental images Shape comparisons (of states: Shepard & Metzler) Size comparisons (Weber fraction or ratio effect)
• What do they tell us about the format of images?• But this applies to nonvisual properties (e.g., price, taste)
More demonstrations of the relation between vision, imagery (and later action)
• Images constructed from descriptions The D-J example(s) Perception or inference/guessing
• But there are even more persuasive counterexamples we will see later• The two-parallelogram example• Amodal completion• Reconstruals: Slezak
Dynamic imageryImagining actions: Paper Folding
Mental rotation
Time to judge whether (a)-(b) or (b)-(c) are the same except for orientation. Time increases linearly with the angle between them (Shepard & Metzler, 1971)
What do you do to judge whether these two figures are the same shape?
When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?
Is this how the process looked to you?
Mental rotation – the real story In mental rotation the phenomenology motivates the theory
of “rotation” – but what the data actually show is that, Mental rotation is only found when the comparison figures are
enantiomorphs or if the difference between figure pairs can only be expressed in figure-centric coordinates eg. they are 3D mirror-images
No rotation occurs if the figures have landmarks that can be used to identify the relations among their parts.
Records of eye movements show that mental rotation is done incrementally: It is not a holistic rotation as often reported. If fact even the phenomenology is not of a smooth continuous rotation.
The “rate of rotation” depends on the conceptual complexity of both the figure and comparison task so that, at least, is not a result of the architecture (Pylyshyn, 1979). There are even demonstrations that it depends on how the subject interprets the figure (Kosslyn, 1994).
Mental Scanning• Hundreds of experiments have now been done
demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance.
• These have been reviewed and described in: Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental
images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.
Studies of mental scanningDoes it show that images have metrical space?
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Relative distance on image
Late
ncy
(sec
s)
Does this show that images are spatial, or have spatial properties, or that they “preserve metrical spatial properties”? (Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60.
The idea of images being in some sense spatial is an interesting and important claim
● I will discuss this claim at some length later because it reveals a deep and all-consuming error that runs through all imagery theorizing – by psychologists, neuroscientists and philosophers.
● This is in addition to the errors I discussed earlier: The idea that subjects understand the task of imagining something to be the task of pretending they are seeing it, and the idea that certain properties of the world are properties of the image (the intentional fallacy)
Constructing an image
● What determines what the image is like when it is constructed from memory or from knowledge?
● After constructing an image can you see novel aspects of the imagined situation?
● Examples
Imagine seeing these events unfolding…• You hit a baseball. What shape trajectory does it trace? It is
coming towards you: Where would you run to catch it? If you have ever played baseball you would have a great deal of “tacit knowledge” of what to do in such (well studied) cases.
• You drop a rubber ball on the pavement. Tap a button every time it hits the ground and bounces. Plot height vs time.
• Drop a heavy steel ball at the same time as you drop a light ball (a tennis ball), e.g., from the leaning tower of Pisa. Indicate when they hit the ground. Repeat for different heights.
• Take a clear glass containing a colored liquid. Tilt it 45º to the left (counter-clockwise). What is the orientation of the liquid?
Examples to probe your intuition and your tacit knowledge
Time since drop
heig
ht What is responsible for the pattern shown here?
What color do you see when two color filters overlap?
?
Where would the water go if you poured it over a full beaker of sugar?
Is there conservation of volume in your image? If not, why not?
Seeing Mental Images
● Do images have size? ● Can we say that one image is larger than another?● If so, what properties do we expect the smaller/larger
image to have?
Do mental images have size?Imagine a very small mouse. Can you see its whiskers? Now imagine a huge mouse. Can you see its whiskers?
Do this imagery exercise:Imagine a parallelogram like this oneNow imagine an identical parallelogram directly below this one
Connect each corner of the top parallelogram with the corresponding corner of the bottom parallelogram
What do you see when you imagine the connections?Did the imagined shape look (and change) like the one you see now?
Slezak figures
Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.
Slezak figures rotated 90o
Space
P 29
Images and the representation of spatial properties
● We need to understand what it could mean for a representation to be spatial.
● At the very least it must mean that there are constraints placed on the form of the representation that do not apply when the representation is not spatial.
The idea that images are in some sense spatial is an interesting and important claim
● I will return to this claim later because it reveals a deep and ubiquitous error that runs through most (all?) imagery theorizing – by psychologists, neuroscientists and philosophers. This is the error of mistaking descriptive adequacy with explanatory adequacy. Let’s call this conflating, the missing constraint error.
● This is in addition to the two errors I discussed earlier: Ignoring the fact that the task of imagining something is
actually the task of pretending you are seeing it, and The mistaken assumption that certain properties of the world
are properties of the image (the intentional fallacy)
Both vision and visual imagery have some connection to the motor system
• There are a number of experiments showing the close connection between images and motor control* You can get Stimulus-Response compatibility effects
between the location of a stimulus in space and the location of the response button in space,
Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles,
Both these findings provide support for the view that the spatial character of images comes from something being projected onto a concurrently perceived scene and then functioning much as objects of perception. This is the main new idea in Chapter 5 of Things & Places)
an image
imagined
Recall the studies of mental scanning…
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Relative distance on imageLa
tenc
y (s
ecs)
scan image
But the way we compute the time it takes to scan across an image is by imagining something moving across the real perceived display. Without this display, we could not use our time-to-collision computation to compute the time to cross various distances on the image because there are no actual distances on the image! (Pylyshyn & Cohen, 1999)
We showed that the image scanning effect is Cognitively Penetrable
Does this result show that images have spatial properties?Does this result show that images have metrical properties?
Using a concurrently perceived room to anchor FINSTs tagged with map labels
The Spatial character of imagesWhat does it mean to say that images are spatial? ● It means that certain constraints hold among spatial measures (e.g.,
axioms of geometry and measure theory, such as triangle inequality, symmetry of distances, Euclidean axioms, Pythagoras’ theorem…}
● That certain constraints hold among “distances”, that certain relations can be defined among these distances (e.g., ‘between’, ‘farther than’), that Newtonian Physics holds between the terms that are used in explanations (e.g., distances and time).
● That mental images and motor control interact with one another to some degree – so you can “point to” objects in your image.
● Certain visual-motor ‘reflexes’ are automatic or preconceptual They are computed within the encapsulated Visual Module Preconceptual motor control is not sensitive to visual illusions, relative
to control that is computed by the cognitive (‘seeing as’) system.
Mental images as “depictive” representations● “A depictive representation is a type of picture, which specifies the
locations and values of configurations of points in a space.● The space in which the points appear need not be physical but can be
like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is “read” and processed that it comes to function as if it were arranged into an array….
● Depictive representations convey meaning via their resemblance to an object.
● When a depictive representation is used, not only is the shape of the represented parts immediately available to appropriate processes , but so is the shape of the empty space … [and] one cannot represent a shape in a depictive representation without also specifying a size and orientation….”
Form vs Content of images● As in earlier discussion, one must be careful in distinguishing
form from content. We know that there is a difference between the content of images and the content of other (nonimaginal) thought: Images concern sensory appearances while ‘propositions’ can express most* other contents.
● In attributing a special form of representation to images one should ask whether some symbolic system (e.g., sentences of LOT) would not do. Simplicity (Occam’s Razor) would then prefer a single format over two, especially if the one format is essential for representing thoughts and inferences [Fodor, J. A. and Z. W. Pylyshyn (1988). "Connectionism and cognitive architecture: A critical analysis." Cognition 28: 3-71.]
● The most promising contents that might require different forms of representation are those that essentially represent magnitudes. Of the magnitudes most often associated with images are spatial ones.
* There has been a long-standing debate in Artificial Intelligence concerning the advantages of logical formats vs other symbol systems vs something completely difference (procedure).
Thou shalt not cheat• There is no natural law that requires the representations of time,
distance and speed to be related according to the motion equation. You could just as easily imagine an object moving instantly or with constant acceleration or with any motion relation you like, since it is your image!
• There are two possible reason why the observed relation
Actual Time = Representation of distance
Representation of speed
typically holds in an image-scanning task:
1. Because subjects have tacit knowledge that this is what would happen if they viewed a real display, or
2. Because the matrix is taken to be a simulation of a real physical display, as it often is in computer science. Notice that in the second case the explanation for the Reaction Time
comes from the simulated real display and not from the matrix.
The missing constraint in appeals to “space” in both scanning and mental rotation
• What is assumed about the format or architecture of the mental representation in the examples of mental rotation?
• According to philosopher Jesse Prinz (2002) p 118,“If visual-image rotation uses a spatial medium of the kind Kosslyn envisions, then images must traverse intermediate positions when they rotate from one position to another. The propositional [i.e., symbolic] system can be designed to represent intermediate positions during rotation, but that is not obligatory.”
• This is a very important observation, but it is incomplete. One still needs to answer the question: What makes it obligatory that the object must ‘pass through intermediate positions’ when rotating in ‘functional space’, and what constitutes an ‘intermediate position’? These terms apply to the represented world, not to the representation!
The important distinction between architecture and represented content
● It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture as opposed to being due to properties of what is represented (i.e., what the observer tacitly knows about the behavior of what is represented) If it is obligatory only because the theorist says it is, score that
as a free empirical parameter that any theory can assume. This failure of image theories is quite general – all picture
theories suffer from the same lack of principled constraints.
The important distinction between descriptive and explanatory adequacy
● It is important to recognize that if we allow one theory to stipulate what is obligatory without there being a principle that mandates it, then any other theory can stipulate the same thing. Such a theories are unconstrained so they can fit any possible observation – i.e., they are able to describe anything but explain nothing.
● A theory that does not explain why some pattern is obligatory can still be useful the way an organized catalog is useful. It may even list the features according to which it is organized. But it does not give an account of why it is organized that way rather than some other way. To do that it needs to appeal to something constant such as a law of nature or a fixed property of the architecture.
How are these ‘obligatory’ constraints realized?● Image properties, such as size and rigidity are assumed
to be inherent in the architecture (e.g., of the ‘display’)● That raises the question of what kind of architecture
could possibly enforce rigidity of shape? Notice that there is nothing about a spatial display, let alone a
functional space, that makes it obligatory that shape be rigidly maintained as orientation is changed.
Such rigidity could not be a necessary property of the architecture of an image system because we can easily imagine that rigidity does not hold (e.g. imagine a rotating snake!).
There is also evidence that ‘mental rotation’ is incremental, not holistic, and the speed of rotation depends on the conceptual complexity of the shape and the comparison task.
What makes some properties seem “natural” in a matrix but not so natural in a symbolic data structure?
1. A matrix is generally viewed as a two-dimensional structure in which specifying the x and y values (rows and columns) specifies the location of any cell. But that’s just the way it is conventionally viewed. Rows, columns and cells are not actually spatial locations.
2. In a computer there is no requirement that in getting from one cell to another one must pass through any other specified cells nor is there any requirement that there be empty cells between any pairs of cells.
What makes some properties “natural” in a matrix while not so natural in a symbolic data structure?
3. The main reason it is natural to view a matrix as having spatial constraints is that one is tacitly assuming that it represents some space. Then it is the represented space that has the constraints, not the matrix. Notice the subtle succumbing to the intentional fallacy again!
4. Any constraints that the functional space exhibits are constraints extrinsic to the format. Such constraints reside in the external world which the ‘functional space’ represents. But such extrinsic constraints can be added to any model of scanning,
including a propositional one.
What warrants the ‘obligatory’ constraint?But it is no more obligatory that the relation between distance, speed and time hold in functional space than in a symbolic (propositional) representation. There is no natural law or principle that requires it. You could imagine an object moving instantly or according to any motion relation you like, and the functional space would then comply with that motion since it has no constraints of its own. So why does it seem natural for imagined moving objects to traverse
a ‘functional space’ than a sequence of symbolic representations of locations?
There are at least two reasons why a ‘functional space’ might seem more natural than a symbolic representation of space, and both depend on (1) subjective experience and (2) the intentional fallacy.
Where does the obligatory constraint come from? There are at least two reasons why the following equation holds in
the mental image scanning task, even though, unlike in the real vision case, it does not follow from a natural law.
Actual Time = Representation of distance
Representation of speed
1. Because subjects have tacit knowledge that this is what would happen if they viewed a real display, and they understand the task to be one of reproducing properties of this viewing, or
2. Because the matrix is taken to be a simulation of real space. In that case the reason that the equation holds is that it is supposed to be simulating real space and the equation holds in real space. In that case it is not something about the form of the representation that
provides the principled constraint, it’s the fact that it is supposed to be simulating real space which is where the obligation comes from. But the same thing can be done for any form of representation.
Why is it ‘natural’ to assume that functional space is like real space?
There are several reasons why a functional space, such as a matrix data structure, appears to have natural spatial properties (e.g., distances, size, empty places):
1. Because when we think of functional space, such as a matrix, we think of how we usually interpret it.
A matrix does not intrinsically have distance, empty places, direction or any other such property, except in the mind of the person who draws it or uses it!
Moving from one cell to another does not require passing through intermediate cells unless we stipulate that it does. The same goes for the concept of ‘intermediate cell’ itself.
Why is it ‘natural’ to assume that functional space is like real space?
2. Because when we think of a functional space, such as a matrix, we think of it as being a way of simulating real space in the model – making it more convenient to build the model which otherwise would require special hardware This is why we think of some cells as being ‘between’ others and
some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating real space.
In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are simulating.
• This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space
• All the reasons why the assumption of real brain space cannot be sustained in explanations of mental imagery phenomena apply to this version of ‘functional space.’
Why is it ‘natural’ to assume that functional space is like real space?
2. Because what we really want to claim is that images are displayed on a real spatial surface – a blackboard. But to model this we would need to build a hardware display. {An easier way to do this is simply to claim explicitly that there is a display or even simulate one using software (such as Kosslyn, et al. (1979) claim to have done*)}. This allows us to view some cells as being ‘between’ others and
some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating or standing in for a real spatial display board or screen.
In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are claiming and simulating. This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space.
Functional space and explanatory power• There is a notion of explanatory power that needs to be kept in
mind. It is best illustrated in terms of models that contain empirical parameters, as in fitting a polynomial curve to data.
• The general fact about fitting a model to data is that the fewer parameters that need to be estimated from the data to be fitted, the more powerful the explanation. The most powerful explanation is one that does not have to use the to-be-fitted data to tune the model.
• In terms of the current example of explaining results of experiments involving mental imagery, appealing to a “functional space” leaves open an indeterminate number of empirical parameters, so it provides a very weak (or vacuous) explanation.
• A literal (brain) space, on the other hand, is highly constrained since it must conform to Euclidean axioms and Newtonian physics – otherwise it would not be the space of natural science. But that kind of space implies that images are displayed on a surface in the brain and while that is a logical possibility it is not an empirical one
Explanation and Description• Another way to look at what is going on is to think about the
difference between a description and an explanation. The two ways of characterizing a set of phenomena appear similar – they both speak of how things are and how they change (think of the Code Box example).
• But a description of a system’s behavior can apply to many different types of system with different mechanisms and different causal properties. And the same mechanisms can also produce very different behaviors under different circumstances. Although a general statement of what constitutes scientific explanation and how it differs from description has a long and controversial history, the simple Code Box example will suffice to suggest the distinction I have in mind.
Cognitive Penetrability again● A description states the observed generalizations (the
observed patterns of behavior). The explanation goes beyond this. The difference is related to the question of how mutable a set of generalizations are and what types of effects can lead to changes in these generalizations.
● Causal accounts tend to have a longer time scale and when they change they tend to change according to different sorts of principles than those that describe the patterns.*
Notice that we have come back to the criterion of cognitive penetrability. According to this way of looking at the question of explanatory adequacy, an theory meets the criteria of explanatory adequacy if it describes the architecture of the system and its operation.
A note about time scales and types of changes
* “Causal accounts tend to have a longer time scale and when they change they tend to change according to different sorts of principles than those that describe the patterns.”
Consider the Code Box example. Changes that are not architectural tend to occur rapidly –
different patterns are observed simply because different topics or words or even languages might be transmitted.
Changes that are architectural require altering which letters or other symbols are transmitted (e.g., they may be numerals) or changing whether the outputs consist of short and long pulses that are interpretable as Morse Code. They require what we might think of as “rewiring”.
A different way of approaching the question of spatial representation
· I offer a provisional proposal that preserves some of the advantages of the global spatial display, but assumes that the relevant spatial properties are in the perceived world and can be accessed if we have the right access mechanisms for selecting and indexing objects in the perceived world
· Let’s call this the Index Projection Hypothesis because it suggests that mental objects are somehow projected onto and associated with perceived objects in real space But this proposal is very different from image-projection because
only a few object-labels are projected – not the rich visual properties suggested by the phenomenology
The Image Projection Hypothesis This projection hypothesis relies on the spatial locations of
objects in the concurrently perceived world to meet the conditions outlined earlier. It rests on two assumptions:1. We have a system of “pointers” (viz, the FINST perceptual
index mechanism to be described) by which a small number (n≤4) of objects in the world can be selected and indexed. Indexes provide demonstrative references to individual targets qua individuals, that keep referring to these objects despite changes in their location or any other properties.
2. When we perceive a scene that contains indexed objects, our perceptual system is able to treat those objects as though they were assigned unique labels. Thus our perceptual system is able to detect configurational properties among the indexed objects.
The index projection hypothesis (2) The hypothesis claims that the subjective impression that we
have access to a panorama of detailed high-resolution perceptual information is illusory. What we have access to is only information about selected or indexed objects. We have the potential to obtain other information from more of
the scene through the use of our system of perceptual indexes. This is the basic insight expressed in the “world as external memory” slogan or the “situated cognition” approach.
In reasoning using mental images we may assign indexes to perceived objects based on our memory of (or our assumptions about) where certain ‘mental’ objects are located
But notice that the memory representation is itself not used in spatial reasoning and therefore need not meet the spatial constraints listed earlier – it can be in some general LOT
Examples of the projection hypothesis
· To illustrate how the projection hypothesis works, first consider index-based projection in the visual modality, where indexes can convert some apparently mental-space phenomena into perceived-space phenomena (more on the non-visual case later)
· Examples from some ‘mental imagery” experiments Mental scanning (Kosslyn, 1973) Mental image superposition (Podgorny& Shepard, 1978) Visual-motor adaptation (Finke, 1979) S-R compatibility to imagined locations (Tlauka, 1998)
Studies of mental scanningOften cited to suggest that spatial representations
are literally spatial and have metrical properties
Distance on image
Tim
e to
“see
” fea
ture
on
imag
etree
tower
windmill
beach
steeple
X
X
X
X
X
X
Brain image or index-based projection?
· A way to do this task: Associate places on the memorized map with
objects located in the same relative locations in the world that you perceive (e.g., the room you are in)
Move your attention or gaze from one place to another as they are named
Using a perceived room to anchor FINSTs tagged with map labels
Using vision with selected ‘labeled’ objects· If you ‘project’ the pattern of map places by indexing objects in the
room in front of you that correspond to the memorized relative locations, then you can scan attention from one such indexed object to another. The relation time = distance speed holds because the space you are scanning is the real physical space in the room.
· You can also use the indexed objects to infer configurational properties you may not have noticed, despite memorizing the location of objects. e.g. Which 3 or more places on the map are collinear? Which place on the map is furthest North (or South, East, West)? Which 3 places form an isosceles triangle?
· Such configurational consequence can be detected as opposed to logically inferred, so long as they involve only a few places, because the visual system can examine the indexed objects in the scene
Connecting Images and Motor actions
Images and visual-motor phenomena S-R Compatibility / Simon effect Finke’s imagined wedge goggles Harry’s subitizing-by-pointing
Both vision and visual imagery have some connection to the motor system
· There are a number of experiments showing the close connection between images and motor control* You can get Stimulus-Response compatibility effects
between the location of a stimulus in space and the location of the response button in space,
Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles,
Both these findings provide support for the view that the spatial character of images comes from something being projected onto a concurrently perceived scene and then functioning much as objects of perception. This is the main new idea in Chapter 5 of Things & Places)
an image
imagined
This story is plausible for visual cases, but how does it work without vision (e.g., in the
dark)?
· We must rely on our remarkable capacity to orient to (point to, navigate towards, …) perceived or recalled objects (including proprioceptive ‘objects’) in space without vision
Call this general capacity our spatial sense· How can the projection hypothesis account for this
apparently world-centered spatial sense without assuming a global allocentric frame of reference? Answer: Just as it does with vision, by anchoring
represented objects to (non-visually) perceived objects in the world
The spatial sense and the projection hypothesis
· Indexing non-visual ‘objects’ must exploit auditory and proprioceptive signals, and perhaps even preparatory motor programs (the ‘intentional’ frame of reference proposed by Anderson & Bruneo, 2002; Duhamel, Colby & Goldberg, 1992) Is there some special problem about proprioceptive
inputs that makes them different from visual inputs?
Is there a problem with proprioceptive inputs indexing objects the way visual indexes do?
· Unlike visual objects, proprioceptive “objects” are not fixed in an allocentric frame of reference – or are all objects the same? Notice that in vision and audition, even though static objects are
fixed in an allocentric frame of reference, they nonetheless move relative to sensors, so their location in an allocentric frame must be updated as the proximal pattern moves (Andersen, 1999; Stricanne, Anderson
& Mazzoni, 1996) The neural implementation of FINST indexes in vision requires an
active updating process of some kind Maybe the same updating operation can also yield the sense of
“same location in space” for proprioceptive ‘objects’ There are good reasons to think that proprioceptive signals may also
be given in an allocentric frame of reference! (Yves Rosetti)
What is the real problem of our sense of space?· In order to solve the problem of how we index objects in the
world using proprioceptive inputs we need to solve the problem of how we recognize two such inputs as corresponding to actions (e.g., reaching) towards the same object in the world
· This is the problem of the equivalence of movements, or of proprioceptive inputs, corresponding to the same object – it is the problem that Henri Poincaré recognized as the central problem of understanding our sense of space (in Poincaré “Why space has three dimensions” Les Dernier Penseés, 1913)
· Solving the equivalence problem would solve the problem of coordinating signals across frames of reference That’s why mechanisms of coordinate transformation are of
central importance – they generate the relevant equivalences!
Assumption: Coordinate transformations are the basis for the illusory “global frame of reference”· A coordinate transformation operation takes a representation of
an object relative to one coordinate system – say retinal coordinates – and produces a representation of that object relative to another frame of reference – say relative to the location of a hand in proprioceptive or kinematic coordinates
· Coordinate transformations define equivalence classes of proprioceptive inputs that correspond to actions (e.g., reaching, eye movements) towards the same object in space Such transformations are well-known and ubiquitous in the brain
(especially in posterior parietal cortex and superior colliculus) A consequence of these mechanisms is that, as (Colby &
Goldberg, 1999) put it, “Direct sensory-to-motor coordinate transformation obviates the need for a single representation of space in environmental coordinates” (p319)
Coordinate transformations need not transform all points in a given frame of reference
· Coordinate transformations need not transform all points (including points in empty space) or all sensory objects: Only a few selected objects need to be transformed at any one time The computational complexity of coordinate transformations
can be made tractable by only transforming selected objects (as is done by matrix operations in computer graphics)
This idea is closely related to the conversion-on-demand hypothesis of Henriques et al. (1998) and Crawford et al. (2004). In the Henriques et al COD proposal, visual information about
object locations is held in a gaze-centered frame of reference and objects are converted to motor coordinates when needed
Coordinate transformations define equivalence classes of gestures which individuate proprioceptive objects just the way that FINST indexes do in vision
· Coordinate transformations compute equivalence classes of proprioceptive signals {s} corresponding to distinct motor actions to individual objects in real space. The equivalence class is given by: s ≡ s′ iff there is a coordinate transformation between S ↔ S′
· As in the visual case, only a few such equivalence classes are computed, corresponding to a few distal objects that were selected and assigned an index, as postulated in FINST Theory
· We can thus bind several objects of thought to objects in real space (including sensory ‘objects’ perceived in proprioceptive modalities) This can explain the ‘spatial’ character of spatial representations,
just the way they did in the purely visual cases illustrated earlier
Mental imagery and neuroscience● Neuroanatomical evidence for a retinotopic display in
the earliest visual area of the brain (V1)● Neural imaging data showing V1 is more active during
mental imagery than during other forms of thought The form of activity differs for small vs large images in the
way that it differs when viewing small and large displays● Transcranial magnetic stimulation of visual areas
interferes more with imagery than other forms of thought
● Clinical cases show that visual and image impairment tend to be similar (Bisiach, Farah)
● More recently psychophysical measures of images shows parallels with comparable measures of vision, and these can be related to the receptive cells in V1
Status of different types of evidence in the debate about the form of mental images
● Phenomenology. Is it epiphenominal?● Neuroscience evidence for:
Role of vision Type and location of neural structures underlying images Are the neural mechanisms for early vision used in imagery? Does neuroanatomy provide evidence for the nature of “depictive”
representations.
Neuroscience has shown that the retinal pattern of activation is displayed on the surface of the cortex
Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218, 902-904.
There is a topographical projection of retinal activity on the visual cortex of the cat and monkey.
Problems with drawing conclusions about the nature of mental images from neuroscience data
1. The capacity for imagery and for vision are known to be independent. Also all imagery results are observed in the blind.
2. Cortical topography is 2-D, but mental images are 3-D – all phenomena (e.g. rotation) occur in depth as well as in the plane.
3. Patterns in the visual cortex are in retinal coordinates whereas images are in world-coordinates Your image stays fixed in the room when you move your eyes or turn
your head or even walk around the room
4. Accessing information from an image is very different from accessing it from the perceived world. Order of access from images is highly constrained. Conceptual rather than graphical properties are relevant to image
complexity (e.g., mental rotation).
Problems with drawing conclusions about mental images from the neuroscience evidence
5. Retinal and cortical images are subject to Emmert’s Law, whereas mental images are not;
6. The signature properties of vision (e.g. spontaneous 3D interpretation, automatic reversals, apparent motion, motion aftereffects, and many other phenomena) are absent in images;
7. A cortical display account of most imagery findings is incompatible with the cognitive penetrability of mental imagery phenomena, such as scanning and image size effects;
8. The fact that the Mind’s Eye is so much like a real eye (e.g., oblique effect, resolution fall-off) should serve to warn us that we may be studying what observers know about how the world looks to them, rather than what form their images take.
Problems with drawing conclusions about mental images from the neuroscience evidence
9. Many clinical cases can be explained by appeal to tacit knowledge and attention The ‘tunnel effect’ found in vision and imagery (Farah) is likely due to
the patient knowing what things now looked like to her post-surgery Hemispatial neglect seems to be a deficit in attention, which also
explains the “representational neglect” in imagery reported by Bisiach A recent study shows that imaginal neglect does not appear if patients
have their eyes closed. This fits well in the account I will offer in which the spatial character of a mental images derives from concurrently perceived space.
10. What if colored three-dimensional images were found in visual cortex? What would that tell you about the role of mental images in reasoning? Would this require a homunculus?
Should we welcome back the homunculus?
● In the limit if the visual cortex contained the contents of ones conscious experience in imagery we would need an interpreter to “see” this display in visual cortex
● But we will never have to face this prospect because many experiments (including ones by Kosslyn) show that the contents of mental images are conceptual (or, as Kosslyn puts it, contain “predigested information”).
● And finally, it is clear to anyone who thinks about it for a few seconds that you can make your image do whatever you want and to have whatever properties you wish. There are no known constraints on mental images that cannot be attributed
to lack of knowledge of the imagined situation (e.g., imagining a 4D cube). All currently claimed properties of mental images are cognitively
penetrable.
Explaining mental scanning, mental rotation and image size effects in terms of “functional space”
● When people are faced with the natural conclusion that the “iconic” position entails space (as in scanning and size effects) they appeal to “functional space”
● A Matrix in a computer are often cited as an example● Consider a functional space account of scanning or of
mental rotation: Why does it take longer to scan a greater distance in a
functional space? Why does it take longer to rotate a mental image a greater
angle?
Why do conscious contents misguide us?
· The contents that appear in our conscious experience almost always concern what we are thinking about and not what we are thinking with – with content rather than form.
· The processes that we see unfolding in our mind are almost always attributable to what we know about how the things we are thinking about would enfold, rather than being due to laws that apply to our cognitive architecture. cf Code Box
· We should take seriously the possibility that (almost) all constraints and law-like behaviors of objects of our experience are constraints due to our knowledge rather than of the mental architecture. Notice the mental rotation example and the mistake that Jesse Prinz makes.
This is what our conscious experience suggests goes on in vision…
Kliban
This is what the demands of explanation suggests must be going on in vision…
Imagine this shape rotating slowly
When you make it rotate in your mind, does it retain its rigid 3D shape without re-computing it? Would you expect to ‘see’ this kind of information process? Does the experience in this case reassure you that the rotation was smooth? Are you sure something rotated?
Is this how it looked to you?
What about the evidence of conscious experience? Is it irrelevant?
· I have often been accused of relegating conscious experience to the category of epiphenomena – something that accompanies a process but does not itself have a role in its causation.
· But images are not illusory or unnatural, they are quite real. The problem is that people have theories of the causal or information-processing that underlies these phenomena and these theories are almost always false because they assume a simple and obvious mapping from the experience to the computational or brain states so that we can see the form of the representation.
· The connection between conscious experience and information processing is deeply mysterious (it’s the mind-body problem). But one thing we do know is that the sequence of events that unfolds when we imagine something does not reveal causal laws because there are no causal laws of conscious states as conscious states.
The important distinction between architecture and represented content
· It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture, as opposed to being due to properties of what is represented (i.e., what the observer tacitly knows about the behavior of what is represented) If it is obligatory only because the theorist says it is, then score
that as a free empirical parameter (a wild card) If we allow one theory to stipulate what is obligatory without
there being a principle that mandates it, then any other theory can stipulate the same thing. Such theories are unconstrained and explain nothing.
This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraints
The important distinction between architecture and represented content
· It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture, as opposed to being due to properties of what is represented (i.e., what the observer tacitly knows about the behavior of what is represented) If it is obligatory only because the theorist says it is, then score
that as a free empirical parameter (a wild card) If we allow one theory to stipulate what is obligatory without
there being a principle that mandates it, then any other theory can stipulate the same thing. Such theories are unconstrained and explain nothing.
This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraints
How are these ‘obligatory’ constraints realized?· Image properties, such as size and rigidity, are assumed to be
inherent in the system of representation (its architecture)· That raises the question of what kind of architecture could
possibly enforce rigidity of shape? Notice that neither a spatial display nor a functional space make it
obligatory that shape be rigidly maintained as orientation is changed. Only certain physical properties can explain rigidity.
Such rigidity could not be part of the architecture of an imagery system because we can easily imagine situations in which rigidity does not hold (e.g. imagine a rotating snake!).
There is also evidence that ‘mental rotation’ is incremental, not holistic, and the speed of rotation depends on the conceptual complexity of the shape and the comparison task.
Aside: What can we conclude from the contents of conscious experience?
What should we conclude about the role of conscious appearance in cognitive science?
We can’t do without it:· When we ask which line appears longest or which version of
an ambiguous figure we see we are asking for a report of conscious content. Much of what we know of how vision works depend on such evidence.
What should we conclude about the role of conscious appearance in cognitive science?
But we can’t accept it at face value:· If you ask yourself “what am I thinking …?” you raise one of the most
mysterious problems in the philosophy of mind. You could not possibly give the correct, or at least an unproblematic answer because it is quite possible that you don’t know what you are thinking! Your experience when thinking is as of speaking or of perceiving – what else could it be?
· Every imagined sentence is infinitely ambiguous and understanding it presupposes a huge amount about the context of its utterance. Your experience of speaking cannot be the same as your thought; your thought precedes your imagined speech and is just one of the contents that are expressed. The sentences you imagine, as the sentences you speak, follow Gricean conversational maxims – e.g., don’t state what your listener already knows, state only what is relevant, state only as much as necessary to convey your intentions. Something similar is true of imaging.
Examples of conscious evidence that leads to false conclusions
· Sentence example· Example of mental diagram and what it must assume
Do we (or can we) experience our thoughts?
· There is much that can be said on this topic, but there is no time for it here. But see: Grice, H. P. (1975). "Logic and Conversation." Syntax and
Semantics 3: 41-58. Hurlburt, R. T., & Schwitzgebel, E. (2007). Describing
Inner Experience? Cambridge, MA: MIT Press. Schwitzgebel, E. (2011). Perplexities of Consciousness.
Cambridge, MA: MIT Press.
What should we conclude about the role of conscious appearance in cognitive science?
But we can’t accept it at face value:· If you ask yourself “what am I thinking …?” you raise one of the most
mysterious problems in the philosophy of mind. You could not possibly give the correct, or at least an unproblematic answer because it is quite possible that you don’t know what you are thinking! Your experience when thinking is as of speaking or of perceiving – what else could it be?
· Every imagined sentence is infinitely ambiguous and understanding it presupposes a huge amount about the context of its utterance. Your experience of speaking cannot be the same as your thought; your thought precedes your imagined speech and is just one of the contents that are expressed. The sentences you imagine, as the sentences you speak, follow Gricean conversational maxims – e.g., don’t state what your listener already knows, state only what is relevant, state only as much as necessary to convey your intentions. Something similar is true of imaging.
But there are examples of solving geometry problems easily with imagery
• There are many problems that you can solve much more easily when you imagine a layout than when you do not.
• In fact many instances of solving problems by imagining a layout that seem very similar to how would solve them if one had pencil-and-paper.
• The question of how pictures, graphs, diagrams, etc help in reasoning is very closely related to the question of how imagined layouts function in reasoning. That is not in question. What is in question is what happens in either the visual or imagined cases and how images can benefit from this processes even though there is no real diagram.
How do real visual displays help thinking? Why do diagrams, graphs, charts, maps, icons and other visual
objects help us to reason and to solve problems? The question why visual aids help is nontrivial and Seeing &
Visualizing, chapter 8 contains some speculative discussion, e.g., they allow the visual system to:• make certain kinds of visual inferences• make use of visual demonstratives to offload some of the memory load• Capitalize on the fact that the displays embody the axioms of measure
theory and of geometry (which are then inherited by thought) The big question is whether any of these advantages carry over to
imaginal thinking! Do mental images have some (or any) of the critical properties that make diagrams helpful in reasoning?
Visual inferences?● If we recall a visual display it is because we have encoded
enough information about its visual-geometrical properties that we can meet some criteria, e.g., we can draw it. But there are innumerably many ways to encode this information that are sufficient for the task (e.g. by encoding pairwise spatial relations, global spatial relations, and so on). For many properties the task of translating from one form to another is much more difficult than the task of visually encoding it – the translation constitutes visual inference.
● The visual system generalizes from particular instances as part of its object-recognition skill (all recognition is recognition-as and therefore assumes generalization from tokens to types). It is also very good at noticing certain properties (e.g., relative sizes, deviations from square or circle, collinearity, inside, and so on). These capabilities can be exploited in graphical layouts.
Memorize this map so you can draw it accurately
From your memory:• Which groups of 3 or more locations are collinear?• Which locations are midway between two others?• Which locations are closest to the center of the island?• Which pairs of locations are at the same latitude?• Which is the top-most (bottom-most) location?
If you could draw the map from memory using whatever properties you noticed and encoded, you could easily answer the questions by looking at your drawing – even if you had not encoded the relations in the queries.
Draw a rectangle. Draw a line from the bottom corners to a point on the opposite vertical side. Do these two lines intersect? Is the point of intersection of the two lines below or above the midpoint? Does it depend on the particular rectangle you drew?
A B
CD
m m’
y
x
Which properties of a real diagram also hold for a mental diagram?
● A mental “diagram” does not have any of the properties that a real diagram gets from being on a rigid 2D surface.
● When you imagine 3 points on a line, labeled A, B, and C, must B be between A and C? What makes that so? Is the distance AC greater than the distance AB or BC?
● When you imagine drawing point C after having drawn points A and B, must the relation between A and B remain unchanged (e.g., the distance between them, their qualitative relation such as above or below). Why?
● These questions raise what is known as the frame problem in Artificial Intelligence. If you plan a sequence of actions, how do you know which properties of the world a particular action will change and which it will not, given that there are an unlimited number of properties and connections in the world?