the cognitive neuroscience of mental imagery (kosslyn 1996)

8/17/2019 The Cognitive Neuroscience of Mental Imagery (Kosslyn 1996)

1/10

~ P e r g a m o n

Neuropsychologia,Vol. 33, No. 11, pp. 1335-1344, 1995

ElsevierSeienoeLtd

Printed n G reat Britain

0028-3932(95)00067--4

T H E C O G N I T I V E N E U R O S C I E N C E O F M E N T A L I M A G E R Y

S . M . K O S S L Y N ,* ~f M . B E H R M A N N : ~ a n d M . J E A N N E R O D §

t D e p a r t m e n t o f P s yc h o l o g y , H a r v a r d U n i v e r si t y , C a m b r i d g e , M a s s a c h u s e t t s , U . S .A . ;

: ~ D e p a rt m e n t o f P s y c h o l o g y , C a r n e g i e M e l l o n U n i v e r s i ty , P i t t s b u r g h , P e n n s y l v a n i a

an d §Vis ion e t M ot r i c i t6 , 16 , avenue du Doy en l_~p ine , 69500 B ro n , F ranc e

Received 12 November 1994; accepted 10 January I995

Thi s spec i a l i s sue i s devo t ed t o r ecen t work on men ta l image ry . To ge t a s ense fo r t he

s u b j e c t m a t t e r , a n s w e r t h e f o l l o w i n g q u e s t i o n s : H o w m a n y w i n d o w s a r e t h e r e i n y o u r

l iv ing room ? W ha t socks d id yo u wea r ye s t e rday? I s t he r e r oom in you r r e f r i ge r a to r f o r a

l it re o f mi lk? These ques t i ons t yp i ca l l y evoke v i sua l men t a l image ry . Fo r exam ple , when

answer ing t he f ir s t, peop l e u sua l l y v i sua l iz e the room , and t hen s can ove r e ach wa l l,

s ee ing each w in dow wi th t he i r m ind s eye . S imi la r ly , i f a sked wh e the r a c a t s m eow has

a h ighe r p i t ch t han t he sou nd o f a b lende r , mo s t peop l e r epo r t hea r i ng t he sounds i n

the i r m ind s e a r . Such t a sks evok e aud i t o ry m en ta l image ry . V i sua l men t a l ima ge ry i s

s ee ing i n the absence o f t he app rop r i a t e imm ed ia t e s enso ry i npu t , aud i t o ry men ta l

image ry i s hea r i ng i n t he absence o f t he app ro p r i a t e imm ed ia t e s enso ry i npu t , and so

on . Im age ry i s d i s t inc t f r om pe rcep t i on , w h ich i s t he r eg i s t ra t i on o f phys i ca l ly p r e sen t

st imuli .

The example s i l l u s t r a t e t ha t image ry p l ays an impor t an t r o l e i n memory and spa t i a l

reasoning , bu t as we sha l l see , these func t ions mere ly sc ra tch the sur face . Imagery

a lso p lays a ro le in abs t rac t reasoning , sk i l l l ea rn ing , and language comprehens ion .

On e o f the m ajo r ins ights a bo ut im agery i s tha t- -- -even w i th in a s ingle senso ry

m o d a l i t y l i t i s no t a sing le , und i f f e ren t i a t ed ab i li t y . Ra the r , a s t he open ing examp le

sugges t s, image ry i nvo lves a h os t o f p roces se s wo rk ing t oge the r. The image o f on e s

l i v ing room mus t be fo rmed a t t he ou t s e t , i t t hen mus t be s canned and i n t e rp r e t ed ,

and m us t be ma in t a ined wh i l e t h is is occu r r ing . M oreov e r , i n ma ny s i t ua t i ons t he

i m a g e d o b j e c t m u s t b e t r a n s fo r m e d . F o r e x a m p l e , w h e n a s k e d w h e t h e r f r o g s h a v e

s tu bb y g r een t a il s, peop l e o f t en r epo r t men t a l l y r o t a t i ng t he am ph ib i an and l ook ing

a t i t s pos t e r i o r qua r t e r s .

I n t e r e s t i n me n ta l imag e ry ha s waxed and w aned fo r cen tu r ie s , and i s now un de rgo ing a

r e su rgence . U n l ike p r ev ious pe r i ods o f en thus i a sm fo r t he t op ic , t he p r e sen t one i s no t

d r i ven by ph i l o soph i ca l conce rns o r c l a ims ba sed on i n t ro spec t ion . Ra the r , t he p r e sen t

* A d d r e s s f o r r e p r i n t r e q u e st s : D e p a r t m e n t o f P s y c h o l og y , H a r v a r d U n i v e r s it y , 33 K i r k l a n d S t r e e t , C a m b r i d g e ,

M A 02138 , U .S .A .

1335


2/10

1336 S.M . KOSSLYN M. BEHRMANN and M. JEANNEROD

interest has been fueled by the emergence of new methods and new ways to conceptualize

imagery. As illustrated in this special issue, the confluence of these factors has produced a

new wave of discoveries not only about the nature and function of imagery, but also about

its neural bases. Indeed, because imagery has been shown to share mechanisms with like-

modality perception, theories of imagery have been able to stand on the shoulders of

theories of perception. This state o f affairs has resulted in the somewhat ironic fact that an

area that was considered beyond the pale 20 years ago is now of the first cognitive domains

to be firmly rooted in the brain.

Interest in imagery can be traced back at least to the time of Plato, who thought that

memories were based on images; on his view, memories are carved into the mind much like

pictures can be carved on a wax tablet. He even considered individual differences, likening

them to differences in the purity of the wax and in the ease of carving figures into it.

Interest in imagery continued on for centuries. For example, the British Associationists

conceived of thought itself as sequences of mental images, and the original psychologies of

Wundt and James placed considerable emphasis on imagery.

Imagery fell from grace, however, when psychology became obsessed with method. The

behaviorists stressed that the scientific method requires that one study phenomena that are

publicly accessible. This, of course, was not what was going on in other sciences, such as

physics--which were quite happy to draw inferences about unobservable phenomena by

studying their footprints (e.g. tracks in cloud chambers). But more than this, the

behaviorists argued effectively that it was not clear how to conceive of mental images.

Pictures in the head? Who looks at them? In retrospect, it seems that the eclipse of research

on imagery was a result of three events: A lack of sophisticated methodologies for studying

mental events, impoverished theoretical constructs, and the apparent success of the

behaviorist approach. This last factor was critical because it justified a shift in the way the

field of psychology was characterized. Whereas James conceived of psychology as the

study of mental life, Watson declared that it should be the study of behavior. There is no

place for mental events, let alone those that do not have distinctive behavioral sequeUae, in

a field that studies behavior

qu

behavior.

Imagery came into its own again in the early 1970s, riding the tide of the cognitive

revolution [see 10]. As the limitations of behaviorism became apparent, researchers once

again became receptive to theorizing about internal events. Three kinds o f work emerged

from the application of the methods of cognitive psychology to the study of imagery. First,

Paivio and his collaborators demonstrated that people can learn and remember highly

imageable materials better than abstract materials [e.g. see 16]. Second, Segal and her

colleagues used signal detection paradigms to show that imagery interferes more with like-

modality perception than with different-modality perception (e.g. visual imagery interferes

with visual perception more than with auditory perception and vice versa for auditory

imagery [18]). And third, Shepard and his collaborators [e.g. 21] demonstrated tha t images

act as surrogate objects during reasoning. For example, when subjects compared two

objects (e.g. three-dimensional block figures) to determine whether they had the same

shape or were mirror images, they required more time when the objects had to be mentally

rotated greater amounts to be aligned. In spite of the fact that images are not actual

objects that must obey the laws of physics, objects in images often behave like actual

objects.

A second innovation in imagery research took place during the 1970s, but this one was

conceptual, not methodological. Imagery was conceptualized as a type of data structure,


3/10

T H E C O G N I T IV E N E U R O S C I E N C E O F M E N T A L I M A G E R Y 3 3 7

w h i c h w a s e m b e d d e d i n a n i n f o r m a t i o n p r o c e s s i n g s y s t e m [ e.g . 2, 1 2 ] . T h i s p e r s p e c t iv e l e d

t o a h o s t o f e x p e r im e n t s t h a t w e r e d e s i g n e d t o r e v e a l s t r u c t u r a l p r o p e r t ie s o f im a g e s a s

s p e ci fi c t y p e s o f d a t a s t r u c tu r e s . A d e b a t e e n s u e d a b o u t t h i s is s u e, w h i c h q u i c k l y b o g g e d

d o w n w h e n r e s e a r c h e r s r e a li z e d th e l i m i t a t i o n s o f p u r e l y b e h a v i o r a l a p p r o a c h e s [ e.g . se e

1].

T h e d i ff i cu l ti e s i n c h a r a c t e r i z i n g t h e w a y i m a g e s a r e r e p r e s e n t e d a n d p r o c e s s e d s e t th e

s t a g e f o r t h e m o s t r e c e n t d e v e l o p m e n t in t h i s fi el d: T h e c o g n i t iv e n e u r o s c i e n c e a p p r o a c h .

R e s e a r c h e rs b e g a n t o u s e n e u r o p s y c h o l o g i c a l d a t a t o i n f o r m t h e o r i es o f th e s t r u c t u r e o f

t h e p r o c e s s i n g s y s t e m [ e.g . 7]. A n d m o r e r e c e n t l y , s u c h d a t a h a v e b e e n u s e d t o c h a r a c t e r i z e

t h e n a t u re o f t h e r e p re s en t a t i o n i t s e l f [ e.g . 1 3 ].

A s e a c h n e w a p p r o a c h h a s m a t u r e d , i t h a s s u p p l e m e n t e d - - n o t r e p l a c e d - - t h e e a r l i e r

a p p r o a c h e s . T h u s , n e w d e v e l o p m e n t s h a v e f l e s h e d o u t t h e f i e l d , w h i c h c a n n o w b e

c h a r a c t e r i z e d a s e n c o m p a s s i n g a n u m b e r o f r e l a t i v e l y d i s t i n c t d o m a i n s .

R E S E A R C H D O M A I N S

T h e s t u d y o f im a g e r y h a s n o w d e v e l o p e d i n t o a f i e ld i n it s o w n r i g h t . I n t h i s s e c ti o n w e

b r i ef l y r e v ie w t h e m a j o r a r e a s o f r e s e a r c h , w h i c h w e h a v e g r o u p e d i n t o f o u r m a j o r

c a t e g o r i e s : L e a r n i n g a n d m e m o r y ; p e r c e p t i o n a n d a c t i o n ; i n f o r m a t i o n p r o c e s s i n g ; a n d

r e a s o n i n g . I n a l l c a s e s , r e s e a r c h e r s b e g a n b y c h a r a c t e r i z i n g t h e f u n c t i o n i t s e l f , a n d o n l y

t h e n s o u g h t t h e n e u r a l b a s e s o f th e s e f u n c t i o n s . W e d o n o t b e li e ve t h a t i t is w is e to t r e a t

t h e n e u r o p s y c h o l o g y o f i m a g e r y a s a d i s t in c t fi el d o r a r e a ; r a t h e r , i t i s a n i n t e g r a l p a r t o f

t h e s t u d y o f i m a g e r y m o r e g e n e r a l ly . N e u r o p s y c h o l o g i c a l f in d i n g s a n d t h e o r i e s i l lu m i n a t e

t h e n a t u r e o f i m a g e r y r e p r e s e n t a t i o n s a n d p r o c e ss e s , a s w e l l a s t h e f u n c t i o n s o f i m a g e r y .

W e s e e k to a v o i d t h e t r a p t h a t h a s b e f al le n s o m e p a r t s o f d e v e l o p m e n t a l p s y c h o lo g y ,

w h e r e t h e o r i e s o f t h e d e v e l o p m e n t o f c o g n it iv e f u n c t i o n m a k e l it tl e c o n t a c t w i t h m o r e

s o p h i s t i c a t e d t h e o r i e s o f t h o s e f u n c t i o n s t h a t h a v e b e e n f o r m u l a t e d b y c o g n i t i v e sc i en t is t s.

Learni ng and m em ory

W e c a n d i s t i n g u i s h t h r e e r o l e s f o r i m a g e r y i n l e a r n i n g a n d m e m o r y .

Imagery and learning

T h e a n c i e n t G r e e k s d i s c o v e re d t h a t o n e s m e m o r y f o r a s et o f

o b j e c ts c o u l d b e g r e a t l y e n h a n c e d i f o n e v i s u a l iz e d t h e m i n t e r a c t i n g i n s o m e w a y . O n c e o n e

h a s v i s u a l iz e d a s c e n e a n d e n c o d e d i t i n t o m e m o r y , i t th e n c a n b e r e c a ll e d in a n i m a g e . F o r

e x a m p l e , i f y o u w a n t t o r e m e m b e r a s e t o f e rr a n d s , y o u c a n v i s u al iz e y o u r s e l f d o i n g t h e m

i n o r d e r . L a t e r y o u c a n v i s u al iz e t h e fi rs t p a r t o f t h e s e q u en c e , a n d s e e y o u r s e l f p r o g r e s s

t h r o u g h t h e e n t ir e s e q u e n c e , w h i c h w i ll h e lp y o u t o r e c al l th e e n t i r e s e t. B o w e r [ 4] s h o w e d

t h a t m u c h o f t h e m n e m o n i c p o w e r o f i m a g e r y c o m e s f r o m i ts a b i li ty to r e p re s e n t

a s so c i a t i o n s b e t w een d i s t i n c t o b j ec t s .

I m agery and m em ory

E v e n i f t h e y d i d n o t s e t o u t t o s t o re i m a g e s i n t e n t i o n a l l y , p e o p le

o f t e n w i l l u s e i m a g e r y l a t e r to r e c a ll i n f o r m a t i o n . I m a g e r y i s u s e d s p o n t a n e o u s l y t o r e c a l l

t h e sh ap e , co l o r , s i z e , an d t ex t u re o f o b j ec t s o r sp a t i a l r e l a t i o n s i n s cen es t h a t : ( a ) h av e

b e e n c o n s i d e r e d r e l a t i v e l y i n f r e q u e n t l y , a n d ( b ) c a n n o t e a s i l y b e i n f e r r e d f r o m f a c t s

a s s o c i a t e d w i t h a s u p e r o r d i n a t e c a t e g o r y o r f r o m v e r b a l m a t e r i a l . F o r e x a m p l e , i m a g e r y is

u s e d t o d e t e r m i n e w h e t h e r a G e r m a n S h e p h e r d d o g s e a r s p r o t r u d e a b o v e t h e t o p o f its

sk u l l , b u t n o t w h e t h e r i t h a s fo u r l eg s [ s ee 1 4 ] .

Using imagery to learn skills

I f o n e v i su a li z e s o n e se l f p e r f o rm i n g a sk i ll o n e i s a cq u i r i n g ,

o n e w i ll a c t u a l ly i m p r o v e a t t h e s k il l. F o r e x a m p l e , i f y o u w e r e l e a r n i n g a n e w d a n c e ,


4/10

1338 S.M. KOSSLYN M. BEHRMANN and M. JEANNEROD

v i sua l i z i ng t he s t eps wou ld l e ad you t o be a be t t e r dance r - -p rov ided t ha t imag ined

prac t ice i s in te rmixed wi th ac tua l prac t ice [e .g . see 17] .

Percept ion and act ion

I m a g e r y d r a w s o n m e c h a n is m s u s e d n o t o n l y i n p e rc e p t io n , b u t a l s o i n m o t o r c o n t ro l .

Shared mechanism with perception

I t is no t an acc iden t t ha t w e speak o f 's e e ing ' ob j ec t s

i n v i sua l images , o r ' h ea r ing ' sounds i n aud i t o ry images . The phe nom eno logy o f men t a l

image ry i s in m any ways l i ke t ha t o f l i ke -moda l i ty pe r cep t i on . Th i s f a c t ha s sugges t ed t o

m any t ha t t he s ame m echan i sms a r e i nvo lved i n a t le a s t som e a spec t s o f ima ge ry and l ike -

m oda l i t y p e r cep t i on . Th e re i s now a l a rge li t e ra tu r e a t t e s t i ng t o t he v a l i d it y o f th i s

inference [e .g . see 9]. For exam ple , C raver -L em ley and Ree ves [5] sho w ed tha t v i sua l iz ing a

l ine in te r fe res wi th the ab i l i ty to see how two l ine segments a re a l igned , which i s as

expec t ed i f image ry engages mech an i sms t ha t o the rwi se wou ld be u sed i n pe r cep t i on . I t i s

l i ke ly t ha t image ry p roces se s ac tua l l y p l ay a r o l e i n pe r cep t i on , e spec i a l l y when i npu t i s

deg raded - - i n wh ich ca se s image ry may he lp t o comple t e ' no i sy ' i npu t [ s ee 13 ] .

Such f i nd ings a r e cons i s ten t w i th s eve ra l imp or t an t f e a tu r e s o f the neu roan a tom y o f t he

v i sua l sy st em, such a s t he f ac t t ha t v i r t ua l ly eve ry v i sua l a r ea t ha t s ends i n fo rma t ion t o

ano the r a r ea a l so r ece ive s f ibe r s f r om tha t o the r a r e a - - an d t he f ibe r s r unn ing i n each

d i r ec t ion a r e o f comp arab l e s iz e [e . g . s ee 8 ]. Moreo ve r , va r i ous b r a in s cann ing t e chn iques

have r evea l ed t ha t v i sua l a r ea s i n t he occ ip i t a l , t empora l , and pa r i e t a l l obes a r e a l so

ac t i va t ed du r ing v i sua l men t a l im age ry [ fo r a r ev iew , s ee 13 ] . I n add i t ion , dam age t o

po r t i on s o f t he b r a in u sed i n v i sual pe r cep t i on o f t en a l so r e su lt i n co r r e spon d ing de f ic i ts in

visual imagery [e .g. see 15; for an interest ing exception, see 3] .

Motor control

G eorg op ou los and h i s co l le agues [1 I ] have sho wn tha t som e fo rms o f

image t r ans fo rma t ion a r e accompan ied by ac t i v i t y i n t he mo to r co r t ex . These

t r ans fo rma t ions i nvo lve an t i c i pa t i ng whe re t o r each , bu t t he mo to r sy s t em may p l ay a

ro l e more gene ra l l y i n image t r ans fo rma t ions . I ndeed , Deu t sch

et al

[6] found tha t the

r i gh t hemi sphe re , pa r t i cu l a r l y t he pa r i e t a l and f ron t a l l obes , a r e pa r t i cu l a r l y ac t i ve when

sub j ec t s pe r fo rm v i sua l men t a l r o t a t i on ; t he se reg ions o f co r t ex i nc lude s eve ra l a r ea s t ha t

p l ay key ro l e s in mo to r con t ro l . Thus , i t is pos sib l e t ha t t he mo to r sy s t em p l ays an

imp or t an t r o l e in image ry , spec if ic a ll y i n 'image t r an s fo rm a t ion ' p roces se s , a s no t ed

be low.

Informa tion processing

Images can be r ega rded a s a pa r t i cu l a r way i n wh ich i n fo rma t ion i s s t o r ed , and t he se

r ep re sen t a t i ons a r e p roces sed i n specif ic w ays . The sys t em in wh ich images a r e p ro ces sed

can be concep tua l i z ed i n t e rms o f f ou r ma jo r ab i l i t i e s , a s no t ed be low. A l though t h i s

app roach ha s been worked ou t i n t he mos t de t a i l f o r v i sua l men t a l images , co r r e spond ing

dis t inc t ions may a l so ex is t in o ther modal i t i es .

Image inspection

Imag e i n spec t ion i s the ab i l i ty t o ex t r ac t i n fo rma t ion t ha t i s dep i c t ed

in an image . Fo r example , one can de t e rmine whe the r t he f i rs t t h r ee no t e s o f Th re e b l i nd

mice go up o r dow n , o r whe the r t he r e a r e any cu rved l ines i n t he uppe rca se l e tt e r ' b ' . I f

one cou ld no t ' i n spec t ' pa t t e rn s i n images , images cou ld no t s e rve a s r epos i t o r i e s o f

i n fo rma t ion . I n t he cou r se o f in spec t ing imaged ob j ec t s , sub j ec ts r epo r t hav ing t o ' zoom

in ' t o ' s e e ' sma l l de t ai ls , and i n f ac t, peop l e r equ i r e more t ime t o ev a lua t e ob j ec t s imaged a t

smal l s izes than to eva lua te ob jec ts imag ed a t l a rger s izes . S imi la r ly , images can only be so

l a rge be fo r e t hey appea r t o ' ove r f l ow ' one ' s f ie ld o f v i ew . I n add i t i on , t he f a r t he r one m us t


5/10

THE COGNITIVE NEUROSCIENCE OF MENTAL IMAGERY 339

s h i ft o n e s m e n t a l g a z e t o s e e a p o r t i o n o f a n i m a g e d o b j e c t, t h e m o r e t i m e is t a k e n . A n d

s o m e p a r t s o f o b j e c t s a r e m o r e d i f f ic u l t t o s e e i n i m a g e s ; f o r e x a m p l e , a p a r a l l e l o g r a m is

h a r d e r t o s e e i n a n i m a g e o f t h e S t a r o f D a v i d t h a n is a t ri a n g l e [ fo r r e v ie w s o f s u c h

f ind ings , s ee 13] .

Image generation

L o g i c a l l y , im a g e s c a n a r i s e in o n l y t w o w a y s : F i r st , s e n s o r y i n p u t c a n

b e r e t a i n e d ( o v e r t h e c o u r s e o f a f e w s e co n d s , n o t s i m p l y a f r a c t i o n o f a s e c o n d ) ; s e c o n d ,

i n f o r m a t i o n s t o r e d in l o n g - t e r m m e m o r y c a n b e a c ti v a t ed . M o s t i m a g e r y a p p e a r s t o a ri se

w h e n i n f o r m a t i o n i n l o n g - t e r m m e m o r y i s a c t i v a t e d . I n d e e d , m a n y i m a g e s a r e n o v e l

c o m b i n a t i o n s o f o b j e c t s o r c h a r a c te r i st i c s t h a t w e r e e n c o d e d a t d i f f e r e n t t i m e s a n d p l ac e s .

F o r e x a m p l e , m o s t p e o p l e c a n v i s u a li z e th e i r f a v o r i te p o li t ic i a n r id i n g a d o n k e y , a n d

r e p o r t w h e t h e r h e o r s h e c o u l d s e e o v e r t h e t o p o f t h e a n i m a l s h e a d - - - e v e n t h o u g h s u c h a

s c e ne w a s n e v e r w i tn e s s ed . M a n y s t u d ie s h a v e n o w s h o w n t h a t s u c h i m a g e g e n e r a t i o n

i n v o l v e s s e r ia l p ro c e s s i n g ; i n d e e d , t h e t i m e t o f o r m i m a g e s t y p i c a l l y i n c r e a s e s l i n e a r l y f o r

e a c h a d d i t i o n a l p a r t o r p r o p e r t y o f t h e t o - b e -v i s u a li z e d o b j ec t , a n d t h e t im e t o v i su a l iz e a

g i v e n p a r t o f t e n c a n b e p r e d i c t e d b y t h e o r d e r i n w h i c h s u b je c ts t y p i c a l ly d r a w t h e p a r t s .

I m a g e , g e n e r a t i o n h a s b e e n t h e t o p i c o f in t e n si v e s tu d y i n n e u r o p s y c h o l o g y , p a r ti c u l a r l y

s in c e F a r a h [7 ] r e v i e w e d t h e c a s e s t u d y l i t e r a t u r e a n d c l a i m e d t h a t t h i s a b i li t y is s e le c t iv e l y

d i s r u p t e d b y d a m a g e t o t h e p o s t e r i o r le f t c e r e b r a l h e m i s p h e r e .

Image transformation I m a g e r y w o u l d b e o f li m i te d v a l u e i f w e c o u l d o n l y r e c al l p a s t

e v e n t s o r r e c o m b i n e t h e s e m e m o r i e s in s p ec if ic w a y s . I n m a n y c a s e s w e w a n t t o t u r n

s o m e t h i n g o v e r i n o u r m i n d s [ se e 19 ]. O n e o f t h e m o s t i n te r e s ti n g f a ct s a b o u t i m a g e

t r a n s f o r m a t i o n s is th a t t h e y o f t e n p r e se r v e th e t i m e c o u r s e o f t h e c o r r e s p o n d i n g a c t u a l

t r a n s f o r m a t i o n . F o r e x a m p l e , i f y o u r o t a t e d t h e u p p e r c a s e l e tt e r n 9 0 ° c l o c k w is e , w o u l d i t

b e a n o t h e r l e tt e r? W h a t i f y o u r o t a t e d a n u p p e r c a s e le t t er p 1 80 °? T h e f a r t h e r y o u w o u l d

h a v e t o r o t a t e t h e o b j e c t , m o r e t i m e y o u w o u l d r e q u i r e . S im i l a rl y , t h e m o r e s q u a r e s o n e

m u s t f o l d t o v i su a l iz e a s e t o f c o n n e c t e d s q u a r e s f o r m i n g a b o x , t h e m o r e t i m e i s r e q u i r e d

[2 0]. S u c h r e s u l ts a r e a s e x p e c t e d i f o n e i s s im u l a t i n g w h a t w o u l d o c c u r i f a n o b j e c t w e r e

a c t u a ll y m a n i p u l a t e d i n a sp e cif ic w a y , a n d - - a s n o t e d a b o v e - - t h e m o t o r s y st e m m a y p l a y

a r o l e i n im a g e t r a n s f o r m a t i o n s .

Image retention

M a n y i m a g e r y t a s k s r e q u i r e c o n s i d e r a b l e t i m e t o c o m p l e t e , a n d t h u s

t h e im a g e m u s t b e m a i n t a i n e d . M a i n t a i n i n g a n i m a g e d p a t t e r n r e q u i r e s e f f o rt , a n d o n l y a

l im i t e d a m o u n t o f i n f o r m a t i o n c a n b e v is u a li z e d a t th e s a m e t i m e . T h i s a s p e c t o f i m a g e r y

h a s r e c e i v e d r e m a r k a b l y l i tt le st u d y , h o w e v e r , a n d t h e a v a i l a b l e re s e a r c h m e r e l y s e rv e s to

v e r i f y t h a t s u c h i n t r o s p e c t i o n s a r e c o r r e c t [ f o r a r e v i e w , s ee 1 3 ].

Reasoning using imagery

O n e r e a s o n w h y im a g e t r a n s f o r m a t i o n s a r e i m p o r t a n t i s t h a t i m a g e s a re m o r e t h a n a

m e a n s b y w h i c h i n f o r m a t i o n i s l e a r n e d , s to r e d a n d r e t r ie v e d . T h e y a l s o p l a y a k e y r o l e in

s e v e r a l t y p e s o f r e a s o n i n g .

Concrete and abstract reasoning

I m a g e r y c a n p l a y a r o l e i n t w o d i s t i n c t t y p e s o f

r e a s o n i n g . F i r s t , o n e c a n r e a s o n a b o u t p e r c e p t u a l p r o p e r t i e s t h e m s e l v e s . F o r e x a m p l e ,

c o n s i d e r h o w y o u d e c i d e w h a t is th e b e s t r o u t e t o t a k e t o g e t t o t h e a i r p o r t a t a s p e ci fi c

t im e o f d a y f r o m w h e r e y o u w o r k , o r h o w y o u w o u l d d e c i d e w h e t h e r a s o f a s e en in t h e

s t o r e w o u l d f it i n y o u r l iv i n g r o o m . I n b o t h c a se s , i m a g e r y is u s e d t o c a r r y o u t a k i n d

o f m e n t a l s i m u l a t i o n . S e c o n d , i m a g e r y c a n b e u s e d i n a b s t r a c t r e a s o n i n g . I n d e e d ,

i m a g e r y a p p a r e n t l y p l a y e d a r o l e i n s o m e o f t h e k e y d i s c o v e r ie s i n s c ie n c e [s ee 1 9]. F o r

e x a m p l e , a n i m a g e o f s n a k e s b i ti n g t h e i r o w n t a i ls a p p a r e n t l y l ed K e k u l e t o i n f e r t h e


6/10


ring-structure of benzene. In this sort of reasoning images are used as symbols. Such

reasoning occurs when one visualizes Venn Diagrams or lines with dots on them as aids

to solving logic problems.

Using imagery to comprehend language

Another kind of reasoning involves the

inferences one makes when comprehending language. Imagery is often contrasted with

verbal abilities, but it is clear that the two faculties work together in many ways. We earlier

noted that imagery can help one to learn new information, including verbal information.

In addition, imagery can help one to comprehend verbal descriptions. This is most obvious

when the description specifies visual or spatial information. For example, imagery is likely

to be used when one understands a description of the arrangement of an apartment or the

design of a complex machine.

OVERVIEW

The articles in this special issue span the range of research on imagery, and either speak

directly to the brain bases of imagery or have direct implications for future

neuropsychological research.

Learning and memory

We begin with an article that harvests the results of decades of research on the role

of imagery in memory. Richardson turns to the use of imagery mnemonics as a way to

assist patients with memory impairments (caused by brain damage or disease). He

found that patients with less severe memory impairments benefited most from imagery

training, but the effectiveness of training was not strongly related to the locus of brain

damage or to the etiology of the disorder. More strongly motivated patients also did

better, but intelligence, imagery ability, and education were not very important

predictors of the degree of improvement. It is of great interest that brain-damaged

patients who could use mnemonics effectively nevertheless may not use this strategy

spontaneously.

The early work on the role of imagery in memory focused on the importance of

concrete vs abstract materials, or on the role of organizational factors in memory. De

Beni and Pazzaglia consider memory for different kinds of mental images, focusing

specifically on the role of contextual and autobiographical variables. They asked normal

subjects to visualize a series of items, and measured how quickly the subjects could form

the images, how well they could later recall the items, and also assessed subjective

ratings of image quality. The items were visualized in isolation or in a specific context

(with three kinds of context), and De Beni and Pazzaglia particularly were interested in

the possibility that images of one s own life experiences have a special status in memory.

And in fact such images required the most time to generate, were recalled very well, and

were very vivid. F rom a neuropsychological perspective, this work is interesting because

it invites researchers to attempt to understand the importance of this particular kind of

imagery. It is possible, for example, that the right hemisphere plays a special role in

storing images of specific exemplars; does it also play a special role for autobiographical

images?


7/10

THE COGNITIVE NEUROSCIENCE OF MENTAL IMAGERY 34

ercept ion a nd action

In some ways imagery may have the same relation to other processes that glasses have to

the nose: Once certain properties of a nose were present (presumably to warm air, direct

scents to different nostrils, and so on), they could be used to hold up glasses. Similarly,

once specific properties o f basic sensory and response mechanisms were in place, they

could be recruited into imagery--which then in turn may have become a function in its

own right, thereafter being the subject of natura l selection.

Goldenberg, Miillbacher and Nowak report the case of a brain-damaged patient who

had cortical blindness, but still had visual mental imagery. This patient denied that she was

blind, and this belief could have been based on her confusing visual mental images for

actual percepts. This patient is interesting in part because her primary visual cortex was

almost totally lesioned; only a small portion of the cortex at the occipital tip of the left

upper calcarine lip was spared. Thus, her imagery apparently did not depend on an intact

area 17, but rather relied (at least in large part) on higher-level visual areas that were not

damaged. The authors note that this patient s imagery could have been triggered by tactile

or acoustic perception. Once her vision recovered in the portion of the field that was

registered by the preserved cortex, nonvisual input no longer appeared to induce the

illusion of seeing. These findings suggested that connections from nonvisual cortex play a

complex role in evoking imagery.

Servos and Goodale also examine a patient who had spared imagery in the face of

impaired perception. This patient had a severe visual form agnosia, but nevertheless was

able to scan mental images to search for specific features and could generate novel images

by combining images of previously encoded stimuli. This patient had been shown earlier to

use information about shape to control skilled grasping movements, even though she

could not use this information to categorize or characterize the form. These findings

suggest that sensory input, imagery, and motor control mechanisms have distinct ways of

accessing stored visual representations; it is interesting that imagery and motor access were

preserved together in the face of impaired sensory access. This patient appeared to have

damage primarily to areas 18 and 19; most of area 17 was intact, as well most extrastriate

cortex. The results may indicate that pathways within areas 18 and 19 are not essential for

visual mental imagery, but play a critical role in providing input to higher visual areas

during perception.

The fact that both motor control mechanisms and imagery can access representations of

shape even when perceptual input cannot is consistent with results summarized in the

following paper by Annett. Annett focuses on the kind of motor imagery that is used when

one mentally practices performing a specific action. Annett reviews theoretical and

practical issues that bear on the subjective, behavioral, and physiological methods used to

study moto r imagery. He argues that motor imagery relies on mechanisms that serve both

spatial and motoric functions, and that the highly distributed nature of motor control

blurs the question of whether motor imagery is primarily perceptual or motoric in nature.

Jeannerod also explores the role of the motor system in imagery. His central idea is that

motor images have the same properties as the corresponding actual motor representations.

He presents much evidence that supports this hypothesis. For example, the timing of

simulated movements respects the same constraints as that of actual movements. In

addition, simply imagining that a limb is moving produces an increase in specific reflexes in

that limb, as occurs during actual movement. Moreover, imaging movement produces


8/10


changes i n co r t ex l i ke t hose t ha t occu r when one ac tua l l y execu t e s an ac t i on . J eanne rod

p rop oses a h i e r a rch i ca l mod e l o f t he o rgan i za t i on o f a c t i on t ha t p rov ides an a ccou n t f o r

t he s imi l a r i t i e s be tween imag ined movemen t and ac tua l movemen t ; t h i s mode l p rov ides

the t heo re t i c a l f oun da t i ons fo r t he e f f ect s o f men t a l p r ac t i c e r ev i ewed i n t he p r ev ious

pape r .

The f ina l pape r i n t h is s ec t i on add re s se s a d i f fe r en t so r t o f mo to r image ry , wh ich i s no t

r e l a ted t o t he man ipu l a t i on o f shape . Wi l son , Smi th and Re i sbe r t examine t he ro le o f

subvo ca l i z a t i on i n aud i t o ry image ry . They examine t he pa r t ne r sh ip be tween t he i nne r

e a r a n d i n n e r v o i c e , a n d f in d th a t t h e tw o f u n c ti o n s u s u a l l y - - b u t n o t a l w a y s - - w o r k

toge ther . Al though th i s paper i s no t expl ic i t ly neuropsychologica l in i t s o r ien ta t ion , i t

p rov ide s a beacon fo r f u r t he r r e sea r ch ; i t s eems c l ea r t ha t aud i t o ry image ry can be more

d e e p l y u n d e r s t o o d b y a d o p t i n g a p a ra l le l a p p r o a c h t o t h a t n o w u s e d t o s t u d y v i s u al

image ry . I ndeed , it m ay t u rn o u t t ha t t he pa r a ll e ls r un deepe r t han t h is ; t he i nne r e a r and

v i sua l image i n spec t ion mechan i sm s may be func t i ona l l y s imi la r, and t he i nne r vo i ce

and som e fo rm s o f v i sua l image gene ra t i on mechan i sms (pa r t icu l a r l y t hose i n wh ich eye

movemen t s c an p rompt images i n a s equence ) may be func t i ona l l y s imi l a r . Such

mech an i sms a r e d i s cus sed i n t he a rt ic l e s t ha t add re s s t he ro l e o f image ry i n i n fo rm a t ion

process ing .

nformation processing

E n o r m o u s p r o g r e ss h a s b e e n m a d e r e c en t ly in u n d e r s t a n d i n g t h e b r a in - b a s e s o f t h e

mech an i sms t ha t g ene ra t e v i sua l men t a l images. Fa r ah r ev iews r ecen t ev idence t ha t speaks

to tw o gene ra l i s sue s abo u t imag e gene ra t ion . F i r s t, she cons ide rs t he ques tiOn o f whe th e r

a d i s t i nc t componen t p roces s i s u sed t o gene ra t e v i sua l men t a l images , o r whe the r

mechan i sms t ha t s e rve o the r pu rpose s work t oge the r t o a ccompl i sh t h i s end . Second , she

r ev iews f i nd ings t ha t bea r on t he neu roana tom ica l l oca l iz a t i on o f t he p roces se s tha t

unde r l i e imag e gene ra t i on , an d co ns ide r s wh ich reg ions o f t he b r a in a r e r ec ru i t ed by t he se

processes . The fo l lowing ar t ic les repor t addi t iona l f ind ings tha t speak to these i ssues .

S t anga l i no , Semenza a nd M ond in i r e po r t a de t a i l ed i nves t iga t i on o f t he e f fec t s o f b r a in

dam age on t he ab i l i ty t o gene ra t e v i sua l men t a l images . They t e s t ed 70 un i l a te r a l ly b r a in -

dam age d pa t i en t s w i th a s e t o f ta sks t ha t w as de signed t o a s ses s d i s ti nc t com po nen t

p roces se s . Cons i s t en t w i th t he o r ig ina l r e t r o spec t ive ana ly s is o f Fa r a h [7] , t he se r e sea r che rs

foun d t ha t d am age t o pos t e r i o r po r t i on s o f the l ef t hemi sphe re m os t f r equen t l y we re

re la ted to def ic i t s in the ab i l i ty to genera te images .

K o s s l y n , H a m i l t o n , M a l j k o v i c , H o r w i t z a n d T h o m p s o n p r o v i d e e v i d e n c e t h a t v i s u a l

men ta l images can be fo rmed i n two ways . One me thod r e l i e s on s t o r ed de sc r i p t i ons t o

a r r ange p a r t s o f image ob j ec t s , and t he o the r r el ie s on s t o r ed i n fo rma t ion ab ou t me t r i c

d i s t ance t o a r r ange pa r t s . The fo rm er me thod is more e f f ec ti ve when s t imu l i a r e p r e sen t ed

in the r igh t v i sua l f ie ld (and hence a re processed in i t i a l ly by the le f t ce rebra l hemisphere) ,

whe rea s t he o the r m e tho d is more e f fec t ive when s t imu l i a r e p r e sen t ed i n t he l e f t v i sua l

f ie ld (and hence a re processed in i t i a l ly by the r igh t cerebra l hemisphere) . Accord ing to

the i r ana ly s i s , p r ev ious r e sea r ch w i th b r a in -damaged pa t i en t s ha s u sed ma te r i a l s t ha t

can no t e a s i ly be v i sua l iz ed u s ing t he r i gh t-hemi sphe re p roces se s , and hence pa t i en t s w i th

l e f t- hemi sphe re dam age have been s e lec t ive ly impa i r ed .

D e n i s , G o n c a l v e s a n d M e m m i f o c u s o n a n o t h e r a s p e c t o f im a g e g e n e r a ti o n , n a m e l y it s

accu racy . I n p r ev ious work , t h i s g roup ha s shown tha t peop l e can u se ve rba l de sc r i p t i ons

to con s t ruc t images , wh ich can t hen be s canned . B ecause time i s r equ i r ed t o s can each


9/10

THE COGNITIVE NEUROSCIENCE OF MENTA L IMAGERY 1343

increment of distance, scan time can be used as an index of how accurately the parts of the

image are positioned. Using this technique, Denis and his colleagues have shown that the

structure of the initial description affects how accurately an image can be generated. But

even when subjects are given poorly structured descriptions, their images improve after

they see the description repeatedly. The present article reports a quantitative model and a

computer simulation that accounts for this increase in image accuracy. In addition, new

data from humans are reported that bear on this account, and underline the importance of

'regions of uncertainty' (i.e. indeterminancies about the locations of parts) in the image

generation process.

Georgopolous and Pellizzer consider another aspect of information processing in

imagery, mental rotation. They compare the processes that underlie mental rotation to

those used to scan through memorized lists of digits. This paper integrates previous work

by this group with monkeys and recent work with humans. The monkey research relied on

single-cell recordings, and showed that motor processes are involved in at least some forms

of mental rotation. The present work shows that mental rotation is in fact distinct from

memory scanning in humans, which is consistent with the role of the motor system in

imagery.

Reasoning

Vecchi, Monticelli and Cornoldi examine one of the key mechanisms that allows us to

use imagery in reasoning: working memory. A major bottleneck in using imagery in

reasoning is the capacity of working memory, and these researchers hypothesize that this

capacity hinges on properties of a passive store and properties of active imagery

operations. To test these ideas, subjects were asked to visualize moving through two-

dimensional and three-dimensional matrices; the matrices had some filled cells, and the

subjects had to retain the image in order to perform the task. Experiments with

congenitally blind people and with sighted people revealed that image capacity is a result

of at least two different factors.

Geminiani, Bisiach, Berti and Rusconi argue that imagery plays such a central role in

thinking that language cannot entirely compensate for severe impairment of 'analogue

representations'. They argue that language does not have an independent role in thought,

and provide suggestive evidence that syntactic structures themselves can be represented in

a non-linguistic medium. This work is intriguing, and moves into an area that has received

very little attention heretofor.

fterword

We finish with some reflections by Cooper on wha t counts as a mental image . After

the wide and varied writings that have come before, this question gains force. Issues like

this can only be addressed in the context of empirical findings, and can help not only to

make sense of those findings but to guide one to consider additional research questions.

REFERENCES

1 . A nde rson , J . R . Argum ent s conce rn ing r ep re sen ta t i ons fo r m en ta l im age ry .

P s y c h o l . R e v .

85, 249-277, 1978.

2. Baylor ,

G . W . A T r e a t i s e o n t h e M i n d s E y e .

Vol . Ed.) See . Ed.). Ca rnegie Mel lon Univers i ty , Pi t t sburgh,

Pennsylvania , 1971.

3 . Behrm ann , M. , Winocur , G . and Moscov i t ch , M. Di ssoc i a t i on be tween m en ta l im age ry and ob j ec t

r ecogn i t i on i n a b ra in -dam aged pa t i en t.

N a t u r e

359, 63~637, 1992.


10/10

1344 S.M . KOSSLYN , M. BEHRMA NN and M. JEANNEROD

4. B ower , G . H. M en ta l im age ry and a ssoc i a ti ve l e a rn ing . In Cognition in Learning and Memory, L . G r e g g

(Edi tor) , pp. 51-88. Wiley, New Y ork, 1972.

5. Crav er-Lem ley, C. and Reeves, A. V isual imagery se lec tively reduces vernier acuity . Perception 16, 533-614,

1987.

6 . Deu t sch , G. , B ourbon , W . T . , Papan i co l aou , A. C . and E i senbe rg , H. Vi suospa ti a l expe r im en t s com pared v i a

ac t iva t ion of regional cerebra l b lood f low. Neuropsychologia26, 445-452, 1988.

7. F arah , M. J . T he neu rologica l basis o f men ta l imagery: A com pon ent ia l ana lysis .

Cognition

18, 245-272,

1984.

8. Fe l lema n, D. J . and V an Essen, D. C. Dist r ibu ted hierarchica l processing in pr imate cerebra l cor tex.

Cerebr.

Cortex 1, 1-47, 1991.

9. Fink e , R. A. and Sh epard, R . N. Visual funct ions of men ta l imagery. In

Handbook of Perception and Human

Performance,K. R . Boff, L . Ka ufm an and J . P. Thom as (Edi tors) , pp. 37-1-37-55. W iley-Intersc ience , N ew

York, 1986.

10. Gardner , H.

The Mind s New Science: A History of the Cognitive Revolution.

Basic Books, New York, 1985.

11. G corg opo lous, A. P. , Lur i to , J . T . , Pe t r ides, M. , Schwartz, A. B. and Massey, J . T . Menta l rota t ion o f the

neurona l popu la t i on vec to r .

Science

243, 234-236, 1989.

12. Kosslyn, S. M. Informat ion representa t ion in visua l images. Cognit. Psychol.7, 341-370, 1975.

13. Kosslyn, S. M.

Image andBrain: The Resolution of the Imagery Debate.

M IT Press, Camb ridge , Mass. , 1994.

14. K osslyn, S. M. an d Jol icoeu r , P. A theory -based app roach to the study o f individual dif ferences in menta l

im age ry . In Aptitude, Learning and Instruction: Cognitive Processes Analysis of Aptitude, R. E . Snow, P . A.

Fed er ico and W. E. M ontag ue (Edi tors) . Law rence Erlbau m, H i l l sda le , Ne w Jersey, 1981.

15. L ev ine , D . N . , Warach , J . and Fa rah , M. J . T w o v i sual sys tem s in m en ta l im age ry : Di ssoc ia t i on o f 'wh a t ' and

'where ' in imagery disorders due to bi la tera l poster ior cerebra l lesions.

Neurology

35, 1010-1018, 1985.

16. Paivio, A. Imagery and Verbal Processes. Hol t , R ineha r t Wins ton , New York , 1971.

17. Richa rdson, A. M enta l prac t ice : A review and discussion (Par t I and I I ) .

Res. Quart. 38,

95-107; 263-273,

1967.

18. Segal , S. J . and Fuse l la , V. Inf luence of imaged pic tures and sounds on de tec t ion of visua l and audi tory

signals. J. Exp. Psyehol. 83, 458-464, 1970.

19. Shepa rd , R . N. and C oope r , L . A. Mental Images and their Transformations.M IT Pre ss , Cam br idge , Mass . ,

1982.

20 . Shepa rd , R . N. and Feng , C . A ch ronom e t r i c s tudy o f m en ta l pape r fo ld ing . Cognit. Psychol. 3, 228-243,

1972.

21. S hepard, R . N. and Metz ler , J . Menta l rota t io n of three-dimensional objec ts. Science 171, 701-703, 1971.

the cognitive neuroscience of mental imagery (kosslyn 1996)

Documents