1

From Presence Towards Consciousness

Maria V. Sanchez-Vives and Mel Slater

Abstract

Immersive virtual environments can break the deep everyday connection

between where our senses tell us that we are and where we actually are

located and whom we are with. ‘Presence research’ studies the phenomenon of

acting and feeling that we are in the world created by computer displays. We

argue that presence is a phenomenon worthy of study by neuroscientists and

may help towards the study of consciousness, since it may be regarded as

consciousness within a restricted domain.

Suppose that you are in a place that you know to be fictitious. It is not a ‘place’ at

all in any physical sense, but an illusion created by a virtual reality system. You know

that there is no place, and you know that the events you see, hear and feel that are

happening there are not really events in the every day physical meaning of that word.

You are conscious of that ‘place’ and those ‘events’, and simultaneously conscious of

that the fact that there is no place and there are no events. Yet, you find yourself

thinking, feeling and behaving as if that place were real, and as if those events were

happening. For example, you see a deep precipice in front of you that you know is not

really there in a physical sense. Your heart races and you are frightened enough by what

you see to be very reluctant to move yourself closer to the edge. From a cognitive point

of view you know that there is nothing there, but both unconsciously (e.g., heart rate)

and consciously (your awareness of your own fear) you respond as if there is something

there. How is this possible? This paradox, observed daily in virtual reality laboratories

around the world, is at the root of the concept of ‘presence’ studied by virtual reality

specialists, and also relates to many of the concepts studied by neuroscientists ranging

from perception through to the consciousness.

2

Immersive Virtual Environments

In 1980 Marvin Minsky introduced the concept of telepresence to describe the

feeling that a human operator may have while interacting via a teleoperator system 1.

The human operator sees through the eyes of the remote machine, and uses his or her

own limbs to manipulate its effectors. A sense of being at the distant place may develop,

with the body of the machine ‘becoming’ the body of the human. This experience of

telepresence was thought to be conducive to effective task performance of the operator

in the remote environment. The concept of telepresence has also been applied to

experiences within virtual environments. In this case the person is immersed within an

environment that is realised through computer controlled display systems, and may be

able to effect changes in that virtual environment. A feeling of being present may

develop in the same way that Minsky noted for physical teleoperator systems.

A rigorous discussion of the concept of a ‘virtual environment’ has defined

virtualisation as “the process by which a human viewer interprets a patterned sensory

impression to be an extended object in an environment other than that in which it

physically exists”2 (see Box 1). A virtual environment (VE) should incorporate the

participant as part of the environment, so that head motions result in motion parallax

from the participant’s viewpoint, and a number of physiological and vestibular

responses associated with focusing and object tracking are stimulated.

Clearly there are many parameters that control the quality of the experience that a

person may have in such a system. The visual display, for example, may have greater or

lesser field-of-view – the effective horizontal and vertical angles through which the

world can be seen. A head-mounted display (HMD) system typically delivers a

relatively low field of view. For example, 60 degrees diagonal – compared with the

more than 180 degree horizontal and 120 degree vertical in normal vision. Second, the

visual display may have more or less resolution – the number of pixels per unit

projected visual area. For example one popular HMD model has pixel resolution of 640

by 480 on a 3.3cm diagonal liquid crystal display (per eye). The frame-rate achieved

by the computer system is a critical factor in maintaining the illusion of there being

‘extended objects’ in an environment. The frame-rate is the number of frames a second

that the computer graphics system can deliver. Although the display will always be

refreshed at a constant cycle (e.g., 60Hz) the computer graphics rendering system may

3

not always be able to maintain this speed of new image generation as a function of head

movement and animation in the scene. If the scene is particularly complex in the

direction of the current view then the computational load on the system may be so high

that as the participant’s head moves the rendering system cannot keep up with the

changes fast enough. This would result in discontinuities in overall image motion – for

example, an object that moves across the field of view of the participant may appear to

jump suddenly from one location to another. Frame rate is one factor in overall system

latency. The latency is the time between the participant initiating an event (such as a

head turn) and the time that the system responds (the images update accordingly). There

are many contributory factors to latency – such as the frame rate, the speed of the

tracking devices, the communication speeds across the different devices including

network speeds, and so on. Note that frame rate can be very high but latency very low

(for example, because of communication bottlenecks) and this would result in smooth

motion but always some milliseconds behind the initiating actions of the participant.

Typically the visual fidelity of a virtual environment display in comparison to

physical reality is low. This is for several reasons. Today’s computer graphics hardware

cannot simulate the complexity of global light transport without substantially sacrificing

real-time performance. Second, the physical world is exceedingly complex, with infinite

layer of detail: imagine rendering a human face, with full subtlety of expression, all the

micro-muscle movements that go into the making of a facial expression, the physical

dynamics of hair, skin, muscle movements and so on. Of course computer graphics has

to rely on abstractions and models that are far removed from physical reality, and most

especially if real-time display and interaction is necessary.

Although we have concentrated here on the visual aspects of virtual environments

the same may be said of the generation of sound and haptics (comprising touch and

force feedback). Although the technology for producing highly convincing auditory

output is advanced 3 generating this on the fly in real-time in a dynamically changing

situation is not possible. Haptics is possible within a limited domain of application.

There are typically two major approaches. The first is to limit haptics to the end-effector

of an instrument which is manipulated by the human user 4,5. Users can feel as if they

are touching objects with the tip of the instrument as they move it around, with its

virtual representation within the VE. As this virtual representation collides or makes

friction with virtual objects, the user is able to feel this, and it can be extremely

convincing. Another approach is to fit the human user partially within the frame of an

4

exoskeleton, which is mechanically controlled to impart forces on the user according to

how the user interacts with objects within the VE6. For example the sensation of the

weight of an object can be imparted to the user in this way.

All the factors that we have been analysing define what is called immersion, a term

that refers to what the overall VE system can deliver: the extent of field of view, the

number of sensory systems it simulates, the quality of rendering in each, the extent of

tracking, the realism of the displayed images, the frame-rate, latency, and so on. Of

particular importance is the degree to which simulated sensory data matches

proprioception – for example, as the participant’s head turns how fast and how

accurately does the system portray the visual and auditory effects of this. Or in a system

such as a head-mounted display where all real-world visual input is removed, the extent

to which the participant sees a virtual body that correctly correlates with the

proprioceptive model of the feeling of the actions of that body.

The degree of immersion is therefore an objective property of a system that in

principle can be measured independently of the human experience that it engenders.

Presence, however, is the human response to the system, and there are many ways in

which the meaning of presence have been formulated.

The Concept of Presence If immersive virtual environment systems were able to deliver the perfect illusion of

being and acting in a virtual world then probably the issue of ‘presence’ would never

have arisen. It is important though because of the need to optimally allocate scarce

computational, display and tracking resources within given economic and technical

constraints. One way to do this would be to follow an application specific route and

optimise resources based on task performance of users. For example, if a VE were used

for the training of surgeons, then transfer of skills from the virtual to real world

performance may be used to indicate the effectiveness of the VE. Presence though is

more like a global currency, a concept that applies across applications, where generic

knowledge about factors influencing presence can be applied in many different

scenarios. But what is ‘presence’?

There are several different approaches. According to one definition ‘experiential

presence’ is ‘a mental state in which a user feels physically present within the computer-

mediated environment’7. This is a way of expressing the common view that presence is

the sense of ‘being there’ in the virtual environment or similarly the sense of being in

5

the place depicted by the virtual reality rather than in the real physical place where the

participant’s body is really located8-12.

A fundamentally different view 13,14 is that presence is ‘... tantamount to

successfully supported action in the environment.’ It is argued that reality is formed

through action, rather than through mental filters and that ‘...the reality of experience is

defined relative to functionality, rather than to appearances’. What is important in this

approach is action (how things are done) and the affordances offered in the virtual

environment, rather than just appearances, and that the sense of ‘being there’ is

grounded on the ability to ‘do’ there. These ideas have been expressed in other ‘body

centred’ approaches where it is argued that it is essential for there to be a close match

between kinesthetic proprioception and the stream of sensory data15,16: for example, if

the participant is supposed to be moving through an environment by walking, then when

visual flow indicates walking presence will be higher the more that the person’s bodily

movements correspond to real walking.

As discussed earlier a distinction is made between immersion and presence by many

researchers7,12. Immersion is simply a description of overall fidelity in relation to

physical reality provided by the display and interaction systems. In this view, presence

research is essentially that of carrying out experiments that manipulate the variables that

make up immersion, in order to build an equation with presence on the left hand side,

and the factors of immersion on the right hand side. Individual psychological

differences between people can be also included as variables on the right hand side17,18.

This is a worth-while effort in the quest to produce a statistical model of how presence

and immersion may be related, based on empirical data (for example 18-22) but it does

not enhance understanding of the processes involved.

Measuring Presence A major challenge in presence research is how to measure presence at all. The

normal approach is to use questionnaires. Participants carry out some task within a

virtual environment, and then after their experience they answer a questionnaire. The

questions have ordinal scales that anchor responses between two extremes – for

example 1 meaning ‘no presence’ in the virtual environment and 7 meaning ‘complete

presence’ (see Box 2 for examples) 23-25. The earliest questionnaires were derived from

observing and listening to subjects in debriefing interviews26. Some later questionnaires

were derived by factor analytic studies from earlier ones16,25,27-29.

6

Questionnaire based presence assessment methods have been shown to be unstable

in the sense that prior information can change the results30. There is also evidence to

suggest that typical questionnaires cannot discriminate between presence in a virtual

environment and physical reality31. The use of questionnaires has been challenged

through the observation that they cannot avoid a methodological circularity – that the

very asking of questions about ‘presence’ may bring into being, post-hoc, the

phenomenon that the questionnaire is supposed to be measuring32.

A second method for measuring presence is behavioural. If participants within a

virtual environment behave as if they were in an equivalent real environment then this is

a sign of presence. Examples include the looming response8, postural sway33,34, after-

effects35 and the resolution of conflicting multi-sensory cues21. These behavioural

measures typically require the introduction of features into the environment that would

cause a bodily response (such as swaying in response to a moving visual field, or

ducking in response to a flying object).

A particular specialisation of the behavioural approach is to use physiological

measures, such as those derived from EKG recordings or galvanic skin response. The

idea in this case is that if the normal response of a person within physical reality to a

particular situation is known and they exhibit the same response within a virtual

environment then this is a sign of presence. The use of physiological measures as

surrogates for presence has been attempted – but have been limited to situations where

the physiological response is obvious (e.g., such as a response to a feared situation) and

the results have been positive36. The drawback here is that physiological responses to

mundane situations such as being in a virtual room which has a table and some chairs

are not obvious.

Another method for measurement of presence is based on the idea of eliciting

moments in time when breaks in presence (BIPs) occur37. A BIP is any perceived

phenomenon during the VE exposure that launches the participant into awareness of the

real-world setting of the experience, and therefore ‘breaks’ their presence in the VE.

Examples include gross events such as collisions with the equipment, through to more

subtle effects such as revelations that come from seeing a tree as a pixel map rather than

a solid object. We proposed a stochastic model that allowed the construction of a

presence measure from knowledge of moments in time when participants reported such

BIPs. This estimator was shown to be correlated with traditional questionnaire

measures. This approach also provides an alternative strategy to the use of physiological

7

measures. Rather than limiting the application of these to stress-inducing environments,

it tries to find the physiological signature of a BIP, since there is evidence that breaks in

presence are associated with physiological responses38. Because BIPs may occur in any

environment, it provides a more general approach – in particular the environments do

not need to be stressful.

Experimental Studies of Presence What do we know about the factors that influence reported presence in virtual

environments? There have been several factorial design experimental studies, the vast

majority of which have examined the influence of various display and interaction styles

– essentially following the traditional paradigm of experiments in psychology.

Display Parameters. Higher graphics frame-rate is positively correlated with

reported presence. The critical minimal frame-rate appears to be around 15Hz24. A study

exploiting changes in heart rate as a surrogate for presence when subjects looked over a

virtual precipice usually referred to as the ‘pit room’ scenario (see Figure 4)36 also

found that 15Hz was the minimal critical value. In general, a lower latency between

head movement and display update (see above) was found to be positively associated

with increases in heart rate in response to the same stressful environment39.

Other investigations have concentrated on the influence of head-tracking, stereopsis

and geometric field of view20,40-42. The results suggested that these all were associated

with higher reported presence.

Visual Realism. It is surprising that the evidence to date does not support the

contention that visual realism is an important contributory factor to presence. Only one

study has clearly found this in the context of a driving simulator application43. Although

another study found that the display of dynamic shadows in an environment was

associated with higher reported and behavioural presence compared to static or no

shadows, this cannot directly be attributed to visual realism, but to the more convincing

display of dynamics21. In other experiment a group of people carried out a task in a

simplistic virtual simulation of a real laboratory delivered through a head-tracked HMD,

and another group of people carried out the same task in the real laboratory. Two

different presence questionnaires were administered to each group in randomised order

after the experiment, and there was no significant difference in the mean reported

8

presence between the two groups31. Another recent experiment that has directly tackled

the issue of realism found no relationship with presence44.

Sound. What difference does sound make to presence in VEs? Anecdotally people

report that sound makes a very significant impact, but there have been few studies of

this. One found that there was a significant effect of sound on reported presence, with

spatialised sound associated with higher reported presence than either no sound or non-

spatialised sound45. Another found that the use of person-specific head-related transfer

functions (HRTFs) was significantly and positively associated with reported presence

and illusory self-motion.46

Further evidence relating to the importance of the auditory modality for presence is

found in studies of a sort of ‘inverse presence’ – what happens to people’s ‘presence’ in

physical reality when they experience a sudden and lasting loss of hearing. It appears

that the impact on their sense of reality and their sense of being in the real world is

profoundly altered47,48. Sometimes sound may be so important for presence that illusory

auditory events may be perceived, as happened in a paranoia experiment (see Figure 6).

Haptics. There have been some studies on the influence of haptic feedback on

presence. One study investigated the influence of haptics on co-presence,49 the sense of

being with another person together in a virtual environment50. The experiment was

designed so that subjects located at remote sites without seeing or hearing each other

could nevertheless feel the forces that each exerted on the hand of the other, by jointly

moving a ring along a wire within a virtual environment. The results showed that for

each subject, and independently of order of presentation, visual plus haptic feedback

was associated with greater reported co-presence than visual feedback of the ring and

wire alone49. A similar cross-Atlantic experiment was conducted where the haptic

feedback was transmitted via the internet between MIT and UCL51, 52.

Although general haptic feedback is today not attainable, it can be very cheaply

simulated with so-called ‘static haptics’. Suppose that a virtual environment is delivered

through a HMD so that the participant cannot see the surrounding real world and that

there are simple versions of physical objects (such as table-tops) in registration with

their virtual counterparts. When a person reaches out to touch a table top they may

indeed feel a simple plasterboard version of one – at least in the same physical place if

not with the exactly expected tactile characteristics. In the same pit-room experiment

9

referred to earlier,36 another group of subjects stood on a small wooden ledge in

registration with the edge of the pit, so that their feet would feel as if they were astride

the edge. This condition again significantly increased heart-rate compared to the other

conditions.

Virtual Body Representation. When a virtual environment is delivered through a

head mounted display the participant cannot see their own body. Indeed anecdotally it

can be a shock while in such an environment to look down and not see a body – it is like

being ‘the invisible man’. There have been some studies which have considered the

relationship between the participant and their ‘virtual body’ and its influence on

presence. One found evidence that reported presence was higher for those subjects

whose body was represented by a complete (if crude) virtual body, compared to those

simply represented by a 3D arrow cursor18. In this experiment when subjects moved

their real right arms they would see their virtual right arms move in synchrony (or

alternatively the 3D arrow cursor). However, if they moved their real left arms the left

virtual arm would not move (since it was not tracked). Some subjects found this

disturbing. For example, they wrote in their debriefing questionnaires statements such

as “I thought there was really something wrong with my [left] arm” ”, suggesting a fast

internalisation of the virtual body, and others who found this dissociation disturbing

wrote of their virtual bodies being – “a dead weight”, “a useless thing”, “nothing to do

with me. It was also noticed that some would go to great lengths to align themselves

with their virtual bodies, positioning their feet exactly where they saw their virtual feet

to be, and so on. There was further evidence of this in a follow up study, where some

subjects always moved their real left arm in synchrony with movements that they saw

their virtual left arm making (which had been programmed to symmetrically mirror the

movements of their real right arms). 17

Body Engagement. When participants wear a HMD they are usually tethered by

cables from the back of the HMD into the main computer. Although they are able to

stand and move around, the range of movement with a traditional electromagnetic

tracker is limited. Therefore in order to cover extensive distances within the virtual

environment some other method than physically walking has to be used – and the usual

solution is to employ a mouse button press on the 3D pointing device that the subject

holds in their hand in order to activate movement. It was noticed in early studies that

10

many subjects, especially those reporting a high degree of presence, were almost unable

to move by button pressing, but kept attempting to physically walk. A method was

derived to reduce this dissociation between proprioception and sensory data by having

subjects ‘walk in place’ as a simulation of walking19. Experimental results showed that

on the average those subjects who moved through the environment using the walking in

place method reported a significantly higher sense of presence than a control group who

used the traditional 3D mouse-button method. This study was later extended by the

incorporation of another experimental group who were really able to walk significant

distances (via a wide-area tracking system)53. The results showed that mean reported

presence was highest for the ‘real walkers’, next highest for those who walked in place

and lowest for the 3D mouse button pressers.

Two other experiments examined the relationship between body engagement and

presence – the hypothesis being that the more participants would use their bodies within

virtual environments to effect changes in a natural way, the greater the sense of

presence. In other words they become ‘anchored’ into the environment through

significant and meaningful motor activity. Each study found a significant positive

association between overall body mobilisation and reported presence15,37.

Further Evidence for Presence – Applications in Therapy A class of applications that cannot work without presence involves the exploitation

of virtual environments for various forms of psychological therapy, typically for the

management of anxiety. As part of a treatment programme a patient may eventually be

placed in a VE that depicts a situation that triggers their anxiety. To what extent is their

response similar to that which would occur when facing the same situation in real life?

The greater the similarity in response, the greater the chance that the virtual

environment can be successfully utilised as part of the therapy programme. This is just

another way to say that a necessary condition for any therapeutic intervention to be

successful is that the patient becomes present in the VE. In this section we present some

evidence that indeed this does occur, and which therefore provides further evidence for

the existence of the presence response. For an overview of the application of virtual

environments to psychotherapy see54,55.

11

Some Specific Phobias. The first reported psychological intervention in this domain was

the application of virtual environments to acrophobia (fear of heights). In an initial pilot

study56 it was found that subjects would exhibit acrophobia signs and symptoms within

a virtual environment exposure, and as the authors noted therefore exhibited a high

degree of presence. The scenarios included an elevator ride, a bridge and a view from a

tall building. A study that used this virtual environment in a graded exposure therapy

found that the experimental group showed significant reductions in anxiety over an

eight week period, compared to a control group on a waiting list57. Of course this did

not show that the virtual environment itself was implicated in the improvement, since

any kind of intervention compared to doing nothing may have helped. Another study

that did not have this defect, however, found the same result, where an in vivo treatment

group was compared with a virtual reality exposure group. This study resulted in similar

imrovements in both groups, as measured by on a battery of tests including a

behavioural avoidance test demonstrating that the virtual reality exposure was

equivalent to in vivo exposure58. A similar study on fear of flying59 also found no

difference between an in vivo exposure group and a virtual reality group, and both

groups were compared to a waiting list control group. Six month and one year follow-up

studies showed that the improvements had been maintained.60 Arachnophobia has also

been studied with similar methods61-63.. In these studies the idea of ‘static haptics’ was

employed, where a physical toy spider was registered in the position of the virtual

spider so that eventually subjects could feel as well as see the spider. It was found that

the tactile augmentation led to better treatment results than the visual simulation alone.

Social Anxiety. Anxiety that involves other people is more difficult to deal with

within a VE. Depicting a bridge, for example, is technically far less challenging that

depicting a group of people. The modelling and dynamic rendering of a virtual human

that can purposefully interact with a real person – for example, through speech

recognition, generation of meaningful sentences, facial expression, emotion

representation, skin colour and tone, modelling and rendering of muscle movements and

joints, accurate depiction of a live acting human body – these are all beyond the

capabilities of today’s real-time computer graphics and artificial intelligence, in spite of

what may be possible in special-effects movie creation. Yet here is also a surprise – the

evidence suggests that people respond to relatively crude virtual humans as if they were

12

real people, that a virtual human that seemingly looks the participant in the eye and

frowns has a similar impact as if that behaviour had been carried out by a real person.

We considered a particular type of social phobia, fear of public speaking. A study

was carried out in which subjects were exposed to virtual audiences in a seminar-style

setting. The subjects had a normal range of public speaking anxiety. The experimental

factor was to have audiences with three different types of behaviour – they were either

static and neutral, dynamic and showing very positive responses towards the audience,

or dynamic and showing very negative responses towards the audience64. Each person

experienced only one of these conditions. Some examples of the positive and negative

audiences are shown in Figure 5, each consisting of the same eight male virtual

characters who changed posture and facial expression, and also made verbal comments

during the progress of the 5 minute talk. The statistical results indicated that for those

who were immersed with the positive or static audience their reported anxiety as a result

of the talk correlated with their usual anxiety in everyday life with respect to fear of

public speaking. However, irrespective of everyday life anxiety in relation to public

speaking, the general trend for those who experienced the negative audience was a very

strong anxiety reaction. The experimenters noted anecdotally that such subjects had

changes in body posture and skin colour, and overall demeanour after experiencing the

negative audience indicating a strong negative reaction.

More interesting than the statistical results were the comments made by the subjects

in debriefing interviews (Figure 5). It may seem that these comments are not surprising

– especially the reactions to the negative audience – for after all, their behaviour was

hostile. However, it is important to remember that there was no audience there. The

situation was entirely virtual. What happened expressed the power of the virtual reality

to evoke a response that was similar to that of reality – a ‘presence’ response.

Paranoid Ideation. Paranoid ideation is the typical pattern of thinking displayed in

cases of paranoia. It is characterised by suspiciousness and beliefs that one is being

followed, plotted against, persecuted, and so on. There are degrees of paranoid

tendencies in a population – ranging from none at all, all the way through to psychotic

illness. An experiment was carried out that tested whether the range of paranoid

thoughts typically present in a normal population (but excluding people with psychosis)

would be reproduced within a virtual reality65. The subjects, whose degree of paranoia

had been assessed in advance using standard scales, had a simple task to move through

13

a virtual library. The characters in the library would look at them and make some facial

expressions, maintaining a neutral attitude towards the subjects. Statistical analysis of

subsequent questionnaire results supported the hypothesis that paranoid thoughts were

triggered in the virtual reality in correlation with subjects’ propensities to experience

these in everyday reality. It should be noted that there were no sounds in these

environments – so that comments that some of the subjects heard were illusory. The

results were quite remarkable – people reacted strongly to the virtual characters, even

though objectively everyone knew that there were no people there at all. This study has

been followed up and repeated again, with publications pending66,67.

Post Traumatic Stress Disorder (PSTD). PSTD can occur when a person is subject

to life-threatening catastrophic events. People who develop PTSD have difficulties in

settling down to a normal life, have recurrent flashbacks, sleep disturbances and a host

of other debilitating symptoms. The group most studied in relation to virtual reality

therapy have been Vietnam War veterans. Two main scenarios have been developed – a

helicopter flight over hostile terrain and a clearing in the jungle where helicopters may

be landing. Desensitisation treatment of Vietnam War veterans led to significant

improvements in their symptoms68,69. A recent study with a survivor of the September

11th 2001 attack on New York also led to significant improvements although no longer

term follow up study was reported70.

Pain Distraction. The idea of presence in a virtual environment involves

transportation – it is as if the consciousness of the participant has been transported to a

place that is different to where their physical body is actually located, so that they feel

and act as if they are in that other place. Pain distraction is a potentially powerful

application of this process, since if the body is experiencing a situation that would be

experienced as pain in physical reality, but the person feels present in a virtual reality,

they should perceive less pain. There are some experimental studies that provide at least

preliminary evidence that lends weight to this. For example, pain during physical

therapy for severe burns treatment was signicantly reduced while subjects experienced

HMD delivered virtual reality studied 71. Similar results have been described in other

studies lending weight to the importance of immersion72,73, 74, 75. A typical strategy is to

present subjects with a situation that is counter to the kind of pain that they are

experiencing. For example, when undergoing burns treatment they may be placed in an

14

environment that depicts snow and ice, and have a task to throw snowballs76. Overall

although none of these studies are methodologically watertight, the evidence does point

in the direction that the ‘transportation of consciousness to another place’ involved in

presence in a virtual environment is strong enough to diminish sensations of pain.

Neuroscience and Presence Research

Presence research was initiated and has largely remained within the ambit of

technologically oriented research departments, and more recently has been of interest to

psychologists and clinical psychologists. The field has remained quite separate from

neuroscience, illustrated by the fact that not a single reference to presence research

appears in the neuroscience literature. With this review we wish to alter this situation

and introduce some of the overlapping areas between the two fields.

First it should be noted that VE technology provides an excellent tool in general for

neuroscience research. It allows the creation of experimental conditions for human

subjects that are almost identically replicable, under full control of the experimenter,

including the creation of scenarios and conditions that are either too expensive,

dangerous or even not possible at all in physical reality. It also easily supports the

creation of ‘magical’ scenarios – for example, the dissociation of sensory modalities

from one another. However, here we differentiate between the use of VEs as a tool in

neuroscience studies (as it is useful in other fields such as archaeology, design or

architecture) from the essential interactions between the issues that are subject of study

in the fields of presence and neuroscience research. A case in which VE can be just a

tool, for example, is the 3D visualization of biological structures and in this case in

particular, neural structures - both static and dynamic - which may report benefits in the

planning of surgical strategies77-79 or in general in the interpretation of morphological

data (pattern distribution, interneuronal relationships, etc). Other possible uses of VE as

a tool, but where at the same time in order to be successful presence is required, are

those applications in different types of training (neurosurgical training, emergency

responses), psychotherapy (see review above) or in neuro-rehabilitation80.

What we would like to highlight here are the conceptual interactions between the

apparently distant fields of presence and neuroscience research. We start with a

15

consideration of perception. In the presence studies reviewed above it is clear that the

characteristics of display of sensory data within a VE are critical to induce higher

presence – head tracking is better than no head tracking, stereo is better than mono,

faster frame-rate is better than lower frame-rate, shorter latency is better than longer

latency, sound is better than no sound, auditory display that takes account of the

individual person’s head and ear shape is better than a general auditory display function,

and interaction methods that utilise the body in a manner similar to real-world

interaction is preferable to traditional computer interface styles. In other words, data has

to be delivered to participants in forms that are structurally similar to real world sensory

data acquisition, and participants interact with the virtual world in a similar manner as

they would in the real world. However, one result was unexpected and surprising: the

realism of what is displayed seems to be far less important for presence. No one could

ever be fooled into believing in the reality of any virtual environment that is capable of

being displayed in real-time with today’s equipment. Yet people become anxious when

seeing a virtual precipice, are upset by virtual characters who behave badly in an

audience and even seem to generate paranoid thoughts in relation to such virtual people.

This result has led to the concept of minimal cues, the minimal elements that a VE

must include in order to induce presence81 and an area of work in presence research

since it is directly related to the economy of the system. How much can we simplify

‘reality’ within a VE and still induce presence in the subject? How many sensory

modalities should be stimulated to induce presence? Which minimal multi-sensory

stimulation works the best for different tasks? To any neuroscientist studying perception

it would seem obvious that such questions go to the roots of the mechanisms of sensory

perception: what are the building blocks of our perceived world? How much of our

perception is actually determined by the external world and how much by our internal

state? Or, what is the balance between bottom-up and top-down processing? Clearly

there must be minimal cues, since presence is reported in very simplistic environments,

and the fact that minimal cues are enough to induce presence implies that the absence of

part of the sensory information is not distracting, probably being filled in by cortical

processing82,83. Top-down influences on perception can also be investigated by giving

specific tasks or prior information to subjects going into VEs and subsequently

determining how their perception was affected. These studies can comprise from

perception of simple stimuli all the way up to perception of social situations, using

virtual characters. Actual stimuli received by the subject are reconstructed (including

16

eye tracked visual scenes) and compared to the perceived one as evaluated by different

means. Therefore, we could consider that when studying perception both, neuroscience

and presence research are asking similar questions from different starting points, and

additionally the use of VE provides a unique tool to be able to answer those questions.

The possibility of dissociation of stimuli that are invariably inseparable in the real

world opens many possibilities in the study of perception, and in particular in the study

of human bodily awareness or more generally, self-perception. In this area, virtual

reality can replace classical strategies such as the use of mirrors, panels or dummy

limbs84,85 with advantages. We talked earlier about studies that measured the impact of

different virtual body self-representations on the presence experienced by the subject in

the VE. The fact that subjects do still feel present in environments where their own

body is represented by some bizarre form implies that this ‘new’ body has been

somehow internalised. Future studies in VE on how we relate to our virtual body, to the

stimuli that it receives or how a virtual body is internalised can be of great importance

to understand how our own body internal representation is generated. By disrupting

distinct streams of information that are physiologically bound together (visual, motor,

propioceptive) we can learn about their individual contributions to the mechanisms of

perception and motor control. In the real world we calibrate our movements by feedback

that is provided by visual and propioceptive afferents, and by tactile information if we

are reaching for something. A virtual world can be programmed such that when our real

arm moves up, our virtual arm moves forward and therefore there would be a disruption

between visual and propioceptive information86. Subjects, however, adapt quite fast to

such disruptions, and learn to operate in the new conditions, so that the temporal

dynamics of those changes reflecting the plasticity of the system, could be measured.

The exploitation of VEs has also resulted in a useful tool in studies of brain

activation during different spatial navigation strategies87,88. In a broader sense, VEs

provide a frame in which to analyse which factors (auditory, visual, vestibular cues)

contribute to inducing spatial aspects of presence, or, in other words, what particular

sensory integration is critical to generate a sense of space. The use of VEs also allows

the dissociation of navigation from the proprioception associated to locomotion,

furthermore, it is compatible with inducing incorrect visual-proprioceptive cues, and

therefore it provides a tool to determine the role of visual or proprioceptive inputs in the

generation of internal spatial representations in humans.

17

Moving to the study of consciousness whatever definition is used, it includes several

layers that blend together in a unitary awareness that integrates the awareness of the self

and of the external world. The phenomenon of presence is based on a transportation of

consciousness into an alternative virtual reality. In a way then, presence is

consciousness in that virtual reality. The fact that the different layers (external world,

self representation and even part of the extended self) can be altered in a highly

controlled form in a VE means that virtual reality can be exploited to analyse

scientifically the basis of consciousness. We propose that precisely because of its

restricted form, it may be more tractable for study than consciousness itself. For

example, in one well-known approach it is said that “Consciousness occurs when we

can generate, automatically, the sense that a given stimulus is being perceived in a

personal perspective; the sense that the stimulus is ‘owned’ by the organism involved in

the perceiving; and, last but not least, the sense that the organism can act on the

stimulus (or fail to do so), that is, the sense of ‘agency’.” 89 Every aspect of this occurs

when a person is immersed in a VE, and not only does it occur but many aspects of this

can be manipulated experimentally.

Presence occurs when what is said about consciousness occurs within the domain of

a virtual reality. What provisional conclusions can we reach about this? First, that it

does occur. Immersion in a VE can transform the consciousness of a person in the sense

that they respond to the virtual place and to events within that place, and feel their body

to be part of that place, even transform their body ownership to the body that they see in

that place. This process can have observable impact on the real body of the person, in

terms of conscious and volitional behaviours in which they engage (e.g., deciding to

walk around the pit rather than fly across it), through to non-conscious behaviours such

as changes in heart rate, breathing and skin conductance response. The transformation

of consciousness can be to such an extent that pain that should be experienced as a

result of activities taking place in the real body is suppressed, and in contrast there is

evidence to suggest that events that are activated on the virtual body may result in the

sensation of pain85 (though such an experiment is yet to be carried out in a computer-

generated virtual environment). Presence occurs when there is successful substitution of

real sensory data by computer generated sensory data, and in a way that people can

engage in normal motor actions in order to carry out actions. By ‘successful’ we mean

that the person responds to the virtual stimuli in a manner similar to how they would

respond to the corresponding real stimuli. Response should be considered at every level,

18

from unconscious physiological behaviours, through automatic reactions, conscious

volitional behaviours, through to cognitive processing - including the sense of ‘being

there’.

What is also interesting is that this takes place in spite of every participant’s

absolute knowledge that the virtual environment is fake, is not really there in the

everyday meaning of the word. Recall also that these virtual environments are not only

fake, but they appear to be fake, certainly in the visual sense. A virtual character does

not look or behave much like a real character for example. This implies that the

mechanisms that govern the domain of consciousness (i.e., the place and situation to

which it is attached at any moment) must have as a necessary condition that

combination of parameters that govern the formation of a virtual environment.

Consciousness will not transform when, for example, the frame rate is below a certain

level, or when the field of view is too narrow – but it may do so when these (and other

structural factors) are at appropriate settings, independently of the level of realism of the

content displayed in the virtual environment. In this situation a person acts as if

conscious in the place and situation depicted by the VE even if at some level there is

awareness that in fact this situation is not really happening and the place is not really

there. The concept of breaks in presence provides a way to analyse those moments when

‘reality’ breaks through, so that consciousness shifts back to the surrounding physical

environment.

Conclusions

We argue that presence research should be opened up, beyond only the domain of

computer science and other technologically oriented disciplines, and become a

mainstream part of neuroscience. Of course, virtual reality can and is being used as a

tool for neuroscience studies. Many of these studies, such as of perception, way-finding,

self representation and sense of self, will also contribute to the understanding of

presence. But, and this is our main point – not only is virtual reality a tool for

neuroscience, but the presence experience that it engenders is also an object of study in

its own right. Moreover, this concept of presence is sufficiently similar to consciousness

that it may help to transform research within that domain.

19

References

1. Minsky, M. Telepresence. Omni, 45-52 (1980).

2. Ellis, S.R. Nature and Origin of Virtual Environments: A Bibliographic Essay.

Computing Systems in Engineering 2, 321-347 (1991).

3. Blauert, J., Lehnert, H., Sahrhage, J. & Strauss, H. An interactive virtual-

environment generator for psychoacoustic research. I: Architecture and

implementation. Acustica 86, 94-102 (2000).

4. Salisbury, J.K. & Srinivasan, M.A. Phantom-based haptic interaction with

virtual objects. Ieee Computer Graphics and Applications 17, 6-10 (1997).

5. Lin, M. & Salisbury, K. Haptic rendering - Beyond visual computing. Ieee

Computer Graphics and Applications 24, 22-23 (2004).

6. Laycock, S.D. & Day, A.M. Recent Developments and Applications of Haptic

Devices. in Computer Graphics Forum Vol. 22 117-132 (2003).

7. Draper, J.V., Kaber, D.B. & Usher, J.M. Telepresence. Human Factors 40, 354-

375 (1998).

8. Held, R.M. & Durlach, N.I. Telepresence. Presence: Teleoper. Virtual Environ.

1, 109-112 (1992).

9. Sheridan, T.B. Musings on Telepresence and Virtual Presence. Presence-

Teleoperators and Virtual Environments 1, 120-126 (1992).

10. Ellis, S.R. Presence of mind: A reaction to Thomas Sheridan's ''further musings

on the psychophysics of presence''. Presence-Teleoperators and Virtual

Environments 5, 247-259 (1996).

11. Sheridan, T.B. Further musings on the psychophysics of presence. Presence-

Teleoperators and Virtual Environments 5, 241-246 (1996).

12. Slater, M. & Wilbur, S. A framework for immersive virtual environments

(FIVE): Speculations on the role of presence in virtual environments. Presence-

Teleoperators and Virtual Environments 6, 603-616 (1997).

13. Zahorik, P. & Jenison, R.L. Presence as being-in-the-world. Presence-

Teleoperators and Virtual Environments 7, 78-89 (1998).

14. Flach, J.M. & Holden, J.G. The reality of experience: Gibson's way. Presence-

Teleoperators and Virtual Environments 7, 90-95 (1998).

20

15. Slater, M., Steed, A., McCarthy, J. & Maringelli, F. The influence of body

movement on subjective presence in virtual environments. Hum Factors 40,

469-77 (1998).

16. Schubert, T., Friedmann, F. & Regenbrecht, H. The experience of presence:

Factor analytic insights. Presence-Teleoperators and Virtual Environments 10,

266-281 (2001).

17. Slater, M., Usoh, M. & Steed, A. Depth of Presence in Immersive Virtual

Environments. Presence-Teleoperators and Virtual Environments 3, 130-144

(1994).

18. Slater, M. & Usoh, M. Representation Systems, Perceptual Position and

Presence in Virtual Environments. Presence-Teleoperators and Virtual

Environments 2, 221-234 (1994).

19. Slater, M., Usoh, M. & Steed, A. Taking steps: the influence of a walking

technique on presence in virtual reality. ACM Trans. Comput.-Hum. Interact. 2,

201-219 (1995).

20. Barfield, W., Baird, K.M. & Bjorneseth, O.J. Presence in virtual environments

as a function of type of input device and display update rate. Displays 19, 91-98

(1998).

21. Slater, M., Usoh, M. & Chrysanthou, Y. The influence of dynamic shadows on

presence in immersive virtual environments. in Selected papers of the

Eurographics workshops on Virtual environments '95 8-21 (Springer-Verlag,

Barcelona, Spain, 1995).

22. Welsh, R.B., Blackman, T.T., Liu, A., Mellers, B.A. & Stark, L.W. The Effects

of Pictorial Realism, Delay of Visual Feedback, and Observer Interactivity on

the Subjective Sense of Presence. Presence-Teleoperators and Virtual

Environments 5, 263-273 (1996).

23. Slater, M. & Usoh, M. Presence in Immersive Virtual Environments. in Ieee

Virtual Reality Annual International Symposium 90-96 (1993).

24. Barfield, W. & Hendrix, C. The Effect of Update Rate on the Sense of Presence

within Virtual Environments. Virtual Reality: The Journal of the Virtual Reality

Society 1, 3-16 (1995).

25. Lessiter, J., Freeman, J., Keogh, E. & Davidoff, J. A cross-media presence

questionnaire: The ITC-Sense of Presence Inventory. Presence-Teleoperators

and Virtual Environments 10, 282-297 (2001).

21

26. Slater, M. & Usoh, M. Presence in Immersive Virtual Environments. in

Proceedings of the IEEE Conference - Virtual Reality Annual International

Symposium 90-96 (IEEE Neural Networks Council, Seattle, WA, 1993).

27. Witmer, B.G. & Singer, M.J. Measuring presence in virtual environments: A

presence questionnaire. Presence-Teleoperators and Virtual Environments 7,

225-240 (1998).

28. Singer, M.J. & Witmer, B.G. On selecting the right yardstick. Presence-

Teleoperators and Virtual Environments 8, 566-573 (1999).

29. Slater, M. Measuring presence: A response to the Witmer and Singer presence

questionnaire. Presence-Teleoperators and Virtual Environments 8, 560-565

(1999).

30. Freeman, J., Avons, S.E., Pearson, D.E. & IJsselsteijn, W.A. Effects of sensory

information and prior experience on direct subjective ratings of presence.

Presence-Teleoperators and Virtual Environments 8, 1-13 (1999).

31. Usoh, M., Catena, E., Arman, S. & Slater, M. Using presence questionnaires in

reality. Presence-Teleoperators and Virtual Environments 9, 497-503 (2000).

32. Slater, M. How colorful was your day? Why questionnaires cannot assess

presence in virtual environments. Presence-Teleoperators and Virtual

Environments 13, 484-493 (2004).

33. Prothero, J., Parker, D., Furness, T. & Wells, M. Towards a Robust, Quantitative

Measure of Presence. in Proceedings of the Conference on Experimental

Analysis and Measurement of Situational Awareness 359-366 (Embry-Riddle

Aeronautical University Press, Datona Beach, FL, 1995).

34. Freeman, J., Avons, S.E., Meddis, R., Pearson, D.E. & IJsselsteijn, W.I. Using

behavioral realism to estimate presence: A study of the utility of postural

responses to motion stimuli. Presence-Teleoperators and Virtual Environments

9, 149-164 (2000).

35. Welch, R.B. The Presence of Aftereffects. in Proceedings of the Seventh

International Conference on Human-Computer Interaction-Volume 1 - Volume I

273-276 (Elsevier Science Inc., 1997).

36. Meehan, M., Insko, B., Whitton, M. & Brooks, F.P. Physiological measures of

presence in stressful virtual environments. Acm Transactions on Graphics 21,

645-652 (2002).

22

37. Slater, M. & Steed, A. A virtual presence counter. Presence-Teleoperators and

Virtual Environments 9, 413-434 (2000).

38. Slater, M., Brogni, A. & Steed, A. Physiological Responses to Breaks in

Presence: A Pilot Study. in The 6th Annual International Workshop on Presence

Vol. 2003 (Aalborg, Denmark, 2003).

39. Meehan, M., Razzaque, S., Whitton, M.C. & Frederick P. Brooks, J. Effect of

Latency on Presence in Stressful Virtual Environments. in Proceedings of the

IEEE Virtual Reality 2003 141 (IEEE Computer Society, 2003).

40. Hendrix, C. & Barfield, W. Presence within Virtual Environments as a Function

of Visual Display Parameters. Presence-Teleoperators and Virtual

Environments 5, 274-289 (1996).

41. Barfield, W., Hendrix, C. & Bystrom, K.E. Effects of stereopsis and head

tracking on performance using desktop virtual environment displays. Presence-

Teleoperators and Virtual Environments 8, 237-240 (1999).

42. Ijsselsteijn, W., de Ridder, H., Freeman, J., Avons, S.E. & Bouwhuis, D. Effects

of stereoscopic presentation, image motion, and screen size on subjective and

objective corroborative measures of presence. Presence-Teleoperators and

Virtual Environments 10, 298-311 (2001).

43. Welch, R.B., Blackmon, T.T., Liu, A., Mellers, B.A. & Stark, L.W. The effects

of pictorial realism, delay of visual feedback, and observer interactivity an the

subjective sense of presence. Presence-Teleoperators and Virtual Environments

5, 263-273 (1996).

44. Zimmons, P. & Panter, A. The Influence of Rendering Quality on Presence And

Task Performance in a Virtual Environment. in Proceedings of the IEEE Virtual

Reality 2003 293 (IEEE Computer Society, 2003).

45. Hendrix, C. & Barfield, W. The sense of presence within auditory virtual

environments. Presence-Teleoperators and Virtual Environments 5, 290-301

(1996).

46. Väljamäe, A., Larsson, P., Västfjäll, D. & Kleiner, M. Auditory Presence,

Individualized Head-Related Transfer Functions, and Illusory Ego-Motion in

Virtual Environments. in 7th International Conference on Presence -

Proceedings of Presence 2004 141-147 (Valencia, Spain, 2004).

23

47. Gilkey, R.H. & Weisenberger, J.M. The Sense of Presence for the Suddenly

Deafened Adult - Implications for Virtual Environments. Presence-

Teleoperators and Virtual Environments 4, 357-363 (1995).

48. Murray, C.D., Arnold, P. & Thornton, B. Presence accompanying induced

hearing loss: Implications for immersive virtual environments. Presence-

Teleoperators and Virtual Environments 9, 137-148 (2000).

49. Basdogan, C., Ho, C.-H., Srinivasan, M.A. & Slater, M. An experimental study

on the role of touch in shared virtual environments. ACM Trans. Comput.-Hum.

Interact. 7, 443-460 (2000).

50. Durlach, N. & Slater, M. Presence in shared virtual environments and virtual

togetherness. Presence-Teleoperators and Virtual Environments 9, 214-217

(2000).

51. Kim, J. et al. Transatlantic touch: A study of haptic collaboration over long

distance. Presence-Teleoperators and Virtual Environments 13, 328-337 (2004).

52. Sallnas, E.-L., Rassmus-Grohn, K. & Sjostrom, C. Supporting presence in

collaborative environments by haptic force feedback. ACM Trans. Comput.-

Hum. Interact. 7, 461-476 (2000).

53. Usoh, M. et al. Walking > walking-in-place > flying, in virtual environments. in

Proceedings of the 26th annual conference on Computer graphics and

interactive techniques 359-364 (ACM Press/Addison-Wesley Publishing Co.,

1999).

54. Rizzo, A. & Buckwalter, J.G. Special issue: Virtual reality applications in

neuropsychology - Guest editors' introduction. Presence-Teleoperators and

Virtual Environments 10, Iii-V (2001).

55. Krijn, M., Emmelkamp, P.M.G., Olafsson, R.P. & Biemond, R. Virtual reality

exposure therapy of anxiety disorders: A review. Clinical Psychology Review

24, 259-281 (2004).

56. Hodges, L.F. et al. Virtual Environments for Treating the Fear of Heights.

Computer 28, 27-34 (1995).

57. Rothbaum, B.O. et al. Effectiveness of Computer-Generated (Virtual-Reality)

Graded Exposure in the Treatment of Acrophobia. American Journal of

Psychiatry 152, 626-628 (1995).

24

58. Emmelkamp, P.M.G. et al. Virtual reality treatment versus exposure in vivo: a

comparative evaluation in acrophobia. Behaviour Research and Therapy 40,

509-516 (2002).

59. Rothbaum, B.O., Hodges, L., Smith, S., Lee, J.H. & Price, L. A controlled study

of virtual reality exposure therapy for the fear of flying. Journal of Consulting

and Clinical Psychology 68, 1020-1026 (2000).

60. Rothbaum, B.O., Hodges, L., Anderson, P.L., Price, L. & Smith, S. Twelve-

month follow-up of virtual reality and standard exposure therapies for the fear of

flying. Journal of Consulting and Clinical Psychology 70, 428-432 (2002).

61. Carlin, A.S., Hoffman, H.G. & Weghorst, S. Virtual reality and tactile

augmentation in the treatment of spider phobia: A case report. Behaviour

Research and Therapy 35, 153-158 (1997).

62. Hoffman, H.G., Garcia-Palacios, A., Carlin, A., Furness, T.A. & Botella-

Arbona, C. Interfaces that heal: Coupling real and virtual objects to treat spider

phobia. International Journal of Human-Computer Interaction 16, 283-300

(2003).

63. Garcia-Palacios, A., Hoffman, H., Carlin, A., Furness, T.A. & Botella, C.

Virtual reality in the treatment of spider phobia: a controlled study. Behaviour

Research and Therapy 40, 983-993 (2002).

64. Pertaub, D.P., Slater, M. & Barker, C. An experiment on public speaking

anxiety in response to three different types of virtual audience. Presence-

Teleoperators and Virtual Environments 11, 68-78 (2002).

65. Freeman, D. et al. Can virtual reality be used to investigate persecutory ideation?

J Nerv Ment Dis 191, 509-14 (2003).

66. Freeman, D. et al. The psychology of persecutory ideation II: A virtual reality

experimental study. J Nerv Ment Dis in press(2005).

67. Freeman, D. et al. The psychology of persecutory ideation I: A questionnaire

study. J Nerv Ment Dis in press(2005).

68. Rothbaum, B.O. et al. Virtual reality exposure therapy for PTSD Vietnam

veterans: A case study. Journal of Traumatic Stress 12, 263-271 (1999).

69. Rothbaum, B.O., Hodges, L.F., Ready, D., Graap, K. & Alarcon, R.D. Virtual

reality exposure therapy for Vietnam veterans with posttraumatic stress disorder.

Journal of Clinical Psychiatry 62, 617-622 (2001).

25

70. Difede, J. & Hoffman, H.G. Virtual reality exposure therapy for World Trade

Center post-traumatic Stress Disorder: A case report. Cyberpsychology &

Behavior 5, 529-535 (2002).

71. Hoffman, H.G., Patterson, D.R. & Carrougher, G.J. Use of virtual reality for

adjunctive treatment of adult burn pain during physical therapy: A controlled

study. Clinical Journal of Pain 16, 244-250 (2000).

72. Hoffman, H.G., Garcia-Palacios, A., Kapa, V., Beecher, J. & Sharar, S.R.

Immersive virtual reality for reducing experimental ischemic pain. International

Journal of Human-Computer Interaction 15, 469-486 (2003).

73. Hoffman, H.G. et al. Water-friendly virtual reality pain control during wound

care. Journal of Clinical Psychology 60, 189-195 (2004).

74. Hoffman, H.G., Doctor, J.N., Patterson, D.R., Carrougher, G.J. & Furness, T.A.

Virtual reality as an adjunctive pain control during burn wound care in

adolescent patients. Pain 85, 305-309 (2000).

75. Gershon, J., Zimand, E., Pickering, M., Rothbaum, B.O. & Hodges, L. A pilot

and feasibility study of virtual reality as a distraction for children with cancer.

Journal of the American Academy of Child and Adolescent Psychiatry 43, 1243-

1249 (2004).

76. Hoffman, H.G. et al. Modulation of thermal pain-related brain activity with

virtual reality: evidence from fMRI. Neuroreport 15, 1245-8 (2004).

77. Burchiel, K. Virtual reality. J Neurosurg 100, 583; discussion 583-4 (2004).

78. Spicer, M.A., van Velsen, M., Caffrey, J.P. & Apuzzo, M.L. Virtual reality

neurosurgery: a simulator blueprint. Neurosurgery 54, 783-97; discussion 797-8

(2004).

79. Kockro, R.A. et al. Planning and simulation of neurosurgery in a virtual reality

environment. Neurosurgery 46, 118-35; discussion 135-7 (2000).

80. Rose, F.D., Attree, E.A. & Johnson, D.A. Virtual reality: an assistive technology

in neurological rehabilitation. Curr Opin Neurol 9, 461-7 (1996).

81. Slater, M. Presence and the sixth sense. Presence-Teleoperators and Virtual

Environments 11, 435-439 (2002).

82. Ramachandran, V.S. & Gregory, R.L. Perceptual filling in of artificially induced

scotomas in human vision. Nature 350, 699-702 (1991).

83. De Weerd, P., Desimone, R. & Ungerleider, L.G. Perceptual filling-in: a

parametric study. Vision Res 38, 2721-34 (1998).

26

84. Ramachandran, V.S. & Rogers-Ramachandran, D. Synaesthesia in phantom

limbs induced with mirrors. Proc R Soc Lond B Biol Sci 263, 377-86 (1996).

85. Armel, K.C. & Ramachandran, V.S. Projecting sensations to external objects:

evidence from skin conductance response. Proceedings of the Royal Society of

London Series B-Biological Sciences 270, 1499-1506 (2003).

86. Sober, S.J. & Sabes, P.N. Multisensory integration during motor planning. J

Neurosci 23, 6982-92 (2003).

87. Hartley, T., Maguire, E.A., Spiers, H.J. & Burgess, N. The well-worn route and

the path less traveled: distinct neural bases of route following and wayfinding in

humans. Neuron 37, 877-88 (2003).

88. Shelton, A.L. & Gabrieli, J.D. Neural correlates of encoding space from route

and survey perspectives. J Neurosci 22, 2711-7 (2002).

89. Damasio, A.R. Investigating the biology of consciousness. Philosophical

Transactions of the Royal Society of London Series B-Biological Sciences 353,

1879-1882 (1998).

Figure 1| The left pane shows a Cave-like system. This specific model is a Trimension ReaCTor. This system consists of 4 projection surfaces, the front, left and right walls are back-projected acrylic screens while the painted wooden floor is projected from above. The screens are seamlessly joined to provide a continuous projection surface. Only the top and the rear faces of the cube are not projection surfaces. This system is driven by an SGI Onyx 2 computer. The right pane shows a particular environment (a kitchen) being projected into the ReaCTor. The image is refreshed at 90Hz. A left eye image is shown at 45Hz and a right eye image also at 45Hz. The participant is wearing CrystalEyes stereo shutter glasses which are controlled by infrared signals to be synchronised with the display refresh. Hence the left eye lens is only open during the left eye image display and similarly for the right eye. The participant is wearing an Intersense IS 900 tracking device attached to the top of the glasses. The head-position and orientation is tracked at approximately 120Hz and this information is relayed to the computer at a latency of about 4ms. The computer graphics software uses this information to compute and display the left- and right-eye images. Since the display is from the point of view of the tracked participant the perspective projection does not seem to be correct when looked at by another person. Hence the distortions in the image above. The participant is also holding a hand tracking device which similarly relays its position and orientation to the main computer. This device has buttons that can be programmed to initiate events (such as virtual locomotion, and also can be used for collision detection with virtual objects thus allowing the participant to interact with objects in the scene.

Figure 2a – This shows a person in the ReaCTor Cave-like system, with all the projectors off.

Figure 2b – When the projectors are on the person may be transported into a different world, here interacting with virtual characters in a party scenario.

Figure 3 |The left pane shows a participant wearing a head-mounted display. This ideally blocks out all surrounding light. There are two images displayed, one for the left eye and the other for the right eye. Each image cannot be seen at all by the opposing eye. The head-mounted display position and orientation is tracked, and the computer updates the image according to perspective projections from the point of view of each eye. The right pane shows a participant in a Cave-like environment. He is wearing 3D shutter glasses and also physiological monitoring equipment so that real-time read-outs of his EKG, GSR and respiration are available. The background scene shows a virtual party which has been used recently in a study of social phobia.

Figure 4 - The ‘pit-room’. This is a virtual environment often used to assess presence. The participant enters into the left-most room which can be used for familiarisation with the virtual environment system and the learning of any procedures such as how to move around or select objects. The participant is then given task to go into the next room, select an object that is left on plank overlooking the pit, and then take it to the chair on the other side. The reactions to the pit are quantified in different forms: behaviourally, through physiological measurements (heart and respiratory rate, galvanic skin response, etc) or through subsequent questionnaires. In the experiments that have been carried out almost all make their way to the chair by edging themselves around the sides of the room along the ledge, even though all subjects know for sure that there is no pit there. The smaller insets show ‘static haptics’ where participants were positioned by small but real ledges, adding to the effect of standing over a real pit.

Figure 5a | A snapshot of the positive audience

Figure 5b | A snapshot of the negative audience

Virtual environment representing a seminar audience. Although the characters certainly do

not look realistically human their behaviour has been modelled on observations from real meetings, though greatly exaggerated – both positively and negatively. In these studies the subjects are asked to give short talks to the virtual audience, that could have a positive, negative or neutral attitude towards the speaker. A response to the audience similar to the one that the same person would have to a real audience was subsequently evaluated as a high presence in the virtual environment. The testimonies shown below illustrate that the responses to the virtual characters went above all rational considerations of what is real, along with the physiological responses measured in the subjects illustrated the fact that people responded similarly to a real environment. Some Responses to the Positive Audience

‘It was clear that the audience was really positive and interested in what I was saying and it made you feel like telling them what you know.’

‘I felt great. Finally nobody was interrupting me. Being a woman, people keep interrupting you in talks much more… But here I felt people were there to listen to me.’

‘They were staring at me. They loved you unconditionally, you could say anything, you didn’t have to work’.

Some Responses to the Negative Audience

‘It felt really bad. I couldn’t just ignore them. I had to talk to them and tell them to sit up and pay attention. Especially the man on the left who put his head in his hands; I had to ask him to sit up and listen…. I entered a negative feedback loop where I would receive bad responses from the audience and my performance would get even worse…. I was performing really badly and that doesn’t normally happen.’

‘I was upset, really thrown. I totally lost my train of thought. They weren’t looking at me and I didn’t know what to do. Should I start again? I was very frustrated. I felt I had no connection to them. They weren’t looking at me. I just forgot what I was talking about.’

Figure 6 | Virtual environment representing a library that was used in a CAVE during paranoia studies. The subject is asked to walk around the library and report afterwards. Virtual characters in the library do occasionally look at the subject and change facial expressions such as smiling or other emotionally neutral expressions. No sound was delivered during the study.

Here we quote some of the remarks made by the subjects in post-experimental interviews that illustrate the

sense of presence that people felt in the environment and their responses to these visually quite unrealistic virtual characters.

“The two people to the left, I didn’t like them very much – well, I don’t know, maybe because when I entered the room I felt I was being watched and then they started talking about me. The other people were more neutral and more inviting except the guy with the beard.”

“It was probably more real to me than I expected it to be. At some point, I was trying to navigate around a table and almost found myself saying sorry to the person sitting there. I felt that they were getting annoyed with me for doing that…”

“It was really weird, because they were all definitely in on something and they were all trying to make me nervous. It was clear that they were trying to mock me, they kept on looking at me and when I looked back, they were uuhh… The guy with the suit was really weird because he kept smiling at me and it was quite sinister.”

Box 1 – Basic Functions of a Virtual Reality System

A virtual reality system typically delivers a left and right eye image forming a stereo pair, ideally with an active stereo system – one in which there is no leakage of the left eye image to the right eye, and vice-versa. The images are generated within the graphics pipeline of a computer system, and updated in real-time. The computer maintains a data base that describes a particular scene – the data base containing geometric, radiant, acoustic, behavioural and physical information about the set of objects comprising the scene. The images that the participant sees are renderings of the data base, perspective projections of the 3D geometry onto the 2D displays, with objects coloured according to computer graphics lighting models. The rendering is from the point of view determined by the head position and orientation of the participant, which must be tracked in real-time by a tracking system. The tracking system continually sends a stream of head-position and orientation data to the computer which is therefore able to generate the appropriate stereo images. Virtual objects are encapsulations of particular nodes in the data base and the programming scripts that determine their behaviour. They typically represent meaningful aspects of the environment – for example, some geometry plus associated radiant information may represent a chair. Some objects may be passive (for example, representing walls of a room) and others active – for example, representing virtual people. The human participant can therefore interact with objects in the environment to a greater or lesser degree. Typically nothing can be done with entirely passive objects such as the walls of a room, but a chair, for example, may be ‘picked up’ and moved to another location, and the representation of a virtual human may be aware of the participant, speak to him or her, and may have various levels of behaviour. Of course the program that controls a virtual human will have access to where the head of the real human is located and its orientation direction because it will have access to the head-tracking data. In order for the participant to be able to interact with the environment – such as picking up objects – there must be additional tracking beside head-tracking. Typically at least one hand is also tracked, often simply by the person holding the equivalent of a ‘mouse’, a 3D pointing device with buttons like a mouse which itself has a 6 degrees of freedom tracker attached to it.

Box 2 Example of Some Questions from a Presence Questionnaire 1. To what extent did you have a sense of being in place X? Not at all 1 2 3 4 5 6 7 Very much so

2. To what extent were there times during the experience when X became the ‘reality’ for you, and you almost forgot about the ‘real world’ of the laboratory in which the whole experience was really taking place? Never 1 2 3 4 5 6 7 Almost all the

time

3. When you think back about your experience, do you think of the virtual X more as images that you saw, or more as somewhere that you visited ? Only as images that I saw

1 2 3 4 5 6 7 Somewhere that I visited