Place Illusion and PlausibilityMel Slater

Department of Computer Science, University College London

Catalan Institute of Research and Advanced Studies (ICREA)University of Barcelona

www.event-lab.org

Outline

• A different look at immersion• Concepts • Experiments on Virtual Body• Discussion

Immersive Systems• Immersive systems

can be characterised by the sensorimotor contingencies (SC) that they support.

• SCs refer to the actions necessary for perception of the virtual world – you know to change gaze direction by moving your head

Immersive Systems

• Immersive systems can in principle be put into partial orders.

• A system of order 1 has SCs that simulate those of physical reality. (‘Simulation’ implies that it is not exact or necessarily complete).

• A system of order i < j can be used to simulate the SCs of j, but j cannot be used to simulate the SCs of i.

• For example, a desktop system can in principle be simulated within a Cave.

• Any immersive system of whatever order will have a set of SCs that enable perception of the virtual world within that system.

• Desktop might be joystick and keys

Qualia

• Describe what it feels to see the colour red.• To see red and to know you’re seeing red• Eg, you may stop at a traffic light automatically,

without consciously being aware of seeing the ‘redness’ of the light

• But if you choose to become aware of it, you can• This is what philosophers and neuroscientists who

study ‘consciousness’ call QUALIA.

Concepts – Place Illusion• Place Illusion (π) - refers to the illusion of being in the

virtual place even though you know for sure that you are not there.

• It is a strong illusion• One that you cannot stop except by shutting out sensory input• You cannot stop it by ‘knowing’ that it is false

• It gives rise to a feeling, a qualia of “ ‘being there’ but knowing that you are not there”

• OMNIMAX, Disneyworld, very large display screens, immersive virtual reality … all give rise to this illusion.

• It is like actually being there – it is not a concept that applies to physical reality.

Place Illusion

• π is not difficult to achieve• When perceiving an environment through sensorimotor

contingencies that are approximately the same as in physical reality

• “The basic claim of the enactive approach is that the perceiver’s ability to perceive is constituted (in part) by sensorimotor knowledge (i.e., by practical grasp of the way sensory stimulation varies as the perceiver moves).” (Action in Perception, A. Noe, p12).

• With SCs similar to those of physical reality • PI will occur automatically – like many visual illusions nothing

special has to be done to experience it

www.lottolab.org

www.lottolab.org

π Can Break

• When people perceptually probe they will inevitably break the illusion

• Latency failure in head tracking• Look at something closer than resolution permits• Touch something and feel nothing• Feel something that should not be there (weight of

helmet, cables, Cave walls)• SCs provide a framework in which π might occur

• It is not deterministic• Our experience is that it quickly recovers

PI in Non-Immersive Systems• Is there PI in Second Life or computer games (as used on

desktop systems)?• If we consider PI as bound to normal SCs for perception

then the answer is ‘no’.• Note that it is possible to simulate the playing of a

computer game itself inside an immersive system.• In an immersive system you can pull up a chair, sit down,

switch on the computer, etc.. The limitation is today’s technology, but in principle it is possible.

• Systems form a simulation hierarchy• PI with respect to the immersive system and the response

to the desktop system must be qualitatively different.

Presence in Non-Immersive Systems

• “The basic claim of the enactive approach is that the perceiver’s ability to perceive is constituted (in part) by sensorimotor knowledge (i.e., by practical grasp of the way sensory stimulation varies as the perceiver moves.” (Action in Perception, A. Noe, p12).

• This does also happen in computer games, but the type of sensorimotor knowledge is different.

• To turn look to the left you use a joystick or key board presses, you don’t turn your head.

• A whole new set of knowledge is established (eventually becomingunconsciously competent), which establishes a particular SC set.

• Of course typically this relies on knowledge from normal everyday SC (but it need not, the mappings could be arbitrary in principle – it only affects the time to expertise, or the learnability of the system).

PI in Non-Immersive Systems• Does PI occur in the scenario of the computer game?• Yes, but it is conditional, your actions are bound to the SCs

that are possible.• You can be present in SL, but it is at a different conceptual and

physical level.• Let’s call an immersive system - first order. In a first order

system you can simulate physical reality and your SCs in relation to perceiving it are normal.

• A second order system can be simulated within a first order system. It will have its own SCs.

• In any order system there can be engagement, interest, flow, …• Only in a first order system can there be PI.

The Body• The body is the fulcrum of (π,ψ) • In an immersive virtual reality there is something very simple that you can do…

• Look down, and you will see your (virtual) body.

• This is an action within the continuum of SC normal to that type of reality.

• There is a continuity between your body and the rest of the environment.

• Using normal actions you know how to change your sensory stimulation as a function of your actions, and one action with no special status or discontinuity from other actions, is to look at yourself.

The Body• In a second order system such as a

computer game, there are also means to look at ‘yourself’, within the set of SC defined by that world.

• Now there is a two level relationship – you can in a first order system see your body,• while using normal SC actions be looking at

yourself in the second order system.

The Body• In other words within each SC system that is tied to a

particular form of virtual reality, there is in principle a way to perceive yourself as being part of that reality,

• since you use the sensorimotor knowledge to look at yourself in a way no different from how you would look at anything else. Hence you must ‘be there’ in that reality.

• But since the realities form a hierarchy, • and the first order systems are simulations of physical

reality, • it is only a first order system that can in a physical sense,

have the property to induce ‘being there’ (Place Illusion).

The Body• Putting it another way: PI in a first order system is not a

mental inference, an emotional feeling, an extrapolation, a memory, an illusion of being there – it a property of the physics of the situation.

• There are lawful relationships between movements and sensory stimulation, including the perception of your physical embodiment (self).

• There will be PI if those laws are similar to those of physical reality, and your body is perceivable as being part of this reality.

• PI is not the whole story …

Plausibility• Plausibility (ψ) is your subjective

probability that observed events are ‘true’

• It is the illusion that situations and events are what they appear to represent

• They are ‘really happening’ as they appear to be happening

• Can apply in real life (‘con tricks’), theatre, cinema, etc and in mixed/virtual reality

ψ Can Break

• Incongruence of events / appearances with expectation

• Lack of correlation with own actions

• Lack of consistency in cause-effect relationships

• …

The Stepford Wives, 1975, Bryan Forbes (Paramount)

Brain as Bayesian Inference Engine• People do not really believe

• that they are in the virtual place (π)• Nor that what might appear to be happening is true (ψ)

• These are illusions, • probably formed by the brain as a Bayesian engine,

continually updating probabilities in the light of evidence

• Cognitive knowledge overrides these automatic inferences possibly dampening down responses.

Hypothesis• Our major hypothesis is that

• When π and ψ both occur• People will respond as if what is happening is real.

• This is not deterministic – it says that π and ψ set a framework in which this might occur – but they are easily broken through probing

• correlation between participant actions and responses is necessary – for both π and ψ

• Can a virtual body or limb feel like it is your own?Question

The Rubberhand Illusion(Botvinick and Cohen, Nature, 391: 756 1998)

Armel and Ramachandran Proc Biol Sci. 270:1499, 2003

Rubber Hand Illusion

• Can the same idea work when the hand andeven the visual stimulation is entirelyvirtual?• Slater M, Perez-Marcos D, Ehrsson HH,

Sanchez-Vives MV. Front Hum Neurosci. 2008;2:6. Epub 2008 Aug 20.

Virtual Arm Illusion

n=21

Questionnaire Results• 1. At some point I had the

feeling that I was receiving the hits in the location in which the virtual arm was.

• 2. During the experiment there were moments in which it seem as if what I was feeling were caused by the yellow ball that I was seeing in the screen.

• 3. During the experiments there were moments in which I felt as if the virtual arm would be my own arm.

Questionnaire Results• 1. At some point I had the

feeling that I was receiving the hits in the location in which the virtual arm was.

• 2. During the experiment there were moments in which it seem as if what I was feeling were caused by the yellow ball that I was seeing in the screen.

• 3. During the experiments there were moments in which I felt as if the virtual arm would be my own arm.

When the tapping is asynchronous with visual feedback the illusion vanishes (n=20) Synchronous: Median D = 30mm

Asynchronous Median D = 0mm

EMG Recordings

• The stimulation continued for 5mins• Then the virtual arm rotated for 12s• EMG is recorded throughout• Each participant had been fitted with an

active electrode in order to record EMG • on the forearm plus a reference electrode on the

forearm and a ground on the biceps.

Activity During Rotation

• EMG analysis methods usually assess voluntary contractions by subjects

• Here we are looking for probably involuntary small movements caused by seeing the virtual arm rotate

• Looked at the number of activity onsets before and after the arm started movingDi Fabio (1987)

• Sustained above threshold on the rectified low-pass filtered signal

Subjective Illusion and EMG

Subjective Illusion and EMG

• The proportion of activity onsets is not greater during the arm moving than before

• While the arm is moving there is a significant correlation with the mean subjective response

• Interesting – suggests that this is worth following up in later work

• There is no connection for the asynchronous condition

• There is no connection with the non-illusion questions in the synchronous condition

Out of Body Illusion• Can ownership extend to a full body?• Evidence suggests that

it may be possible• A PRESENCCIA goal

• Synchronous tapping• Illusion of being at the

camera position• Seeing own body as displaced

and an empty shell• H. Ehrsson (2007) The Experimental Induction of Out-

of-Body Experiences, Science, 24/8/2007

Recent Experiment• Two conditions:

• Real-time global illumination (virtual light field)• OpenGL (standard graphics) shading

• 10 participants each condition• Both had shadows and reflections.• Interested in ‘presence’ differences between the two• If none then it is shadows and reflections that are

important, not illumination realism • (based on earlier results).

• Preliminary analysis suggests that there is no difference in response between the two conditions.

Scenario

Influence of the Virtual Body

• Experiment also used to examine people’s responses to their shadow body

• Left virtual arm mirrors right (tracked) arm• Books fall and one left on the head• Do people turn to look at an event behind

them in the mirror?• How do people rate their mirror body?

Latest Experiment

Responses to Mirror

• 9 out of 20 subjects sometimes matched their real left arm with the mirror one

• 7/20 touched their head when they saw the book on it in the mirror

• 50% exhibited fear when the boy virtual character suddenly appears

• ~100% looked behind them when they first saw it in the mirror

• Q: “Even though the person in the mirror did not look like me I sometimes had the feeling as if I were seeing myself”, the median score was 82% and interquartile range 30%, where 100% meant ‘totally agree’.

Discussion - Plausibility

• Virtual hand illusion does not work…• When real arm and virtual one are not aligned• When the visual and real tapping are not seen/felt at the

same place on the hand• If the virtual ball sinks in or doesn’t quite touch the

virtual hand• The experience is broken by avatars that do not

move or respond to you appropriately• All of these suggest that plausibility is an

important issue

Discussion

• Plausibility does not require physical realism• E.g. Pinocchio illusion• Shrinking waist illusion

• Brain seems to be very liberal inventing interpretations

• Seems to respect logic but not physics

Discussion – Multisensory Correlation

• Evidence suggests that plausible multisensory correlations can induce these illusions

• Projecting ourselves into a virtual body is probably possible

• Also evidence that this works generally with π and ψ

Discussion – Two Concepts

• There are two orthogonal concepts useful for understanding people’s responses• Place Illusion – the illusion of being in the

virtual place • Plausibility – the subjective probability that

events are what they appear to be• π is to do with how we perceive• ψ is to do with what we perceive• The confusion of ‘presence’

Discussion

• π - the world can be static, but the participant is perceiving through movement

• ψ - the world may be dynamic • Is it consistent and/or congruent with

expectations, logically possible (if not physically possible)?

• Does it respond to the participant’s actions?• (π,ψ) ⇒ RAVE

Discussion – Beyond Reality• SCs provide a (known)

map from action to perception.

• SCs can be fundamentally changed with respect to virtual environments

• E.g. Brain-Computer Interfaces

• Neurofeedback• Upside down effects etc

• Each type of SC set can result in a different type of feeling of our relationship with the environment.

People on Body Representation

Daniel Perez Marcos

Bernhard Spanlang

Mavi Sanchez Vives

Olaf Blanke

Vanessa F. Descalzo

Henrik Ehrsson

The Virtual Light Field Team

Pankaj Khanna Jesper Mortensen Insu Yu

Department of Computer Science, UCLwww.cs.ucl.ac.uk/research/vr/Projects/VLF/Media

EVENT Lab, BarcelonaThanks also to• Anna Bellido• Ausias Pomes • Christoph Groenegress• Elias Giannopoulos • Jason Kastanis• Joan Llobera (Starlab) • Nuria Pelechano (UPC Prof.)• Raphael Carbonell• Verónica Orvalho (Universidad de Porto)