mixed reality_mfa thesis

Electronic Integrated Arts (EIA) MFA thesis, School of Art and Design, NYSCC at Alfred University, Alfred, NY, 2010

Moyi Zhang Mixed Reality

The simulacrum is never what hides the truth-it is truth that hides the fact that there is none. The simulacrum is ture.

——Ecclesiastes

Contents

Foreword

Introduction

Chapter 1The way to Augmented Reality

About my research

Chapter 2Being and surroundings

“My Baby”

Chapter 3Perception in communication

“My best friend”

Chapter 4None-bodyment and bodily form

“The New Diamond Sutra”

Chapter 5Subjectivity and Selfhood

“Me and I”

List of pieces included

Bibliography

Technical Experience

Foreword

Every invention or creation is crystalline in nature, having many sides. Although they look opaque they

are nothing other than the reflections of something else. They are mediums that enable us to see past the il-

lusionary state of existence, and help us to understand reality on a deeper layer or higher levels that leads to

a place out of which our memories and thoughts evolve. Every creation is like a telescope. It focuses on truths

that are out there, and compel us to ask where and what are they or what they are. Through this process we

begin to examine some of the more historical questions like where are we in relationship to the space that lies

beyond the familiar boundaries of our world and what or where is our true place in the universe at large. These

creations force us to think constantly about their reality, even though we don’t know what they are or what it is

exactly. We have to vigorously attempt to seize their meaning. Perhaps it is us who are ultimately seized within

the grasp of their meaning. Although we may explain more through the virtue of our technology in this digital

age, these answers only raises more questions, making us even more aware of old adage: “That the more we

learn, the less we know”. The “horizon of not knowing” moves, but is not removed.

Conversely, “What has been is what will be, and what has been done is what will be –done; there

is nothing new under the sun” Eccl. 1:9 RSV. It could means although people claim to create new inventions

they are not creating new. Nothing is original or unique. All being that exists has already existed. They are just

repetitions that exist in a different context with a different form. People can say: “This is something new” in the

present at least. The sense of present means ‘where I am’ and now means ‘when I am’. Thus, the common expe-

rience is that the present is inextricably linked to oneself. The notion of the present is when we feel the body’s

existence in a certain time and space. This gives raise to the creation of the sense of selfhood, other selves, or a

universal self. Around ourselves, we create reality. So reality is actually something we could perceive within the

domain of our consciousness and it extends to the range of wherever our consciousness is able to reach. Imag-

ine we are sitting in the center of our own reality, and then beyond this is a bigger range that is virtuality.

Much of the literature on virtuality contains a false assumption, that reality is the actual and virtuality

is the illusive in opposition to reality. The Oxford English Dictionary lists one of the definitions for virtuality as

“essential nature of being, apart from external form or embodiment. So reality could be the representation as a

form of an actual entity. It is an actualization to our perceptions, but it doesn’t mean “real”. The word virtuality

derives from the Latin virtus, which means strength and virtue. It refers to the strength of a non-physical thing

in light of the fact that such things can affect people deeply. Virtuality is invisible; concealed by reality, but it

doesn’t mean it is “unreal”. Although it has a great impact on us that may not stay within our awareness. There

is another false assumption that virtual space could only exist in cyberspace and virtuality is a new term specific

for the digital age. However, virtuality should exist in everything everywhere anytime. Virtuality doesn’t present

itself, we have to have a deep sense of what are we are experiencing to encounter virtuality.

Artists are very sensitive question beings, especially in this particular age when people’s lives are dra-

matically changed by digital media. Consequently artists should embrace and undertake the various challenges

presented by virtuality.

Introduction

My investigation is the unpacking of virtuality as a dimension of our experience.

I am looking at virtuality as it is embedded deeply within perception, memory, identity,

and communication, in contemporary life. This dimension of experience is intensified

during the information age. Therefore my investigation engages experiences of desire,

doubt and confusion about what is “real”. In my works, series narrative fictions were cre-

ated. They are allegories to represent or convey meanings beyond the literal. They reveal

the internal relationships and unsolvable contradiction between our understanding of

virtual and reality. The digital figures in the fictions create a mixed reality. They observe

and experience “reality” through their own perspective. They are representations of

un-present selves. They are put into the physical world by augmented reality technology.

The AR figures’ present in “reality” rely s on a maker’s presence in the scenes. They have

to be seen through the computer version, the extension of our physical visions. Their

digital simulated bodies are non-physical but do exist.

Thesis Committee: Peer Bode (chair), Xiaowen Chen, Andrew Deutsch, Barbara Lattanzi

Chapter 1

The way to Augmented Reality, About my research

The first year of my research,

the deconstruction and reconstruction of time and space

My first year of making art in Alfred follows two separate but related themes. The first is an investiga-

tion of how our understanding of reality is affected by our perception, memory and altered mental states such

as trauma, panic, anger, paranoia and depression. The second examines our understanding of reality and its

relation to time and space. I did many works researching issues along these two lines, with the approach of

deconstruction and reconstruction I can demonstrate how existing stereotypes may be overturned.

Two of the pieces being made at the first semester using Finalcut Pro and After Effect involve the

recordings from the university observatory and a playground. “Time Game” is a site specific performance video.

It plays with the idea of time overlapping and the notion that time exists simultaneously. By questioning the

illusion that time is linear, and creating a space in which time and movement are at play together, the viewer

observes not only the passing of time but time in reverse, time stuttering and time looping back on itself. Ob-

serving the layers of time within a static environment reveals the complexity of times multi-dimensional nature.

My performance and movement through the site visualizes time moving in multiple directions and in multiple

ways. The performer literally carries time with her as she performs and moves through the observatory space.

An observatory is a location used for observing celestial events. It is home to telescopes. Because light takes

time to travel, we see objects not as they are now but they were at the time when they release the light that

has traveled across the universe to us. So we can look farther back through time through telescopes. They are

“time machines” to help us to travel million light-years away. So our perceptions of time and space may not as

we normal perceive.

“Circle” is another site specific performance recorded at a playground. “Circle” piece is a narrative

that, just like a circle, is without a beginning or end. The ambiguous event between the two figures is happen-

ing over time and space while the camera is going around itself. Instead of following the people, the camera

is scanning the surroundings, spinning by the scene. The landscape, which is almost static, becomes an empty

space, because there are no other events to catch the viewer’s eyes. The viewer could only see the event when

the camera meets the figures at a certain point. All these points are being constructed with our imagination,

generating narrative structures. This video work is trying to explore the relationship between the changing

image and the looping of time. The work poses the question about our understanding of time in us, time in

machine and time in nature.

The piece “The doll house” is a video installation also plays the idea of time and space. The frame of

the video disappears. It is the structure being built holds the space and time. The projected imagery on the back

of the house is the digital simulated space while the house itself is the physical simulated one. It was multiples

of myself walking and carrying things from inside to outside of the house through doors which are actually the

projected image in each of the sections. The doors they entered open up an imaginative space which are out of

people’s visions. The house has an intricate structure combines digital, physical and imaginative space in one.

All the shots were collected in Harder Hall in Alfred, NY. This building is the place where my studio located in

these two years. The structure of Hard Hall is like a labyrinth also related to the idea of this “doll house”. There is

always a way in and a way out. But the uncertainty of the direction and our location are always existing.

These three pieces may response to a question asked by Susan Blackmore in the book “Consciousness:

A Very Short Introduction”. “Can states of consciousness be mapped? Moving from one state to another can feel

like moving in a vast multidimensional space.” In filmmaking, there are two problems people have to deal with.

How to create a sense of time as long as hundred years or five minutes within a certain timeline? How to create

a sense of space as big as a universe or one piece of paper within one frame? My question is how do we connect

all these elements of time and space together to make up a whole story in our brain? How do we read them and

how do we experience them? Gilles Deleusze used the term crystalline narration to describe time and space in

montage. “We no longer have an indirect image of time which derives from movement, but a direct time-image

from which movement derives. We no longer have a chronological time which can be overturned by movement

which are contingently abnormal; we have a chronic non-chronological which produces movement necessarily

‘abnormal’, essentially ‘false’. ” It proves that our sense of reality could be altered by reconstruction and decon-

struction of time and space in relationship to our bodies, perceptions and memories.

The pieces being made after them involve a lot of images of water and fire. The way of my working

became more intuitive and impulsive without much interference from my analytical mind. Part of the reasons is

water and fire happened a lot in my dreams during that period of time. Such as the sea of fire devoured the all

Alfred. Or the city of my hometown was in flood. For me, they are not only symbols of disasters, but also means

renascence. It contains people’s desire, rage, hope and desperation in reality. In the water piece, the girl is al-

ways surrounded by water and she is somehow trapped into this imaginative psychological space which possibly

is her own reality she is trying to get out from. In the fire piece, all the footages I got were shot in Alfred. The

video starts from normal shots and then starts to shake, smoke and burn. The houses, bedroom, streets, cars

including people were all set on fire. However, it is not real fire, but digital simulated fire and smoke created by a

software called Motion. So that things are still keep moving or being normal when they are burning. The sense of

humor in it gives it another layer of the meaning of this “reality”. People might doubt if it is real or non-real. Are

we confronting a true disaster or an imaginative situation? Should we being frightened or think it as a joke?

Bill viola is video artist also interested in water and fire. He said, “When you make a video of fire and of

water, the resonance is incredible…a flowing stream interpreted by flowing electrons. I was drawn to water in-

tuitively, with a camera in my hand… drawn to its pulsing waves and undulating surfaces, and of course, optically

to the images it reflected.” In most of his works, he explores the spiritual and perceptual side of human experi-

ence, focusing on themes of birth and death, the unfolding of consciousness. His work has a strong connection

to tradition of religious iconography in metaphysical imagery. So where he looks at painting of scene, paintings of

religious people and how they represented and becomes part of what he is tapping into.

BillViola-“Acceptance“2008,<http://images.artnet.com/artwork_images_633_407256_bill-viola>.

Slavoj Zizek is a Slovenian continental philosopher and critical theorist. His book “Enjoy Your Symptom!

Jacques Lacan in Hollywood and Out” talks about the films of Alfred Hitchcock. His works are very much about

emotion, fantasy, desire, what can be said, what can’t be said. They are very psychological in many ways. If you

look at all of Hitchcock’s work, you see there are certain motives, certain kind of images, gestures that happens

across all of the films over and over again. Such as people looking at people, the staircases or falling down from

staircases…in a sense, they are almost like dream imageries. The point is Hitchcock came up with stories that

would allow him to represent the images. The images were the most important thing, the things of what that is,

the thing about how do they look each other, the thing of staircases, that was really the imagery he was abscised

by and he came up with narratives that allows him to use those imageries. So in a funny way there is a similar

process to what I was talking about. To find a form for these obsessions and dreams images to live in.

“Meshes of the Afternoon” (1943) is short film directed by Maya Deren and her husband. The narra-

tion of this film is circular. It repeats a number of psychologically symbolic images including a flower on a long

driveway, a key falling, a door unlocked, a knife in a loaf of bread, a mysterious figure with a mirror face... It is a

surrealist film that depicts a world more and more difficult to catch reality.

All of the themes that are in my works could carry out in painting or in any medium. However, under-

standing of the potential of new medium language and working with new mediums of technology could bring

me more experience. New media has affected the way of our thinking and the way of our behaviors in many

ways. It is the media that created us and made our lives became different. It even formed the structure of our

society. To me, it is something more related to our contemporary lives. Using new medium could be the best

for me to deal with contemporary issues and problems. Marshall Mcluhan who is a Canadian philosopher and

scholar, know for the expressions of “the medium is the message” and “global village”. He defined media as “the

extensions of man”. Mcluhan proves that how media extends our perception and extends our awareness of real-

ity. He wrote, “We shape our tools and afterwards our tools shape us.”

The other writer Siegfried Zielinski helped me a lot with understanding of media development. In the

book “Deep Time of the Media, Toward an Archaeology of Hearing and Seeing by Technical Means”, he argues

that the history of the media does not proceed predictably from primitive tools to complex machinery. It is not

liner but more like a genealogy. There are turning points of media history or many dead ends where things stop.

Dynamic moments of intense activity in media design and construction that have been largely ignored–the hid-

den layers of media development.

In my research, I also tried to study the processes and techniques of analog and digital image pro-

cessing and to explore the possibilities of different ways of working. Such as manipulating video with a Frame

Buffer in conjunction with Max/MSP/Jitter; screen capturing the realtime video streams run form the program;

scanning the electronic signals and compositing the scanned images as animations with After Effect. Though

working with the input and output of digital and analog signals, I found a way that provides an alterative view

of reality. It helps me to explore the potential of creating visual images that helps me to understand reality and

memory that is beyond or even more real than the way it is normally perceived. The recordings I did with The

Dave Jones Digital Video Frame Buffer have a very strong landscape reference and the resulting sound defined

by them. In fact, they are not images of landscape but abstracted pixelization that it is the digital process and

the iconic aspect creating the illusion of landscape. They are something we feel we are familiar with but really

not. They are imaginative and illusive imagery of something real. People might have references to the things

they already know. They might think what they were and what did they become.

Google Earth, a space opened up in time

Myhomesfrom1983-2010

Because of the continuous progress of media and the development of transportation, the world is

becoming smaller. People can more easily and freely travel between different cultural context and systems of

thought, and swim between various languages and media. However, the confusions about time and space and

the difficulty of locating oneself in the world have lead to the decreasing sense of belonging.

When Google Earth (GE) came into my sight, some thoughts in my heart start vibrating strongly. GE is a

program displays superimposed images provided by satellite, allowing users to visually see the earth resting in

space and looking closely to the cities, houses at different angles. Through this access, users are able to gather

mapping and geographic information of the address typed into the program. It utilized 3D triangulation photog-

raphy and CAD (Computer-aided design) to created a computer-simulated environment which is a mirror world,

a hybrid space, a representation of the earth we are living in.

AlfredUniversityinGoogleEarth

GE makes a person feel omnipotent in the world and able to navigate the

world at will. The user could unlimited access to this space, a space opened up in

time. Anywhere in the world could be arrived at. In a way, the whole earth is pos-

sessed by individuals. Each of my previous homes could be easily found which would

take me back to that specific period. And more unknown world could be learned

beyond my knowledge and life scope. Although it is a simulation or image of the

physical earth, it has its own order of time and space. In general, our behaviors are

formed according to the environmental conditions. When people stand on the actual

surface of the earth, we determine if a place is further or closer from the point where

we are standing, based on the linear distance measured from point to point, then

the time consumed would be calculated. Time itself does not exist, but series logical

connections generated by the human brain, based on space and sequence events

happening in spaces. In GE, the viewers do not have to consider how much time was

consumed when passing through distances, and they are also not affected by the

time lag. People are standing in a space outside the earth, in which there is no need

to measure the distance between the starting point and end point, there is no up and

down, there is no order either. All the information about time and space have been

written into this massive image database, in which all the information stays in parallel

position. The viewers could travel to anywhere in time and space by reading a certain

information according to their own willingness. Human consciousness is holistic,

fluid, and none-linear. The meaning of GE is far beyond a map, the way of seeing was

completely changed. In this simulation of reality, the body is liberated, which provide

people a kind of experience beyond time and space, beyond body and matter. The

human being seems forgotten, the consciousness seems universal, and it could be

infinitely small or infinitely big, from known to unknown, shuttled freely.

The first piece I did with GE “Mind Traveling” was a 10h 20mins video. The whole recording was the

virtual vision of my eyes traveling from Beijing and going all the way back to where I started. The person who

is actually seeing it becomes the one who is flying there. The geographical environment during the journey

is gradually transforming through time. Also, we could see the division between really detailed imagery of

the earth and the imagery roughly represented by 3D graphic based on the population of the region. Both of

them are photographed in different time and composed together. They are relatively real or not real in com-

parison with the actual earth.

Google Sketchup, Montage of reality

Google Earth is a huge image database. The aim of creating this database is to simulate reality

as close as possible rather than creating reality. In GE viewers only have the ability to read without the

opportunity to change the database. Compared with reality, GE is just an accessory, a tool for reality. GE

functions as a “searching” engine, but not as something for creating. Then Google SketchUp appeared.

SketchUp is a 3D modeling software designed for architects, game developers and related professions.

It includes features to facilitate the placement of models in Google Earth. So that people can create their

own models and then upload them to Google earth. Surprisingly, the 3D models can be accurately placed in

any position of the digital globe, that is, GE has become an “space of ideas”, and is completely open. Cities,

villages, mountains and ocean. . . These public spaces have become available as autonomous private space.

Through people’s participation, GE, based on reality, has become one part of our reality. There are no walls

and obstacles in this space which exist in reality. Then one question appeared: If you can own the entire

planet, what would you do with it?

The other feature of SketchUp is the 3D Warehouse that lets users search for models made by others

or contribute their own models. I collect 3D models from Google 3D models Warehouse. Then I manipulated

them and put different models together to create new scenes in Autodesk Maya. They are reconstructed reali-

ties usually contain pieces of land or other kind of spaces isolated from the rest of world. This body of works

includes digital prints and animations based on the scenes I created in Maya. Print is another display technol-

ogy. I chose Design Winder and Giclee drum printing to print my prints. They have a particular quality because

of the way of the inks mixed and how the ink soak into paper is quite different than the Epson printers which

have a more commercial contemporary style. So that in the same way, there is a differences between tradition-

ally be on rice paper or other kind of paper. In tradition art making, we identify the significant differences.

3DmodelinginGoogleSketchUp

Augmented Reality in ARSights and Linceo VR,an extension of the human senses

TheblackandwhitemakerinARSights

As I continue to become familiar with the program Sketchup, one of the plug-ins of the program

ARSights became more important. ARSights relies on the technology of Augmented Reality (AR). Augmented

Reality is a term for a live direct or indirect view of a physical environment which augmented by virtual com-

puter-generated imagery. Its research explores the application of computer-generated imagery in live-video

streams as a way to expand the physical world. As a result, the technology functions by enhancing one’s

current perception of reality. For example, people can see the 3D model in hand just like watching a physical

object. It’s like a real time feedback loop seeing myself holding something that in fact I’m not holding. But as

an image, I’m holding it.

Augmented Reality in ARSights

Virtual reality (VR) is a term we are more familiar with. It applies to computer-simulated environ-

ments that can simulate places in the physical world as well as in imaginary world. Users can interact with a vir-

tual environment or a virtual artifact either through the use of standard input devices such as a keyboard and

mouse, or through multimodal devices such as a wired glove. It has been commonly used in pilot training and

VR games. Virtual reality technology always associated with terms such as simulation, interaction, artificiality,

immersion, telepresence, full-body immersion, and network communication.

“As is the case for virtual reality (VR), several formal definitions and classifications for augmented real-

ity exist. Some define AR as a special case of VR; others argue that AR is more general concept and see VR as

a special case of AR…The fact is that in contrast to traditional VR, in AR the real environment is not completely

suppressed; instead it plays a dominant role. Rather than immersing a person into a completely synthetic

world, AR attempts to embed synthetic supplements into the real environment (or into a live video of the

physical environment).”

The commonly know examples of AR is, in a television broadcasting of swimming events, there is

always a virtual line which indicates the position of the current leading swimmer. The AR system can be used in

many video games too. It simulates the viewpoint of the user and give visual directions to a location, mark the

distance, give information about equipment, and display other images based on whatever the game designers

intend. In some current applications such as the airplanes, there is usually a head-up display integrated into

the windshield.

The main hardware components for augmented reality are: display, tracking system, input devices and

computer.

The three major display technologies for AR are: Head Mounted Displays, Handheld Displays and

Spatial Displays. Head Mounted Display (HMD) are usually optical see-through or video see-through in nature.

Optical see-through allows the image of physical environment to pass through the lens and the registered

virtual graphical objects or information to be projected into user’s eyes. This display technology is being used

in medical examinations and car repair. For example, when you wear a Head Mounted Display, instruction

texts will show up according to what you see in the imagery from the glasses. In this case, it is about additional

information that is present while you are looking at. Questions about the text are answered by that additional

augmented information. It is a direct instruction according to your vision. You don’t have to find the informa-

tion but it’s already in front of your visions.

Handheld Displays employs a hand held display installed on a small computer device such as an

iphone, a camera cell phone product made by Apple Company. It applies video see-through technique to over-

lay the virtual graphic to the physical environment in the live video steams. Instead of wearing or carrying the

display, Spatial Augmented Reality (SAR) projects the image of graphical object or information onto physical

objects. A projector or multiple projectors can be used to expand the display area on any surfaces or indoor

walls. The viewers are able to touch the object to interact with the projected imageries.

The tracking technologies include: digital cameras, optical sensors, accelerometers, GPS, RFID (Radio-

frequency identification), wireless sensors. Each of these tracking technologies has different levels of accuracy.

They applied to track the viewer’s head position or hand(s)/hand input device.

For consistent merging 3D images and the image of the physical world in real time, the software for

Augmented Reality has to process some part of the camera images information into coordinate system data.

So that the virtual objects or information can be merged into the image from the camera according to the 3D

geometry data.

The software I used in my works is Linceo VR. The techniques I am practicing in Linceo VR includes:

single camera position/orientation tracking

tracking code using one or multiple black square patterns and image realtime raytracing

realtime rendering

realtime animations

realtime contextualization of the digital models in any surrounding environment

publish online realtime AR applications using user-interface

generate QuickTime 360° views of one’s model in varies scenes

By using many techniques above, I could create a more realistic 3D model to its surroundings and

adjust the lighting and texture in real time. The 3D models could even project their own shadows on the physi-

cal surface they are standing at. The realtime features could also allow the viewers’ body to actually interact

with my models such as wearing a T-shirt with a black and white square maker printed on them. There are

many solutions I make to build very site-specific makers for different circumstance in order to track the maker’s

orientation and position better. I also could have Augmented &Virtual Reality applications published online so

that I can have the application embedded in web and design my own interface for it in HTML/CSS or Flash.

By screen capturing the realtime video streams run form the program, I collect all the footage I need

and then bring them to Finalcut Pro to edit and process them with After Effect or Max/MSP/Jitter.

The different pieces I made using the AR system are not really as a type of an image. But as an episte-

mology, which is the name for the branch of philosophy that is about the nature and limitation of knowledge.

It is about a kind of knowledge, information and it’s quiet unique. The months I spent on my AR research of

learning the techniques include creating models, textures, reflections and the suitable lighting. They are similar

problem while I was dealing with painting. How do I visually construct the imagery using the software to create

a more combined vision experience? In the future I am also going to apply animations on the 3D models. So

it’s not only about the repositioning but they could also move. It’s going to be the next step in terms of the

research. On the level of learning the software, learning the language of those processes.

This technology allowed me to think one step further. I found a simple and intuitive ways of breaking

down the digital space and physical space limits. Like physical objects which interact with the surroundings,

digital objects can also interact with people’s movements and viewing points. AR technology is not a simple

magic to deceive people’s perception, but a new sensory system which is an extension of the human senses.

Since we already know, physical space and digital space are not in binary opposition, but act as one as a unit,

it has become an irreversible trend for human senses to be digitalized. AR technology should be used to an en-

hance and a strengthen reality. The definition of real has to be enlarged. If we continue to believe that digital

simulations are untrue to our perception, then what we once believed to be true of any experience might also

become simulations. Our life would be the biggest virtual reality; virtual reality would be our biggest life. But in

fact, they are not separate.

The first piece I made with AR system is “My Baby”. I had a very limited control on the baby model. I just

deal with the techniques I had for that time. Limitations didn’t stop the idea but became part of the idea. Just

like the analogue signals, they are not perfect imageries but giving people more space to think about it. I don’t

mind those limitations but I don’t mind to go further either. That’s again talking about the process in the re-

search that does have to do with the tools and technology and using them along the way as I become more and

more proficient.

My concern is that virtuality is deeply embedded in our perception, memory, identity, communica-

tion, culture and symbols in our contemporary life. It includes the human pursuit of truth, doubt and confusion

with the impact of digital media technology. Information culture dramatically changes people’s lives. I hope I

can maintain a sensitive and witty mind to avoid the rashly following of any cultural phenomenon, to find some

revealing places, something can awake people from immersing in a narcissistic state with clarification and under-

standing of media literacy.

With the development of media technology, people are so free to use any kind of form of media. People

usually maintain the impact of media on an unconscious level, like a fish unaware of the presence of water. But

there are many things we can not ignore, the increasing desire of control, the enriching form and scope with

people’s relationships, the storage of human memory became smaller and smaller, people become more and

more procedural way of thinking like a machine, emotional sustenance in the use of the media. . . Although me-

dia technology can change our way of life, to help us better understand ourselves, to help us better understand

the world, it has brought more new problems. Media literacy has to be used to help us critically analyze mes-

sages, broaden our experience of media and develop creative skills in making our own media messages.

Chapter 2

Being and surroundings, “My Baby”

About the story

A spirit from a virtual world was born in the form of a baby and then came into reality. He observes

people’s life from his inexperienced perspective. His intelligence is like a human so he can understand human

life. But his fleshless body keeps him from truly becoming involved with his surroundings. His confusions about

the world seems to be the same as the confusions of every baby facing life experience.

To immerse into a space, to experience a reality beyond our daily lives…It is always a question for me

to think how far virtual reality could go. From painting, theater, cinema to a computer-simulated environment

includes additional sensory information. Virtual reality is more and more close to real to our perception. It con-

nects the viewers and the environments with a spatial relationship which is something we created in our mind.

But there is an interface actually sitting in between. It is a boundary we could hardly go across.

People’s desires and ambitions to experience an unusual life are endless. But it doesn’t mean our daily

lives are going to be abandoned. It’s hard to escape or runaround from what we have in our lives. Something

we have to face each day are not easy to understand or easy to gain.

However, what about letting a virtual body enter the physical environment? Let a virtual body experi-

ence our lives with his/her perspective. So we could stand at his/her point of view, forgetting our own bodies,

forgetting the way of our living, forgetting our knowledge, everything start from a zero point. What would that

feeling be? I believed a newborn baby would be the best character to choose.

The baby has to learn everything and start to think everything from the very beginning in this brand-

new world just like a real baby. However, he has an appearance of human being but no sense organs. What he

has to face will be more challenging, he cannot breathe as a human being, he can barely move but only float

in the air. He cannot speak, but keep thinking vigorously, which based upon a situation of not. He wants to

communicate with others, but nobody could know his existence. How could he become truly involved into here

with such a virtual body? How could he survive in here?

This piece also represents my desire to be involve in different surroundings, from China to America, from

one language context to the other, from one life style to a completely different one. It also represents the strug-

gling that I have to communicate and understand a new environment. Standing from a distance to see one culture

through the other culture I came from. The strangeness of it is very similar to this virtual baby. Through the baby’s

eyes, a different perspective was provided so that the viewers could keep a critical distance to review our lives

again.

At the end of this piece, the baby said: “I wonder if people feel as I do about being alive?” He doubt peo-

ple have been through the same situation as him. They are living in this way and act as a human being because

they want to feel they actually exist in this reality just like somebody else. Even though this is such a “childish”

conclusion he indicates the question offensively to people’s values toward life and world views. Maybe everyone

is like an alien to our reality just like this digital baby. No matter where are we from, we still believe we belong

here because we are simply used to being here.

The digital baby also has another layer of meaning which is a seed planted into our reality. He looks

weak but has strong life force. He breaks through the defined boundary between digital environment and physical

environment. The interface in virtual reality disappeared. Or the boundary never existed at all in no matter what

kind of context. Dualism does not work in my art. It became a parochial way to classify the world just like how we

consider ourselves belongs to this reality but not virtuality. Digital technology gives us a chance to see the other

side of our physical world, to rethink our physical world, time, presence and the act of creation. The virtual and

the material are parallel, intercrossed and revealing to each other.

Title: My baby

Date: 2009.10

Duration: 07:10

Display: Sony Vravia 40’’ LCD TV

Subtitle

I don’t know why I’m here

But still I’m here

I believe I’m here

I have a shape of the body, but no skin

I have no bones, but I can stand

I have no eyes, but I can see

I cannot speak, but I know how to express myself

I want to feel alive

Just like everybody else

I want to

Sleep in a bed

Eat at a table

Go to the bathroom

Read a lot

Watch TV

Learn how to drive

Make friends

Find a job

However

I don’t know what tiredness feels like

I won’t be hungry

I even don’t know what cold feels like

People can’t see me

It’s like I’m the air

I don’t know how they communicate

I don’t know if what I am doing is meaningful

I think I am a joke

I feel like a fish out of water

I don’t know what people are doing

Why are they doing so?

Life is not easy here

It is full of blindness

I wonder if people feel as I do about being alive?

Chapter 3

Perception in communication, “My best friend”

About the story

A girl was living with a dog beside the lake. One day, she found a little man in a red box. He is very

special. He has no weight, the mirror cannot see him, he could float in the air, he could walk on water, and he

could stand on the tree dancing with the wind. The girl did a portrait drawing in order to remember him, even

her dog came to see him. The dialog happened between the two drove by their questions and curiosity to each

other. However, the girl realized she couldn’t physically touch him while she decides to get closer to him. She

can not stand with this situation anymore and cast away the red box where the little man came from. He lay in

the grass and gradually disappeared with only his outline of body left.

The “friend piece” was a two channels projection in a corner of a space. The images are 90 degrees

from each other. The girl and the little man always exist in different frames. They can’t get across the gap

between them. but their conversation from their internal make them linked together. The simple relationship

between these two characters indicate an unsolvable contradiction. They are closed but can not touch each

other; they know each other but can not truly communicate; they stay at parallel spaces but have no intersec-

tion of time.

The friend, if considered as the girl’s imagination, this girl and the world she lives in could be a

dream of this little man. He passed by this physical world, passed by this girl’s heart. However, he couldn’t

break through the gap of their communication and totally involved in her life where he did not exist. Although

the girl finally decided to break the illusion of her imagination, there is even bigger emptiness left in her “re-

ality”.

This piece also discusses the question of communication. But it is not like daily communication. First-

ly, these two figures are not living in one time line, their dialogs are not happening in real time. The dialogs

happened because of the editing, like a time montage, the viewers match the images and language in their

brains. For the space, the situation of the two are quiet different too. The lake, where the girl lives is more

like a closed mental space. The boy breaks into this space and exists in this space in virtue of a medium-a box.

But his body could not truly involve into his surroundings. His existence is temporary, unsteady and passive;

he couldn’t interact or make connections to the environment.

Title: My best friend

Date: 2009.10

Duration: 06:10

Display: 2 HD projectors

Design Lab Synchronizer system, Dave Jones synchronized DVD players

Who are you?

Where are you from?

Do you like here?

Why don’t you come up?

Why are you watching at me on the shore?

Why don’t you fly?

You are too heavy.

I think you are different from me.

How do I look in your eyes?

I want to get closer to you.

I’m going on another trip.

Subtitle

This is where I live.

One day, I found a box.

Who are you?

Where are you from?

Do you like here?

Don’t you be afraid of falling down?

Don’t you be afraid of getting wet?

Don’t you be afraid of cold?

Let’s play a game.

I like the way of you look.

I want to remember your face

What does it feel like to be touched?

I couldn’t live in your illusion.

I set up the context of the story to establish the existence of communication as abstracted from daily

life, such as BBS (Bulletin Board System) in the message, the video game or various forms of social network such

as the dialogue in Facebook, Kaixin, skype, the time and space in them have very similar characteristics as this

piece. The first is unreal-time, the second is people and their environment can not make a physical link.

For example, in the video game, we communicate with a number of digital simulation of the figures,

we play games or are even involved in a war in which we can choose our own appearances and roles, choose

a place and time to start the game. We exist in the digital environment by using a network identity, as the little

man breaks into this physical space. In the digital space, our lives are very temporary and unstable, you can easily

“die”, or even self-delete. While we have our internal time, we also established another set of time system in

video games, which is relatively continuous by their own, but for us, it is just a timeline being patched up from

our time clips, this time is completely controlled by us. We can easily pause, continue, slow down or skip. In this

work, I placed the state of such communication in the physical space, so that the viewers as the subject of the

physical space could feel the experience of digital body in our lives, to feel their birth and death. Likewise, they

have their own set of time, our time is just a fragment of their time, our space is a place they passed by, thus to

question our so-called reality, our existence and perception and emotion. Our existence in “reality” is a virtual

existence; our brain is the largest virtual machine; our ways of communication is a virtual communication.

Cao Fei is one of the most important Chinese contemporary artists in her generation. She describes one

of her pieces “i. mirror”, a three-part 28 minutes Second Life-machinima, as a “virtual documentary” of her life in

alternate reality. She was documenting those digital figures like they were real persons. Not only records stories

from her Second life, “i. Mirror” is also a search for an individual existence and spiritual sustenance somewhere

between reality and virtual life. “I knew that this world sometimes mirrored the real world. We are missing a lot

of things in real life, the same as in Second Life...Second Life makes me figure out how to face real life.” Cao Fei

thinks Second Life as an extension of the self. In it she has created another self, and through this virtual self she

observes and understands her own depth and complexity while, at the same time, broadening its scope.

CaoFei-“i.Mirror”2007<http://cn.creativecommons.org/wp-content/uploads/2010/01/iMirror.jpg>

Chapter 4

None-bodyment and bodily form, “The New Diamond Sutra”

About the story

The little Buddha enter the city to beg for food and then came back to do mediation everyday. He takes

the subway and bus shuttle to travel around the city. He observes contemporary society, helps to liberate the

suffering of people from their contemporary life. He tried to use language to communicate with people around

him, to promote Buddhism, to save the living souls from the sea of misery. But in the busy crowd, no one knew

of his existence, no one can stop to listen and accept his teachings. The Eastern thought he represents is alien to

the Western context full of ideas and symbols of contemporary culture. He exists in reality, but he stands in an

opposite place to deny the reality he observed from everything. Eventually, he is blown down by a wind caused

by a whizzing past train.

The Buddha is a rich semantic symbol contains historical facts and religious illusion. It has a dual mean-

ing of real and simulation. He was a distinguished man and worship idol. People could not achieve his spiritual

level. However, in “The New Diamond Sutra”, the Buddha appeared in our physical world, a 21st century mod-

ern city, where he lives as an ordinary person with a humble identity to experience the modern way of life. His

body is a digital simulation, the development of electronic technology enables him to be born again among us,

and his soul is activated from then.

The core discourse of the Buddha’s speech and the inherent meaning in the form of the Buddha are

the same, People believe he is unreal because his body is non-physical, people can not see him. However, he

is using the “unreal” body which created by this reality to deny the unreal reality as he acclaimed. In the same

time, he is also using a language in a contemporary form of expression and concepts to deny the language and

concepts formed by contemporary ideology. The biggest contradiction in the work is that the unknowable lan-

guage and unknowable thought can not be conveyed. So what do we need to express? Why using the language?

Why do we communicate?

The Diamond Sutra is the world’s earliest dated printed book in the year 868 AD. It is one of the most

important Buddhist text that has a long and profound history. Towards the end of the book, the Buddha calls it

“the diamond of transcendent wisdom”, because its teaching will cut like a diamond blade through illusions of

reality that surround them. In the sutra, there is a repetitive dialogue between the Buddha and his disciple Elder

Subhuti regarding the nature of perception. The Buddha is trying to help him to unlearn his preconceived, lim-

ited notions of reality and enlightenment and help him to transform the way he perceive the world. The Buddha

describes reality as “ A start at dawn, a bubble in a stream; A flash of lightning in a summer cloud; A flickering

lamp, a phantom, and a dream.”

My piece “ The New Diamond Sutra” is a very simplified version of the text I edited according to my re-

search to Diamond Sutra. Its idea didn’t go beyond the text but very strictly followed the original. The text itself

has nothing new in it. The new thing about it is the context. The words were spoken by the digital virtual Buddha

in such a contemporary surroundings. The new meanings came from the imagery but not the text itself. When an

idea, sprit, concept existing in different context, people understand them differently, they become different.

“TV-Buddha”isoneofNamJunePaik’smostfamousvideoinstallations.Ithasalivecameraonthe

Buddha’ssculptureandtheBuddhaislookingathisliveimage,sothereislivefeedbackloop.Itmeansthe

pastandpresentaregazinguponeachother.TheyencounterinthisparticularsituationPaikcreated.Itisan

encounterbetweenOrientaldeityandWesternmedia.TheBuddhaisinthestateofmeditationandself-reflec-

tionwhileheissittinginfrontofhisvirtualimage.Itisalsoaself-examinationofthetechnology,massmedia,

cultureandreality.

NamJunePaik“TVBuddha”<http://bridell.com/blog/wp-content/uploads/tv-buddha.jpg>

In my piece “The New Diamond Sutra”, the video is presented across 16 Apple computer displays.

The original video frame was vertically split into two halves right in the middle in order to play on two differ-

ent computers patched together. It is the live Jitter patch program plays the two halves in two computers. The

parent Jitter patch in one of the computers is sending the messages to the other child Jitter patch in the other

computer, what is the time code of the video it is playing right now in the video timeline, so that the other

computers can follow and make these two separate videos to be synchronized. It takes all of these displays for

the image to be legible. Still there are a lot of frame activities going on when all the displays’ color are slightly

different and they are out of sync for not more a second sometimes. The technology I used with jitter is case of

a different kind of synchronization but it’s not absolutely sync, there is a little bit drift as timing those monitors.

However, they are very hard to be observed. Part of it is the result of in the era of the technology and part of it

that is also what is that piece is. This piece is in some relationship to that form.

Title: The New Diamond Sutra

Date: 2009.10

Duration: 07:10

Display: 2 sets of the 8s (16 Apple Display)

Subtitle

I entered the city to beg for food

then came back

sitting with my legs crossed.

Men and women,

How to subdue your craving desires?

How to develop the most awakened mind?

Listen carefully

with your

full attention.

It is an illusion of the mortal mind.

Like a bubble floating in a stream.

So I say to you,

this is your conditioned existence in this fleeting world.

You shall be perfectly content in seclusion.

Wherever my words are honored, there are my most venerable disciples.

Chapter 5

Subjectivity and Selfhood, “Me and I”

In this video installation “me and I”, there are eight TV glass tube Sony Trinitron monitors horizontally

arranged from left to right, but the whole installation is a symmetrical mirror image relationship. From the left

first monitor to the third are images of interactions between girls and their digital simulation of themselves.

And the numbers from the right side of the first to third are the boys and their interactions with another digital

version of themselves too. The middle two monitors are places where the boys and girls encounter. Starting

from both ends of the array were boys or girls trying to express and communicate with people wearing a mask

of themselves; from the ends of the second monitor is the boy or the girl facing themselves or touch and kiss

themselves; from the two ends of the third monitor are the images of themselves continued pursuit or follow

themselves, while one of the middle two monitors is the girl and the boy stands closely to each other, but their

digital simulated body are facing back to each other. While the other image is the opposite way, the boy and the

girl are standing on both sides of the screen, eyes looking at the outside of the frame, they are far away from

each other, but their digital simulated body are looking at each other very closely.

This eight channels installation make a leap from other AR pieces. It has no speaking and no text. A

body performative gesture, a limited set of gestures and movements are being used instead of the language.

There are no imageries of different situations. The background is very clean so that the gestures can stand out.

The background which was the same background as in the monitors’ background, the image of the gallery

wall. The installation was set up as a narrative form, from his/her own behalf, to their separations of himself/

herself, into himself/herself pursuit their own, or self-confrontation, and finally to their relations with others.

Each screen, the characters, both physical and digital, are all the same as the phantom who disappeared into the

white walls in the background. It is a story without a beginning and an end, but it is a constant cycle of appear-

ance and disappearance relationships.

In this installation ideas are focused on the relationship with the self. The people behind their masks

are trying to be self-expressive, being covered by themselves and trying to escape from the shadow of selves.

The action of facing themselves “eye to eye” implies self-examination, meditation, and also narcissistic and

self- criticism; chasing self and being in self-pursuit. Desperately these figures want to capture the traces of their

own, follow their own shadow, this shadow is implied in the pursuit of their ideal selves and finding selves or

self-control, being freed hem from the themselves, and this digital self is actually determined by themselves.

What they pursuit is the shadow of them. They are always there. We can not escape from them and they can

not be conquered. They belong to them but we will never get hold of them. We are ambiguous in the relation-

ship with others. When our physical bodies are far away from each other back to back, or our minds are closed.

When the two physical bodies are facing each other, our self-images are separate. When we feel separate from

ourselves. What our choice would be? What we will comply with?

Janine Antoni is a contemporary artist who focuses on the interrelationships of her body and different

parts of it. For the piece she did called “ Lick and Lather”, she said: “I wanted to work with the tradition of self-

portraiture but also with the classical bust...I had the idea that I would make a replica of myself in chocolate and

in soap, and I would feed myself with my self, and wash myself with my self. Both the licking and the bathing are

quite gentle and loving acts, but what’s interesting is that I’m slowly erasing myself through the process. So for

me it’s about that conflict, that love/hate relationship we have with our physical appearance, and the problem I

have with looking in the mirror and thinking, ‘Is that who I am?’”

All of the AR pieces have a documentation quality to the image I collected. There is situation that I was

recording, that is a very complicated social, cultural, political situation. For myself, the AR system I used has such a

revolutionary function to the process of my video works.

The future

On the technical levels, the possibilities of AR technology seems endless to me. All the AR pieces in my

thesis show were presented by videos or video installations. We were still outside of the video frames. There were

no participations.

For the long term, as my research and techniques in AR technology are being more developed, I hope I

can let the viewers interact with my digital models or even modify the models by their own will. They can control

a digital-simulated control panel or affect the models by their body movement instead of pressing buttons on a

piece of physical equipment. After the AR display technology becomes more widely used and affordable, it is pos-

sible that my viewers can wear a head mounted display or hold a handhold display so that they can see my works

directly from their point of view but not seeing through a static interface.

For the short term, I chose 3D human figures as my AR figures in the thesis show. Because I wanted to

give them spirit, give them lives and let them speak for themselves. They were strangers to us but they can feel

the world as we do. Their viewpoint was provided to us so we can feel their feelings as well. In this way, a more

intimate connection was created between the AR figures and the viewers. It is part of the reason why I used nar-

ration in the videos to present the ideas. The AR pieces I am going to do next after the thesis show will involve AR

text, AR image or AR animated objects rather than using only AR human figures. They may not be presented in a

narrative way or shot live. It could be more conceptual and more performative. There could be a lot of action in

repetition, interaction between multi-AR objects or a lot of site-specific performance. The goal is to explore the

meaning of virtuality in contemporary life in a bigger range and a deeper level. Virtuality exists in human percep-

tion, memory, identity, and communication. Virtuality exists in a word, a concept and a history of a culture.

“Time Game”

Duration: 06:29

Dimensions: 720×480

Date: 10.2008

“Cycle”

Duration: 10:21


Date: 10.2008

“The Doll house”

6 channels video installation

Duration: 07:23


Date: 04.2009

“The fire”

Duration: 6:35


Date: 02. 2009

List of pieces included

“The water”

Duration: 12:06


Date: 02.2010

“Undertow”

Duration: 08:55


Date: 12. 2008

“XII-Is”

Duration: 07:16


Date: 12. 2008

“The Organ City”

Duration: 09: 12


Date: 03. 2009

“Mind Traveling”

Duration: 10:21:15


Date: 04. 2009

“In the middle of nowhere”

Digital Print

Dimensions: 20.5 ×17 inches

Date: 02.2009

“My Baby”

Duration: 07:10


Date: 12.2009

“My best Friend”


Duration: 06:25

Dimensions: 720 480

Date: 03.2010

“The New Diamond Sutra”


Duration: 06:18


Date: 04.2010

“Me and I”


Duration: in loop


Date: 06.2010

Bibliography

Susan Blackmore, Consciousness: A very short introduction (Oxford University Press, 2005), 109

Jean Baudrillard, Simulations (Semiotext, Inc., 1983)

Slavoj Zizek, Enjoy your symptom! Jacques Lacan in Hollywood and Out (Routledge, 2001)

Marshall McLuhan, Understanding media: the extension of man (The MIT Press, 1994)

Gilles Deleuze, Cinema 2: the time image (U of Minnesota Press, 1989)

Oliver Bimber, Ramesh Raskar, Spatial augmented reality: merging real and virtual worlds (A K Peters, Ltd., 2005)

Siegfried Zielinski, Deep time of the media: toward an archaeology of hearing and seeing by technical means

(MIT Press, 2006)

Soeng Mu, Mu Soeng Diamond Sutra: transforming the way we perceive the world (Wisdom Publications, 2000)

Technical Experience

Video and Sound Software:

Fianl Cut Pro, Live Type, Motion, Color, Compressor, DVD Studio Pro, Adobe After Effects, Soudntrack Pro, spe-

cialty in HD workflows and surround sound

Video and Sound Hardware:

Jones Sequencer, Jones Frame Buffer, Sandin Image Processor, Doepfer Modular Audio Synthesizer, Analog non-

liner editing.

Interactive Digital Media:

Linceo VR, MAX/MSP and Jitter, Processing, Arduino, Adobe Flash, Adobe Director, specialty in Augmented Real-

ity techniques

3D Modeling Software:

Autodesk Maya, Google SketchUp (work with Google Earth)

Graphic Design Software:

Photoshop, Adobe Indesign, Adobe Illustrator

Printmaking, Digital Print and Photography:

Woodcut, etching, lithography, Giclee and Design Winder drum printing, Large format inkjet printing, Color cali-

bration, 35mm B&W and Color Darkroom, Alternative photography (Cyanotype and Anthotype)

mixed reality_mfa thesis

Documents

new inventions

sense of present means

moyi zhang mixed reality

alfred university

bodys existence

sense of selfhood

different context

different form