gheorghiu stefan 2015 - chnt.at · pdf fileaugmenting immersion: the implementation of the...
TRANSCRIPT
Augmenting immersion:
The implementation of the real world in virtual rea lity
Dragoş GHEORGHIU1 | Livia ȘTEFAN2
1 National University of Arts, Bucharest, Romania | 2 Institute for Computers, Bucharest, Romania
Abstract: The last few decades have seen the rise of a new archaeology characterized by a propensity for
high realism in the representation of the past under the form of virtual reconstructions, which allow the viewer
to have an immersive experience. Our project proposes the insertion of experiments performed in real re-
constructed contexts inside complex 3D virtual reconstructions, thus augmenting the degree of “reality” and of
immersion. The result is a creative mix of virtual reality and reality (mixed reality) under the form of “windows
of immersion”, which can stimulate the archaeological imagination.
Keywords: high realism, virtual archaeology, mixed reality, immersion.
Introduction
Following the modern tradition of 'visual science‘(ARNOLD 2000, 71), the archaeological imagination
(SELLEN 2012) used and uses ‘intermediaries’ (BONDE and HOUSTON 2013, 1) such as photographic (see
SHANKS 2012, 102;1997) images (see SMILES and MOSER 2005) due to their ‘absolutely analogical’
(MITCHELL 1986, 61) character.
Continuing this visual trend of mimetic representation (see BERGER 1984), supported by the new digital
technologies, and due to a sort of ‘technophilia’ (SHANKS and WEBMOOR 2013, 88) or ‘technological
fetishism’ (HUGGETT 2004), the last decades have seen the rise of a trend in archaeology characterized by a
propensity for high realism in the representations of the Past (SHANKS and WEBMOOR 2013, 89), under the
form of virtual reconstructions (BARCELÓ 2001; ABBOTT 2012). Bucking the trend there are however voices
that warn us of the impossibility of a perfect replication (BRYSON 1983, 6; LEIBHAMMER 2000, 131).
The high realism promoted by ‘Virtual Archaeology’ (FORTE and SILIOTTI 1997; BARCELÓ et al. 2000;
NICCOLUCCI 2002) transformed it into an emergent popular subject, because of its capacity to create
photorealistic products, i.e. images with a high content of similitude with the real.
This tendency in archaeology can also be discerned in contemporary culture which also gravitates towards
photorealism (see HYATT 1946; MILLS 1998; EARL 2009), or to a physically-based rendering (BRÖCKER
2010, 3) due, in great measure, to the notable progress of digital imaging techniques. But this tendency to
reproduce nature in all its minutest details has a long tradition in European culture. As early as the
Renaissance, when the difference between artists and scientists was not so clear cut, the new ‘machines for
seeing’ (ARNOLD 2000, 71), i.e. the microscopes, telescopes, and camera obscura (MILLS 1998) permitted
an accurate perception of detail and generated a tendency to a sort of photographic realism, a phenomenon
observed in the scientific illustration and in the paintings of different art schools.
International Conference on Cultural Heritage and New Technologies | Vienna | 2015
2
In archaeology as well one can observe this gradually increasing tendency towards photorealism (in scientific
illustrations), as the last two centuries have seen the transition from bi-dimensional images of objects and
architecture (see BRUSIUS 2012) to the three dimensional reconstructions in Virtual Reality (VR).
Today, when applied to the reconstruction of ancient objects and architecture, the hyperrealism (see
PAPADOPOULOS and EARL 2009) created with the help of digital technologies has proven remarkably
efficient since it induces in the viewer a strong sense of ‘authenticity’ (CHALMERS et al. 2006) and a ‘high
fidelity’ (MCNAMARA et al. 1998; ROUSSOS and CHALMERS 2003) in a ’realistic’ reconstruction of the past
(DEVLIN and CHALMERS 2001; HAPPA et. al. 2009a; HAPPA et al. 2009b).
In the present paper the authors address the “reality” issue of the virtual reconstructions from the point of view
of the visitors’ immersion in the virtual reconstructed medium (KATEROS et al. 2014) presenting the Time
Maps Project. A website (TIMEMAPS 2016a) was designed to support the research and the proposed
solutions for the immersion employing other methods along with the hyper-realistic design and rendering.
Current research
From this perspective of “improvement” of the reality, VAN BOXTEL (2013) asserts that cultural and historical
conventions affect the users’ understanding of virtual reconstructions and pleads for developing a visual
literacy on the example of the “Rome Reborn” project. The visual literacy is defined by PAUWELS (2008) as
“learning to look more consciously at visual manifestations of reality, being able to place images and visual
representations in a broader context of production and consumption, and becoming aware of the personal and
cultural coloring in visual reflection and action”.
To enhance the sense of physically being present according to the definitions of SANCHEZ-VIVES, &
SLATER (2005), KATEROS et al. (2014) employed a gamification paradigm and the newly available VR
equipment Oculus Rift HMD in two different 3D simulation frameworks, Unity3D and an own-developed
gaming framework. In their research the authors concluded that a low-realism but coherent 3D world, “can
lead to higher presence []. If you’re targeting a visually realistic environment, it is more likely to generate
breaks in presence [] because the human brain will expect many things that we are not yet able to achieve
technically: perfect physics, sound, force feedback“ (KATEROS et al. 2014:10).
Unity3D as a research tool
Unity3D is a free gaming engine with 5 million registered developers and 600 million gamers (UNITY3D
2016b). Even though it is a proprietary framework Unity3D is very popular among game developers and
researchers due to its ease of use, high-quality rendering and rapid deployment to web, standalone, mobile
target platforms and recently to WebGL supported browsers. It is also well documented, with a wealth of
examples as well as with an asset store. The programming model employs modern software paradigms and is
flexible, being supported mainly by C# language and JavaScript for Unity (UNITY3D, 2016d).
At its basic level Unity3D can be employed by less experienced programmers, i.e. for creating a 3D scene with
different 3D assets, light and camera settings. For more experienced users, Unity3D offers physical
simulations, real-time lighting, graphical editing, animations, Artificial Intelligence (UNITY3D 2016a). Moreover,
Unity3D is an augmented/virtual reality (AR/VR)-ready platform, by allowing the integration with AR libraries,
Gheorghiu | Ştefan – Augmenting immersion
3
e.g. Vuforia (VUFORIA 2016), and with Oculus Rift or Samsung Gear VR devices (UNITY3D 2016c). Unity3D
web applications require a web plugin, downloadable as an ActiveX component.
A competitive authoring tool is the Unreal gaming engine (UNREAL 2016), which also supports the
development of VR applications with Oculus Rift HMDs. Unreal requires C++ programming which is not so
popular but which provide better web performance and compatibility with the web standards.
Time Maps’ technical solution
The Time Maps project intends to recover the ancient technologies and to study the spatial experience of the
contexts in which these have been utilized (GHEORGHIU and ȘTEFAN 2012). A case study of the project was
the experimental reproduction of the technologies of a Roman villa rustica. One corner of the villa has been
reconstructed in reality (Fig. 1) and several archaeological experiments with glass, ceramics and textiles have
been filmed. Afterwards, the whole villa was modelled in 3dMax and a Unity3D-based VR application was
implemented and integrated in the TimeMaps website. The rendering is performed by the Unity3D web plugin.
The architectural form and the context were rendered with textures and colors taken from the archaeological
record. The illumination (GHEORGHIU in press) took account of a specific moment of the day, thus trying to
provide a more accurate (realistic) image of the site. In other experiments, e.g. Mangalia and Albești, the
forms and textures of the ancient objects were scanned for a hyper-realistic representation.
Another solution for improving the “reality” in the 3D reconstructions was the scanning of objects and of
historical characters, (i.e. actors dressed in epoch costumes), and their transfer in the virtual reconstructions.
In the “Mangalia” (TIMEMAPS 2016b) and “Albeşti” (TIMEMAPS 2016c) pages of the TimeMaps website, the
scanned objects and ”live persons” were integrated in the VR applications, beside the video hotspots with the
same characters performing in re-enactments, their highly definition details augmenting the degree of
immersion of the visitor in the 3D simulation (Figs. 2, 3)
In the VR application several video hotspots linking to the video films already mentioned have been inserted in
the exact place where the technologies have been utilized. The insertion have been made possible by means
of sensitive images that when clicked open a video player (Fig. 4).
International Conference on Cultural Heritage and New Technologies | Vienna | 2015
4
Fig. 1 – Experimental reproduction of a Roman villa rustica (Gheorghiu and Ștefan 2012)
Fig. 2 – A Hellenistic villa from Kallatis-Mangalia
Gheorghiu | Ştefan – Augmenting immersion
5
Fig. 3 – A Hellenistic villa in Albeşti village
Fig. 4 – The insertion of video hotspot in the 3D reconstruction
International Conference on Cultural Heritage and New Technologies | Vienna | 2015
6
Results
In the case of the virtual reconstruction of the Roman villa rustica the viewer has the possibility to perceive the
whole reconstructed space (rendered in a highly realistic manner) and to have a spatial experience of the
original context. Several hotspots (Fig. 5) lead to videos (Fig. 6) depicting technologies existent at that time
(metal, glass, ceramics, textiles) and augmented with historical and contextual information
By accessing the hotspots, the user navigating through the VR application exits the virtual medium through a
“window” and enters the “real” world. Thus, the “reality” of the hotspot offers more credibility (reality) to the
highly-rendered virtual reconstruction, and improves the immersion.
The educational value of the project due to its high immersive potential was proved in various schools in urban
and rural areas in Romania (ȘTEFAN and GHEORGHIU 2014) and Portugal. Students have used the virtual
reconstructions as a game, and explored both the 3D space and the immersive “windows”.
Fig. 5 – Hotspots leading to videos depicting technologies existent in Roman villa rustica
Fig. 6 – A video film showing the use of a kiln.
Gheorghiu | Ştefan – Augmenting immersion
7
Lessons learned
Experiments
The experience of mixing the virtual with the real underlined the importance of the presentation of the
experiment in a high-realistic context. Even if the context was virtual, this has situated the human action in a
larger frame of forms, materials and illuminations, and permitted a spatial experience of the environment,
difficult to implement in reality.
Experientiality
A second lesson is that of the experientiality (i.e. subjective experience) (see CUNNINGHAM et al. 2008, VI)
which is generated by the recuperation of the “reality” of the past accomplished by combining the scanned
objects and humans with the video films. The experientiality of the virtual space (the first phase of the
immersion) is continued by the experientiality from the real hotspot along with the scanned objects.
The double experientiality, which implies the body agency, also permits a greater perception of “presence”
(LOMBARD and DITTON 1997; HERRERA et al. 2006; JI and WAKEFIELD 2016:17).
Altogether the methods enhanced the reality of the reconstruction and provided an improved educational
value.
Imagination
Last but not least, the advanced state of immersion has stimulated the archaeological (SHANKS 2012) and
creative imagination (see MCCAULEY 2009, 62ff) both of the youngsters and of the specialist users who
participated in the project.
Future work
In spite of the positive results among history teachers, primary school students and scientists, our Mixed
Reality (MR)-based solutions can be improved for a tighter mixture of the virtual and real, to produce a
continuum virtual-real-virtual. In the current solution, the users are exiting the virtual context to be able to
immerse in the video re-enactments. This limitation of our design is caused by the Unity3D web plugin, which
acts as a “weak link”: the requirements to install this plugin provide delays in the loading of the web VR
applications and incompatibilities with mobile access of the VR applications. Furthermore, the future browsers,
such as WebGL, will no longer support the plugins (CHROME 2016), including the Unity3D web plugin, as a
method to promote the use of standard web technologies and provide a more natural web experience.
Running the present TimeMaps MR application on the web will not be possible, provided that older Internet
Explorer or Mozilla browsers will be employed.
As a future work we plan to create a continuum comprising the scanned objects and the video films. For these
purposes we shall experiment the new Unity3D 5.0 deployment for WebGL (WEBGL 2016), a JavaScript
library for rendering interactive 3D and 2D graphics within any compatible web browser and without the use of
plugins. We also will attempt a solution based on X3DOM, a cross-platform JavaScript open-source
framework. X3DOM (X3DOM 2016) supports native implementations (iOS8, Chrome and Firefox for Android),
International Conference on Cultural Heritage and New Technologies | Vienna | 2015
8
so this solution could solve the compatibility with the mobile devices. X3DOM supports more immersive MR
applications (X3DOM 2016), by a natural integration of the videos within the 3D content, which could better
serve the objectives of our project.
Acknowledgements
The authors thank Mr. Wolfgang H. Börner for his kind invitation to publish the paper presented at the 20th
International Conference on Cultural Heritage and New Technologies, November 2015, Vienna, and Dr.
Willem Beex, Prof.Dr. Giorgio Verdiani and Prof. Dr. Peter Ferschin for their useful remarks during their
session “Virtual, augmented reality and other techniques in Cultural Heritage for the general public”. Many
thanks to M. Bogdan Căpruciu for his help in improving the translation of the present text.
The reconstructions from the Vadastra site were coordinated by Professors Andreea Hasnaş and Dragoş
Gheorghiu and were integrated in the Unity3D-based VR application by Golem Company. The reconstruction
from the Albeşti site was coordinated by Dr. Robert Constantinescu, and the scanning and the animation from
Mangalia and Albeşti were performed by Marius Hodea and Liviu Ungureanu.
TimeMaps is funded by an exploratory research, grant PN II IDEI (Time Maps. Real communities, Virtual
worlds, Experimented Pasts).
References ABBOTT, M. (2012). Visualising Stonehenge: A virtual archaeology, In: D. Arnold, J. Kaminski, F. Niccolucci, and A. Stork, eds., The 13th
International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST (2012). pp.1-4.
ARNOLD, K. (2000). Between explanation and inspiration: Images in science. In: E. Sian, ed., Strange and charmed. Science in the
contemporary visual arts. London: Calouste Gulbenkian Foundation. pp.68-83.
BARCELÓ, J.A. (2001). Virtual reality for archaeological explanation beyond ‘picturesque’ reconstruction. Archeologia e calcolatori,
pp.221-244.
BARCELÓ, J. A., M. FORTE and D. H. SANDERS (eds.) (2000). Virtual reality in archaeology. British Archaeological Reports
International Series 843. Oxford: Archaeopress.
BERGER, J. (1984). Ways of seeing. London: BBC.
BONDE, S. and S: HOUSTON (2013). Re-presenting the Past. Archaeology through text and image. Oxford and Oakville: Oxbow.
BOXTEL, S. van (2013). Reviving the Past. Conventions of realism in the virtual reconstruction of Rome Reborn, Retrieved from
https://siemvanboxtel.files.wordpress.com/2012/12/reviving-the-past.pdf, Accessed January 2016.
BRÖCKER, M. (2010). Photorealistic rendering in the context of spatial Augmented Reality. Techniques and implementation.
Saarbrücken: VDM Verlag Dr.Müller.
BRUSIUS, M. (2012). Misfit object: Layard’s excavations in Ancient Mesopotamia and the Biblical imagination in mid-nineteenth century
Britain. Journal of Literature and Science 5 (1), pp.38-52.
BRYSON, N. (1983). Vision and painting: the logic of the gaze. New Haven and London: Yale University Press.
CHALMERS, A., I. ROUSSOS and P. LEDDA (2006). Authentic Illumination of Archaeological Site Reconstructions. CGIV, pp.431-434.
CUNNINGHAM, P., J. HEEB and R. PAARDEKOOPER (eds.) ( 2008). Experiencing archaeology by experiment. Oxford: Oxbow Books.
DEVLIN, K., and A. CHALMERS (2001). Realistic visualisation of the Pompeii frescoes, Proceedings of the 1st international conference
on Computer graphics, virtual reality and visualization, ACM, pp. 43-48.
Gheorghiu | Ştefan – Augmenting immersion
9
EARL, G. (2009). Physical and photo-realism: the Herculaneum Amazon. Arqueologica, 16 - 19 Jun 2009, 2.0, Seville, Spain.
EVANS, A., M. ROMEO, A. BAHREHMAND, J. AGENJO and J: BLAT (2014). 3D graphics on the web: A survey, Accessed January
2016 www.Academia.edu.
FORTE, M. and A. SILIOTTI (eds.) (1997). Virtual Archaeology: Great Discoveries Brought to Life through Virtual Reality. London:
Thames and Hudson.
GHEORGHIU, D. (in press). Lighting in reconstructed contexts: Archaeological experiments and experientiality with pyrotechnologies, In:
Papadopoulos, C. (ed.), Archaeology and the Language of Light: Interdisciplinary Studies in Experience and Perception, Oxford, Oxford
University Press.
GHEORGHIU, D. and L. ȘTEFAN (2015). Preserving Monuments In The Memory Of Local Communities Using Immersive MAR
Applications As Educational Tools, The 10th International Conference on Virtual Learning, Universitatea de Vest, Timisoara.
GHEORGHIU, D. and L. ȘTEFAN (2014). Memory and Immersive Applications. The use of MAR to preserve Local Tangible and
Intangible Heritage (poster), E-iED 2014: 4th European Immersive Education Summit, University of Applied Science BFI, 24-26 November
2014, Vienna, Austria.
GHEORGHIU, D. and L. ȘTEFAN (2012). Mobile Technologies and the Use of Augmented Reality for Saving the Immaterial Heritage,
The 13th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST2012, Brighton, pp.21-24.
HAPPA‚ J., A. ARTUSI‚ P. DUBLA‚ T. BASHFORD−ROGERS‚ K. DEBATTISTA‚ V. HULUSIĆ and A: CHALMERS ( 2009a). The Virtual
Reconstruction and Daylight Illumination of the Panagia Angeloktisti. Proceedings of VAST: International Symposium on Virtual Reality‚
Archaeology and Cultural Heritage 2009. pp.49-56.
HAPPA, J., M. A. WILLIAMS, G. A. TURLEY, G: EARL, P.DUBLA, G. BEALE, G. GIBBONS, K. DEBATTISTA and A. Chalmers (2009b).
Virtual relighting of a Roman statue head from Herculaneum: a case study. Afrigraph 2009. pp.5-12.
Herrera, G., R. JORDAN, and L. VERA (2006). Agency and presence: A common dependence on subjectivity?, Presence 15 (5), pp. 539-
552.
HUGGETT, J. (2004). Archaeology and the new technology fetishism, Archeologia e Calcolatori.
HYATT, M. A. (1946). The Photographic Eye. Bulletin of the Metropolitan Museum of Art, New Series, Vol.5, No.1. pp.15-26.
JI, H., and G. WAKEFIELD (2016). Endogenous biologically inspired art of complex systems, IEE Computer graphics and Applications 36
(1), pp.16-21.
KATEROS, S., S. GEORGIOU, G. PAPAGIANNAKIS, and M. TSIOUMAS (2014). A comparison of gamified, immersive VR curation
methods for enhanced presence, Eurographics 2014, Strasbourg.
LEIBHAMMER, N. (2000). Rendering realities. In: I. Hodder, ed. Towards Reflexive Method in Archaeology: The Example at Çatalhöyük,
BIAA Monograph 28, Cambridge: The McDonald Institute of Archaeological Research. pp.129-142.
LOMBARD, M. and T. DITTON (1997). At the hearth of it all: The concept of presence, Journal of Computed-Mediated Communication 3
(2).
MCCAULEY, D. (2009). Night and shadows: The space and Place of Darkness. Environment, Space, Place, vol. 1. Bucharest: Zeta
Books, pp.51-76.
MCNAMARA, A., A. CHALMERS, T. TROSCIANKO and E. REINHARD (1998). Fidelity of Graphics Reconstructions: A Psychophysical
Investigation. In Rendering Techniques, pp.237-246.
MILLS, A. A. (1998). Vermeer and the Camera Obscura: Some Practical Considerations. Leonardo, Vol.31, No.3, pp.213-218.
MITCHELL, W. J. T. (1986). Iconology: Image, text, ideology. Chicago: University of Chicago Press.
NICCOLUCCI, F., (ed.) (2002). Virtual Archaeology, Proceedings of the VAST Conference, Arezzo, Italy. B.A.R. International Series 1075.
Oxford: Archaeopress.
International Conference on Cultural Heritage and New Technologies | Vienna | 2015
10
PAPADOPOULOS, C. and G. EARL (2009). Structural and Lighting Models for the Minoan Cemetery at Phourni, Crete, In: Debattista, K.,
Perlingieri, C., Pitzalis, D. and Spina, S. eds. Proceedings of the 10th VAST International Symposium on Virtual Reality, Archaeology and
Cultural Heritage, VAST2009, Malta.
PAUWELS, L. (2008). Visual Literacy and Visual Culture: Reflections on Developing More Varied and Explicit Visual Competencies. The
Open Communication Journal, 2, 79-85.
ROUSSOS, I., and A. CHALMERS (2003). High Fidelity Lighting of Knossos. Proceedings of the 4th VAST International Symposium on
Virtual Reality, Archaeology and Cultural Heritage, VAST 2003, Brighton, pp.195-202.
SELLEN, A. T. (2012). Nineteenth-Century photographs of archaeological collection from Mexico. In: Pillsbury, J. ed., Past Presented.
Archaeological illustration and the Ancient Americas, Dumbarton Oaks Pre-Columbian symposia and colloquia, Washington D.C. pp.207-
229.
SHANKS, M. (2012). The Archaeoological imagination. Walnut Creek: Left Coast Press.
SHANKS, M., and T. WEBMOOR (2013). A political economy of visual media in archaeology, In: Bonde, S. and Houston, S., Re-
presenting the Past. Archaeology through text and image. Oxford and Oakville: Oxbow. pp.85-108.
SMILES, S. and S. MOSER, S. (eds.) (2005). Envisioning the Past: Archaeology and the Image. Oxford: Blackwell.
SANCHEZ-VIVES, M., and M. SLATER (2005). From presence to consciousness through virtual reality. Nature Reviews Neuroscience 6,
4, pp. 332–339.
ȘTEFAN, L., and D. GHEORGHIU (2014). 3D cyber-communities of learning. An immersive educational strategy for rural areas,
International Conference Smart 2014 Social Media in Academia: Research and Teaching, 18-21 September 2014, Timisoara, Romania.
CHROME (2016). https://www.chromium.org/developers/npapi-deprecation
X3DOM. (2016). http://www.x3dom.org/documentation/tutorials/generic-3d-data-conversion
WEBGL (2016). http://www.khronos.org/webgl
TIMEMAPS. (2016a). http://timemaps.net
TIMEMAPS. (2016b). http://timemaps.net/timemap/mangalia/?page_id=3401
TIMEMAPS. (2016c). http://timemaps.net/timemap/albesti/?page_id=3288
UNITY3D. (2016a). http://www.unity3d.com
UNITY3D. (2016b). http://unity3d.com/public-relations
UNITY3D. (2016c). http://docs.unity3d.com/Manual/VRDevices-Oculus.html
Unity3D. (2016d). http://docs.unity3d.com/Manual/CreatingAndUsingScripts.html
UNREAL ENGINE. (2016). http://www.unrealengine.com
VUFORIA. (2016). https://developer.vuforia.com/library/articles/Solution/Installing-the-Unity-Extension
Imprint:
Proceedings of the 20th International Conference on Cultural Heritage and New Technologies 2015 (CHNT 20, 2015)
Vienna 2016
http://www.chnt.at/proceedings-chnt-20/
ISBN 978-3-200-04698-6
Editor/Publisher: Museen der Stadt Wien – Stadtarchäologie
Editorial Team: Wolfgang Börner, Susanne Uhlirz
The editor’s office is not responsible for the linguistic correctness of the manuscripts.
Authors are responsible for the contents and copyrights of the illustrations/photographs.