Storytelling and Content presentation with the Virtual Showcase in a museum context

Steffi Beckhaus1, Florian Ledermann2, Oliver Bimber3

1 Fraunhofer IMK, Germany, 2 Technische Universität Wien, Austria, 3 Bauhaus Universität Weimar, Germany

Abstract

This paper gives an overview of the Virtual Showcase as an augmented reality display system for museums. It explains about different hardware prototypes, interaction tools as well as several software techniques to utilize the features of the Virtual Showcase. It also presents two case studies, one from paleontology and one from archeology.

Keywords: Virtual Showcase, Digital Storytelling, Augmented Reality

1 Introduction

Virtual and augmented environments can be utilized to enhance the experience of museum visitors. With Augmented Reality (AR), the extension of real objects with virtual content, additional layers of information and experiences can be added to otherwise static objects. As virtual environments (VEs), augmented environments (AEs) can address all human senses. The presented content can be synthetically complemented with visual, auditory, haptic, and even olfactory (smell) information. We believe that these new forms of presentation hold the prospect of enhancing the experience of museum visitors. Computer-generated content can be either predefined to tell a story, or it can be fully interactive, allowing the visitors to explore the content by themselves.

This paper will give an overview of the usage of the Virtual Showcase (VS) in the museum context. We describe the Virtual Showcase as an innovative spatial Augmented Reality display that offers an imaginative and exciting way of accessing, presenting, and interacting with scientific and cultural content. We give an overview of different existing prototypes, their technical components, and software features in Section 2. In Section 3, we briefly describe storytelling possibilities with the VS and explain about technical methods to interact with the VS. We will then describe the recent applications of the Virtual Showcase to the field of digital story-telling for education and scientific visualization in Section 4. In particular we will describe two case-studies. The first presents the scientific findings of leading paleontolologists to a non-scientific audience. The second presents an archeological model of a gate ruin. The acceptance of this new presentation form by potential museum visitors, as well as the quality of the visual appearance of such mixed content was evaluated in a user study [Bimber et al., 2003]. Section 5 concludes the paper.

2 Virtual Showcase Technology

The Virtual Showcase, first presented in [Bimber et al. 2001], is a new Augmented Reality (AR) display system. This museum display provides an imaginative method for accessing, presenting, and interacting with scientific and cultural content. Conceptually, the Virtual Showcase is compatible with conventional showcases. Real scientific and cultural artifacts are placed inside the Virtual Showcase allowing their augmentation with computer- generated 3D graphics and animations within the same 3D space.

As virtual representations and real artifacts share the same space, this provides new ways of merging and exploring real and virtual content. Solely virtual exhibits may also be displayed. The VS supports

up to four users looking at the Virtual Showcase from different sides watching the same content from different sides. This feature allows the collaborative exploration of artifacts shown in the Virtual Showcase.

Three different kinds of prototypes yet exist: a pyramid shaped projection based setup, a cone shaped projection based setup, and a cost effective CRT setup.

2.1 The Pyramid and Cone Shaped Projection Based Virtual Showcase

The Virtual Showcase consists of two main parts (the numbers in parentheses correspond to the numbers in Figure 1): a convex assembly of half silvered mirrors (1) and a graphics display (2). Two different mirror configurations can be used. The first prototype consists of four half silvered mirrors assembled as a truncated pyramid (see Figure 1a). The second prototype uses a single mirror sheet to form a truncated cone (see Figure 1b). We placed these mirror assemblies on top of a projection screen (2). Users can see real objects inside the showcase through the half-silvered mirrors merged with the graphics displayed on the projection screen. The showcases’ content is illuminated with a controllable light source (3) while presenting view-dependent stereoscopic graphics to the observer. These prototypes use standard shutter glasses (5) controlled by infrared emitters (4). Head tracking which lets the system render the augmented graphics from the correct perspective is accomplished using an electromagnetic tracking device (6). It is also possible to use a wireless infrared tracking device.

Figure 1: Pyramid Shaped (a) and Cone Shaped (b) Virtual Showcase

The pyramid-shaped prototype supports up to four viewers simultaneously looking at the showcase from four different sides. The cone-shaped prototype provides a seamless surround view onto the displayed artifact.

2.2 CRT based Virtual Showcase

In contrast to the previous prototypes, the CRT variation of the VS utilizes up to four CRT monitors, instead of a single CRT projector [Bimber et al., 2003]. This reduces the cost of the entire display to a fraction of the cost of a suitable projector. In the projection based approaches, multiple users were supported by partitioning the screen area (as well as the image resolution) into five sections: one for each of the four possible users, and one for the image area that is located underneath the mirror optics. The monitor-based setup provides an UXGA resolution per screen and consequently increases the image resolution per user by a factor of five.

The display panel of the monitors has been coated with a light directing foil. This foil directs the light exclusively from the monitor towards the mirror optics, which makes the stereoscopic images appear only inside the mirror optics – not on the monitor. Thus, observers are not distracted by the source images on their own, or on the opposite display anymore. Since the monitors can be tilted towards the mirrors by an arbitrary angle, it is possible to cover the entire height of the mirror assembly by graphical overlays. All technology, such as PCs, tracking system, monitors, video beamers and sound system have been integrated seamlessly into the frame of the display. They are not visible from the outside anymore.

Figure 2: CRT based Virtual Showcase

Figure 2 shows the CRT based version of the Virtual Showcase with again the numbers in parentheses corresponding to the numbers in Figure 2: a pyramid shaped assembly of half silvered mirrors (1), one of up to four CRT monitors (2) whose image forms the virtual object as can be seen in the mirror (9), a light projector (3), standard shutter glasses with a tracker sensor (5), an electromagnetic tracking device (6). This prototype features the IR stylus described below with a camera (7) and the stylus in the user’s hand (8). Inside the prototype is storage for PCs (10).

2.3 Software and Computer Hardware

The VS is driven by off-the-shelf PCs with conventional 3D graphics cards. Three different software solutions are currently available to drive the Virtual Showcase display. The MS Windows based VS player was explicitly designed for the use with the VS. It features all necessary techniques for the setup and rendering in the VS context, some of them described in this chapter. This software is able to drive two monitors simultaneously with an UXGA resolution. Consequently, only two PCs are required to drive a four-user configuration. Depending on the requirements, additional PCs may be needed to support a projector-based illumination.

More complex, interactive presentations are created using our APRIL authoring framework [APRIL]. APRIL provides concepts for hardware-configuration, story authoring, animation and interaction in an

XML-based file format, easy to use also by non-experts in computer graphics or programming. APRIL uses the concept of state-engine driven stories to overcome the apparent contradiction of realizing complex, interactive presentations with simple, robust interaction tools in a museum setting. In addition, it is possible to use different interaction mappings, depending on the target audience and the setting of a presentation - stories can be run in a fully interactive mode, or a "demo mode" that runs without any user interaction.

Avango™ is a virtual environment development platform and a software framework for complex and distributed virtual environments. It was recently extended to include tools for non-linear storytelling in virtual environments and it offers support for many VR interaction tools like the IR stylus. To use its wide range of features, it is currently adapted to drive the Virtual Showcase. Avango is available for SGI Irix and Linux.

2.4 Presentation Features of the Virtual Showcase

2.4.1 Overlay of Real and Virtual Objects For the integration of real and virtual objects in the Virtual Showcase, the scanned geometric representation must be registered against its physical counterpart first. Doing so lets the Virtual Showcase compute the illumination and occlusion effects directly onto the real object’s surface. A simple mouse-based interface allows to interactively place the pre-modeled virtual objects on or inside the real object at their appropriate position.

2.4.2 Projector-based Illumination Simulating realistic occlusion and illumination effects between the real objects, e.g. physical bones in our Raptor example, and the virtual objects, e.g. soft tissues, is essential for the presentation (see Figure 4 (c,d,f)). A drawback of such displays is that achieving realistic occlusion effects between real and virtual components is difficult. To overcome this difficulty, the Virtual Showcase uses controllable video projectors instead of simple light bulbs [Bimber and Fröhlich, 2002, Bimber et al., 2003 b].

The video projectors create view-dependent lighting effects on the real objects surface. Generating shadows on the physical object exactly beneath the overlaid graphics, for example, lets virtual parts mutually occlude the underlying real surfaces. Having the real object’s depth information, on the other hand, makes it possible to cull the occluded graphics before they display. Thus, the real objects can occlude virtual components and vice versa as best visible in Figure 4 (f). This strongly enhances the interactive presentation’s realism.

2.4.3 Projector-based Augmentation By replacing a physical object—with its inherent color, texture, and material properties—with a neutral object – same in shape but lacking these properties - and projected imagery, projector-based augmentation can directly reproduce either the object’s original or altered appearance. This approach effectively lifts the object’s visual properties into the video projector. Projector-based augmentation is ideal when the physical object is available for visualization, even if it forms a complex geometric shape. Multiple users can view the augmented object simultaneously, without using stereo glasses or head-mounted tracking devices. For a high-quality augmentation, the system computes an image of the original object’s textured 3D graphics model from the projector’s viewpoint. When projected, the rendered image appears smoothly registered with the neutrally colored object, which changes its appearance accordingly [Raskar et al. 2001].

3 Digital Storytelling and Interaction with the Virtual Showcase

Presentations showing non static content can potentially explain more about an exhibit and are more interesting to the visitor than static exhibits. Interactive installations can make the experience of museums even more vivid and interesting. By engaging users into the presentation process they are more involved, their attention is fully caught, and they can learn more.

A presentation or story can for example be influenced on the timeline. This enables users to move forward and backward in the otherwise linear presentation to revise certain parts or modify the duration of parts of the presentation for a more thorough experience. Much more complex story structures are possible. For example in the alVRed Project [alVRed, 2002], non-linear stories for

virtual environments are generated. Event-based, hierarchical stories allow for any degree of complexity of the story. The drawback may be that for a clear presentation of museum content the story has to have a certain dramaturgical and content structure. To combine both, fully interactive parts of the story can be integrated into an otherwise linear or branching storyline.

To influence the story, ways to interact with the story have to be provided. Developing interaction techniques and devices for the museum context that can be integrated into digital stories is a challenging task. This is even more difficult, if multiple users are involved. For multiple user scenarios, additionally it can be differentiated between a guided, individual and cooperative interaction [Bimber et al., 2003].

The interaction with a story can be guided by a dedicated user, while the other users observe the same outcome and continuation. An individual interaction allows each user to interact with its own variation of the same story. In this case, each user is completely independent of the others. A cooperative interaction allows each user to influence the continuation of the same story that is observed simultaneously by all users. In contrast to the other approaches, mechanisms that resolve conflicting interaction and continuation situations are required for this case.

Note that the VS player software currently supports only linear timeline based stories. For this, only a mouse-based, story independent and guided interaction is offered to transform single virtual components. Avango™ provides a full VR framework which is capable of implementing hierarchical non-linear stories. This is used in large projection-based theatres for presenting stories to a large audience.

3.1 Interaction Tools

To control an interactive story or installation, interaction tools have to be available to the visitor of a museum. These have to be adapted to the visitor structure. As they are open to the public and often are visited by children, young adults and school groups, displays and their interaction tools have to be easy to understand and robust enough to survive rough treatment. This means that the interaction with the presentation or story has to be either touch less or is done with simple, robust tools.

Touch-less interaction is for example possible by using cameras and hand gesture recognition. For the latter, Arcade game controllers offer a wide range of interaction tools like trackballs, button, joysticks and such (see Figure 3). It is easy to integrate those into the VS, as all of the prototypes feature a table high setting. This allows for the integration of such tools in user hand height. With tracked props – simple replicas of objects in the presentation which manipulate their virtual counterpart in position and orientation – intuitive interaction with objects can be provided. The meaning is gasped by recognizing the similarity between the virtual and the real object, providing that the virtual object acts according to the real objects (the props) behavior. This requires tracking (knowing the position and orientation) of the prop.

Tracking of a prop or the user’s head can be done by using optical or electromagnetic tracking. Optical trackers do not require a connection of the prop with the VS. On the other hand they need cameras and visible markers to work properly. Electromagnetic trackers generate a strong electromagnetic field. The prop itself has to incorporate a sensor which is attached to the computer. Many VR systems work with tracked tools like the Stylus, a pen like device. For object selection, e.g. a ray can be shot into the virtual space. Or an object can be directly manipulated by moving the Stylus. If only rotational information (the orientation of the prop) is required, there are trackers available which work with sensors using gravitation. They do not need optical marks or an electromagnetic field to work.

The IR stylus, shown in Figure 2, is an optical tracker which needs a camera but no markers. It is a laser pointer technique, where a laser beam emitted by the stylus is splitted and produces two infrared spots on the semi transparent mirror of the Virtual Showcase. These are evaluated to retrieve the position and orientation of the stylus [Matveyev et al, 2003].

Figure 3: Trackball, Button and Joystick as Robust Interaction Tools for the Virtual Showcase

A natural way of interacting with objects in the Virtual Showcase is pointing with the finger. A video camera with a gesture recognition software can analyze the user’s gestures. A more robust method is to add a transparent touch screen to the front of the showcase to detect the position of the finger tip. Together with the user’s head position information, this provides a low cost and natural interaction. A new method of detecting the intrusion of a finger into a plane was developed by HoloTouch™. It uses an infra-red detector to scan the plane and would make even the touch screen needless [HoloTouch].

4 Case Studies

We will describe two case studies using the Virtual Showcase for the presentation of content. The presentations were designed by the authors but each user will live through their individual story by experiencing the interactive presentations.

4.1 The Raptor

The first case study is from paleontology. The Deinonychus is an early cretaceous dinosaur. Often only bone structures remain of extinct creatures like these and it is the task of Paleontologists to research and visualize the rest. They use two-dimensional drawings sketched over fossil bone photographs or 3D clay or plastic models to illustrate how soft-tissue structures share the limited space within a complex anatomical area such as the skull of such ancient creatures. Although museum displays commonly feature these forms of expression, a major drawback is that they lack dynamics and interactivity.

The Virtual Showcase together with the VS player software was used to dynamically visualize reconstructed soft tissues that have been fully integrated with the fossilized bone structure [Bimber et al., 2002]. This helps to assess the conformational relationships of the reconstructed components to test the soft-tissue anatomy model. For museum visitors, this augmented paleontology combined with digital storytelling enables a more exciting and interactive experience, and it has the potential for improving knowledge transfer.

In this example, the cast skull of a Deinonychus was augmented with 3D computer models of reconstructed soft tissues and missing bones. Observers can perceive the real and virtual components together in stereo from any perspective using the Virtual Showcase. Figure 4 shows the physical skull placed inside the 3D display (a), the virtual representation of the skull (b), the physical skull augmented with the reconstructed soft tissues (c,d), and covered with virtual skin (e). Figure 4 (f) shows that with projection-based illumination correct mixtures of real (the board) and virtual objects (the skull) are possible. Note that the virtual object additionally castes a shadow onto the real object.

Off-the-shelf tools, such as 3D modeling and animation software, as well as 3D laser scanning technology of the real artifacts were used to generate the content of the Raptor application. Photographs of the skull were received from the paleontologists, which contain hand-drawings of the soft-tissues’ locations within the skull area. In addition they told how these components interacted with each other. The physical skull was laser-scanned. A low-resolution version of scanned geometry was used to compute occlusion and illumination effects during the presentation. The reconstructed soft-

tissues, such as several muscle components, the paranasal air sinus, the bony eye rings, the skin, as well as text-labels have been modeled and animated with 3D-Studio MaxTM. Audio files were recorded by the paleontologist in a studio to provide additional verbal information.

To simulate interaction and behavior in different situations, the virtual components were animated. To render the graphics, a conventional game engine was used that provides both high-quality animation and interactive frame rates. Enhancing the presentation with synchronized audio output and projector-based illumination achieves an effective form of digital storytelling by dynamically fading in and out specific parts of the physical skull. During the presentation, different soft-tissue layers and components display over time, while a variety of multimedia aids—such as voice, text annotations, graphical animation, and lighting effects—explain their functions and relationships.

This application was already demonstrated at several exhibitions including the Emerging Technology Laboratory at ACM Siggraph 2002, USA and the Learntec Conference in Karlsruhe, Germany in 2003. It was used by over 5000 people in a short period of time. Results of this case-study are published in [Bimber et al., 2003 a]

Figure 4: Several views of the Raptor Demo

4.2 The Heidetor Presentation

The Heidentor (Heathen Gate) is an ancient roman ruin, located in Petronell-Carnuntum/Austria, and one of the most well known roman ruins in Austria (Figure 5). Originally, the Heidentor was not a gate, but it had 4 pylons forming a so called tetrapylum, a double-passage arc located at the intersection of two major roads. The pylons were supporting a 2-floor building on top of them. Today, only 2 of the pylons are still intact, and form a gate-like ruin that is visible from far away. The exact purpose of the Heidentor remains unclear; it might have been a tomb or a triumphal arch.

For the Virtual Showcase, the TUW has built a scale model of the current Heidentor ruin, to be augmented with additional explanations and reconstructions of the original building. The focus in this presentation is on user interaction. Users should be able to interactively explore the model and discover explanations about the historical facts. As the presentation is targeted towards a younger audience, the interactive features should help to gain attention and raise interest in the subject by offering the possibility to explore it individually instead of just watching a linear presentation.

The scale model was built out of cardboard based on exact archaeological measurements. It was then laser-scanned. For the virtual part of the presentation, two different historical reconstructions of the original building were used, as well as photographs, sketches and a video.

As visitors approach the exhibit, they are introduced to the subject by an animated virtual character (Figure 6). The character tries to get the attention of the visitor, and presents the model of the ruin inside the showcase. After a basic introduction, the visitor can use either virtual buttons to advance the presentation in a linear fashion, or use the provided tools for interactive exploration to get additional explanations for certain parts of the building.

Since the model is enclosed inside the Virtual Showcase, visitors cannot interact directly in the space of the model. Instead, they can use a virtual laser pointer to point at parts of the model that they are interested in (Figure 7). Also they can make use of a virtual clipping plane, to “cut away” the virtual model and make the real model underneath visible (Figure 8). This helps the visitors understand the relationships between the ruin as it looks like today, and the original building.

Figure 5: The Heidentor ruin, with a low-tech augmented reality setup in place to show a possible reconstruction of its original state.

Figure 6: An animated virtual character introduces the Heidentor.

Figure 7: Using a virtual laser pointer to explore additional information. Here, a sketch of the reconstruction of a similar building is shown.

Figure 8: Blending between the real model and the reconstruction with a virtual clipping plane.

The interactive presentation is authored and controlled using the APRIL authoring framework. An abstract representation of the interactive story is used to control the flow of the presentation and the interactions that are offered to the user (Figure 9). Depending on the stories state, interaction tools can be shown, contents can be animated and additional media like sound or video can be played. Therefore, the Heidentor presentation allows the visitors to leave the linear path of a presentation, and start exploring the exhibit on their own, with the environment reacting accordingly to their actions.

Figure 9: The Heidentor presentation represented as a state machine. Nodes in the graph represent behaviours, edges represent user interactions.

5 Conclusion

In this paper we presented the Virtual Showcase as an augmented reality device. It enables museums to present real and virtual content to tell more interesting stories to their visitors. With the Virtual Showcase it is possible to extend real artifacts with virtual content. Non-available parts of a real object can be added to show the original appearance of an object. Both case studies used this technique to reconstruct the original body and gate.

With virtual annotations real or virtual objects can be explained and parts of the objects can be highlighted. Other than with real objects, virtual content can be animated without specific hardware. Layers of information, both textual and graphical can easily be added or subtracted to allow for an understanding of interdependencies, the interior of closed objects and more. These features allow for a complex explanation of museum artifacts.

Interaction tools allow for influencing an otherwise linear story. This influence can take place on the timeline but it can also provide for a highly complex interactive experience.

Current hardware developments make it possible to construct a Virtual Showcase for a reasonable price. Also the current CRT version is more robust than projector based prototypes. We believe that with the current Virtual Showcase developments a promising platform for presentation of real and virtual content in the museum context is available. .

References

alVRed http://www.alvred.de, http://informatiksysteme.pt-it.de/vr-ar-2/projekte/alvred/beitrag_ALVRED.pdf, 2002

APRIL, http://www.studierstube.org/april

Bimber, O. Interactive Rendering for Projection-Based Augmented Reality Displays, Ph.D.Dissertation, Darmstadt University of Technology,2002.

Bimber, O., Encarnação, L.M., and Schmalstieg, D., The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling. 9. Eurographics Workshop on VE, The Eurographics Association, 2003 (a).

Bimber, O. and Fröhlich, B. Occlusion Shadows:Using Projected Light to Generate Realistic Occlusion Effects for View-Dependent Optical See-Through Displays. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR’02), pp. 186-195, 2002.

Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. The Virtual Showcase. IEEEComputer Graphics & Applications, vol. 21, no.6,pp. 48-55, 2001.

Bimber, O., Gatesy, S.M., Witmer, L.M., Raskar, R., and Encarnação, L.M. Merging Fossil Specimens with Computer-Generated Information. IEEE Computer, September, pp. 45-50, 2002.

Bimber, O., Grundhöfer, A., Wetzstein, G., and Knödel S. Consistent Illumination within Optical See-Through Augmented Environments. To appear in proceedings of IEEE/ACM International Symposium on Augmented and Mixed Reality (ISMAR'03), 2003 (b).

HoloTouch, www.holotouch.biz

Matveyev, S. Göbel, M., Frolov, P. 2003. Laser Pointer Interaction with Hand Tremor Elimination, Proceedings of HCI'2003, Greece, 2003

Raskar, R. Welch, G., Low, K.L., and Bandyopadhyay, D. Shader Lamps: Animating real objects with image-based illumination. In Proc. Of Eurographics Rendering Workshop (EGRW’01), 2001.

Schmalstieg, D., Fuhrmann, A., Hesina, G.,Szalavari, Zs., Encarnação, L. M., Gervautz, M.,Purgathofer, W. The Studierstube Augmented Reality Project. PRESENCE - Teleoperators and Virtual Environments, vol. 11, no. 1, pp. 32-54, MIT Press.

VSML - An integration tool to bridge the various components of the VS software framework, http://www.studierstube.org/virtualshowcase/details/vsml.html, 2002.