DEPARTMENT OF INFORMATICSTECHNICAL UNIVERSITY OF MUNICH

Master’s Thesis in Informatics: Games Engineering

Enhancing visual guidance for small-FOVaugmented reality headsets

Christian Schnelzer

DEPARTMENT OF INFORMATICSTECHNICAL UNIVERSITY OF MUNICH

Master’s Thesis in Informatics: Games Engineering

Enhancing visual guidance for small-FOVaugmented reality headsets

Author: Christian SchnelzerSupervisor: Prof. Dr. Nassir NavabAdvisor: Felix BorkSubmission Date: 15.05.2018

I confirm that this master’s thesis in informatics: games engineering is my own workand I have documented all sources and material used.

Munich, 15.05.2018 Christian Schnelzer

Acknowledgments

I want to thank Prof. Dr. Nassir Navab, Dr. Ulrich Eck and especially my supervisorFelix Bork from the Chair for Computer Aided Medical Procedures & AugmentedReality at the TUM for their feedback and support. Furthermore, I want to thankthe iteratec GmbH, especially Daniel Stahr and Anton Brass for their collaboration,feedback, and support.

Abstract

Understanding, navigating, and performing goal-oriented actions in Mixed Reality(MR) environments is a challenging task and requires adequate information conveyanceabout the location of all virtual objects in a scene. Current Head-Mounted Displays(HMDs) have a limited field-of-view where augmented objects may be displayed.Furthermore, complex MR environments may be comprised of a large number ofobjects which can be distributed in the extended surrounding space of the user. Thisthesis presents two novel techniques for visually guiding the attention of users towardsout-of-view objects in HMD-based MR: the 3D Radar and the Mirror Ball. We evaluateour approaches against existing techniques during three different object collectionscenarios, which simulate real-world exploratory and goal-oriented visual search tasks.To better understand how the different visualizations guide the attention of users,we analyzed the head rotation data for all techniques and introduce a novel methodto evaluate and classify head rotation trajectories. Our findings provide supportingevidence that the type of visual guidance technique impacts the way users search forvirtual objects in MR.

iv

Contents

Acknowledgments iii

Abstract iv

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background Theory 42.1 Mixed Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1.2 Display Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1.3 Tracking and SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Visual Guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Microsoft Hololens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Related Work 103.1 Visualization Techniques for Out-of-View Objects . . . . . . . . . . . . . 10

3.1.1 2D Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1.2 3D Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 Evaluation of Out-of-View Visualization Techniques . . . . . . . . . . . . 153.2.1 Evaluation of 2D Techniques . . . . . . . . . . . . . . . . . . . . . 153.2.2 Evaluation of 3D Techniques . . . . . . . . . . . . . . . . . . . . . 17

3.3 Head Rotation Trajectory Analysis . . . . . . . . . . . . . . . . . . . . . . 19

4 Methods 214.1 Selecting Techniques for Comparison . . . . . . . . . . . . . . . . . . . . 214.2 Method Implementation Details . . . . . . . . . . . . . . . . . . . . . . . 22

4.2.1 3D Arrows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2.2 sidebARs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2.3 AroundPlot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

v

Contents

4.2.4 EyeSee360 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.2.5 Mirror Ball . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.2.6 3D Radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.4 Head Rotation Trajectory Classification . . . . . . . . . . . . . . . . . . . 25

5 User Study 315.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Task & Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.2.1 Calibration & Interactive Tutorial . . . . . . . . . . . . . . . . . . 315.2.2 Object Collection Scenarios . . . . . . . . . . . . . . . . . . . . . . 325.2.3 Post Experiment Surveys . . . . . . . . . . . . . . . . . . . . . . . 33

5.3 Object Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.4 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.5 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Results 366.1 Completion Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

6.1.1 Sequential Collection Scenario . . . . . . . . . . . . . . . . . . . . 366.1.2 Random Collection Scenario . . . . . . . . . . . . . . . . . . . . . 366.1.3 Specific Collection Scenario . . . . . . . . . . . . . . . . . . . . . . 376.1.4 Augmented Reality and Gaming Experience . . . . . . . . . . . . 37

6.2 Head Rotation Trajectory Analysis . . . . . . . . . . . . . . . . . . . . . . 376.2.1 Outlier Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406.2.2 Class Definitions & Thresholds . . . . . . . . . . . . . . . . . . . . 416.2.3 Sequential Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.2.4 Specific Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.2.5 System Usability Scale . . . . . . . . . . . . . . . . . . . . . . . . . 426.2.6 Mental Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

7 Discussion 45

8 Conclusion 47

List of Figures 49

List of Tables 51

Bibliography 52

vi

1 Introduction

1.1 Motivation

Human-computer interaction in Mixed Reality environments has been explored ina variety of studies over the last years. By combining the display of virtual contentalongside the physical world on the one hand with complex interaction paradigmson the other hand, novel applications and methods have emerged in various domainsthat leverage the power of interaction through augmented environments. Recenttechnological advances in head-mounted displays (HMD’s), as well as the increasingfocus on Mixed Reality by large technology companies, suggests that Mixed Reality ishere to stay and will evolve even more rapidly in the upcoming years.

1.2 Problem Statement

Mixed Reality (MR) environments present both real and virtual world objects simultane-ously on a single display [35]. One major advantage of such augmented environmentsis that the 3D location of virtual-world objects is commonly known. Therefore, efficienttechniques for guiding the attention of users towards such objects can be developed,which requires both tracking and detection methods for real-world objects. Providingthe user with positional information about all virtual objects enables quick explorationand navigation of MR environments as well as precise and goal-oriented actions.

One challenging task that remains an open research problem is how to conveyinformation about surrounding virtual objects to the user. MR environments canpotentially comprise a large number of virtual objects at various locations, makingit difficult to understand and navigate an augmented scene. This problem is furtherenhanced by the fact that current Head-Mounted Displays (HMDs) still have a limitedfield-of-view which does not resemble the one of the human visual system. Therefore,only a small portion of the virtual environment is visible and many virtual objects arelikely to be out-of-view. Several visualization techniques have been proposed in thepast to locate and guide attention towards such out-of-view objects in both mobile andHMD-based MR environments. However, none of the proposed methods universallysolves the problem. Drawbacks of existing methods include taking up large amounts of

1

1 Introduction

display real-estate, visual clutter, and occlusion issues for a large number of objects.Additionally, existing studies often lack a fundamental analysis on how participantsutilize a certain visual guidance technique for the task of localizing virtual objects.In our opinion, a crucial building block in developing such an understanding is theanalysis of head rotation trajectory data and the influence of different visualizationtechniques on this data.

1.3 Contributions

The contributions presented in this thesis are two-fold: first, we introduce two novelmethods for visualizing out-of-view objects in MR environments and evaluate them ina large user study. The first one is the Mirror Ball, which resembles a reflective sphereand displays distorted reflections of virtual objects according to their location in theenvironment. The second method we introduce is the 3D Radar, a technique that isoften used in commercial computer games, especially in space simulators. Similar totraditional top-down mini-maps and radars, virtual objects are substituted by a smallproxy icon on the radar. The position on the 2D radar plane reflects the horizontalangle from the user, while the vertical angle is encoded by orthogonal lines from the 2Dradar plane towards the object icon. We compared our proposed methods with currentstate-of-the-art techniques for visualizing out-of-view objects during a user study withtwenty-four participants. We measured task completion times in three different objectcollection scenarios that resemble real-world exploratory and goal-oriented visualsearch tasks. The 3D Radar was found to yield comparable results to state-of-the-artvisual guidance techniques, while simultaneously covering a much smaller portion ofthe field-of-view than comparable methods. The Mirror Ball faced several perceptualchallenges which resulted in higher mental effort scores and slower task completiontimes.

Our second contribution is a new area-based method for evaluating and classifyinghead rotation trajectory data. Our algorithm can be employed to discriminate betweentwo distinct object targeting approaches in HMD-based MR environments: a direct,one-way approach following the shortest path between a start and target orientationvs. an indirect, two-way approach which aligns the horizontal and vertical anglesequentially in an L-shaped fashion. Our results indicate that the 3D Radar encouragesthe latter class of trajectories, while users predominantly chose the direct, one-wayapproach for all other visualization methods.

2

1 Introduction

1.4 Outline

Chapter 2 provides an overview of background theory and related work. The conceptof MR and important technologies that enable MR experiences are explained andvisual guidance in our application context is defined. Chapter 3 presents relatedwork regarding visual guidance techniques in traditional as well as MR applications.Additionally, an overview of existing methods to analyze head rotation trajectoriesis given. In chapter 4 the application we developed to measure performance andhead trajectory data is described. Design and implementation of the visual guidancetechniques we want to compare are presented and our novel head rotation trajectoryanalysis approach is described. Chapter 5 depicts the conducted user study in detail.The results of our study are presented in chapter 6, followed by a discussion andinterpretation in chapter 7. Chapter 8 concludes the thesis.

3

2 Background Theory

2.1 Mixed Reality

In this section mixed reality (MR) is explained in greater detail. A definition for MRis given and the differences between augmented reality (AR), virtual reality (VR),physical reality and MR are highlighted. We continue to give a high-level overview oftechnologies that are essential for MR applications.

2.1.1 Definition

The term MR is best explained when viewed in relation to VR and physical reality. In1995 Milgram et al. introduce the Reality-Virtuality (RV) Continuum, which describesthe transition from physical reality to VR [36]. The RV Continuum can be illustrated bya one-dimensional line that is constrained on both sides as seen in Figure 2.1.

Figure 2.1: RV continuum illustrated as a one-dimensional line.

On one end of the continuum is the physical reality. In the physical reality, theenvironment is entirely real and strictly abides by the laws of physics. There are novirtual elements. On the other end of the continuum is the VR, where the environmentis consisting entirely of virtual objects and may or may not abide by the laws of physics.Everything on this continuum that is not entirely physical or virtual reality is referredto as MR. The transition from physical to virtual is continuous. The MR space is

4

2 Background Theory

then further divided into augmented reality (AR) and augmented virtuality (AV). InAV some degree of reality is added to an otherwise completely virtual environmentwhereas in AR the environment is entirely real and some degree of virtuality is addedto it.

One of the earliest and well-known definitions for AR is presented by Azuma [3]. Hedefines an AR system as a system that has three key features:

1. The system combines virtual and real elements.

2. The system is interactive in real time.

3. The system is registered in 3D.

While Milgram’s RV-continuum is useful to understand the relationship betweenthe different reality concepts, Azuma’s definition is more meaningful for real-worldAR applications. Since Azuma’s definition is independent of the degree of reality andvirtuality present in a system, his definition of AR corresponds more to Milgram’sdefinition of MR.

Today the terms AR and MR are often used as synonyms, while the term augmentedvirtuality is hardly used. In this thesis, the terms AR and MR are used as synonymsand both refer to Azuma’s definition of an AR system, independent of the degree ofreality and virtuality involved.

2.1.2 Display Types

Different display types can be used for MR applications. Zhou et al [56] differentiatebetween three different types of displays: see-through head-mounted displays (HMDs),handheld displays and projection based displays.

HMDs are placed on the user’s head with the display being directly in front of theuser’s eyes. Therefore they provide an egocentric view of the environment and theaugmentations. See through HMDs are divided into two categories: optical see-through(OST) and video see-through (VST). OST HMDs have semi-transparent displays thatallow the user to directly view the environment around them. Different technologiesare used to overlay the user’s view with virtual objects and augmentations. The displayon VST HMDs is opaque. They use a camera that is mounted to the HMD to capturethe user’s surroundings and display them on the opaque display. Augmentations canthen simply be displayed on the same display as the environment.

Handheld devices that have a camera can also be used for AR. Unlike HMDs theyprovide an exocentric view from the viewpoint of the camera. Handheld devices are apopular platform for AR because smartphones are highly available and non-intrusive.

5

2 Background Theory

Other handheld devices that can be used for AR applications include but are not limitedto notebooks and tablets.

Projection-based displays use projectors to display virtual content directly on realobjects and surfaces. Unlike other display types, projection-based displays actuallychange the look of an object for anyone that looks at it, not just for persons who arelooking at the object through some other device. This makes these displays suitable forapplications where several users need to be able to see the augmentations.

2.1.3 Tracking and SLAM

A key feature for many MR applications is to make virtual objects interact with thephysical environment to some extent [13, 28]. In order to make this interaction possible,the system needs to have information about the user’s environment. One way toprovide this information would be to provide a model of the user’s surrounding, forexample a CAD model, to the system. However, in most situations such a model is notavailable. In order to support interaction with the environment anyway, the MR systemis responsible for building a model of an unknown environment on the fly. A commonapproach to this problem that originated in robotics, but is also widely used in MRapplications, is simultaneous localization an mapping (SLAM) [19]. In the following,methods to keep track of objects in the environment by using different kinds of sensorsare introduced, a problem also known as tracking. The SLAM approach is then coveredin greater detail.

Tracking

In general, tracking describes the problem of identifying physical objects or simplifiedrepresentations of objects, such as geometrical primitives and keeping track of themover time. Different methods that use different sensors and devices for tracking exist.In the following, a short overview of tracking techniques commonly used in MRapplications is given, namely sensor-based tracking and vision-based tracking [56].

In sensor-based tracking different kinds of sensors, for example magnetic, acousticor inertial sensors, are used to get information about the environment and the user’smovement. Inertial sensors that are commonly used in smartphones and HMDs areaccelerometers and gyroscopes, which measure changes in acceleration and rotationrespectively. Rolland et al. [42] provide a more detailed overview of sensor-basedtracking.

For vision-based approaches, only cameras are used. Operations are then performedon the image data captured by the cameras in order to identify objects or more abstractfeatures. Zhou et al. [56] differentiate between three types of vision-based tracking:

6

2 Background Theory

marker-based tracking, feature-based tracking and model-based tracking. Marker-basedtracking uses physical markers that are placed on objects in the environment. Themarkers look similar to QR codes and are easy to find in the image data captured bya camera. The biggest disadvantage of marker-based tracking is the need of physicalmarkers in the environment. Feature-based tracking works without any physicalmarkers. Two-dimensional features, such as points, circles or object contours areextracted from the image data. In subsequent frames, the system tries to find the samefeatures and match them to the features found in the previous frames. Model-basedtracking methods internally use an explicit representation of the objects to be tracked,for example a CAD model. With the help of the model, the system tries to find theobject in the image data provided by the camera.

Hybrid methods often combine sensor-based and vision-based tracking for morerobustness.

SLAM

Fuentes-Pacheco et al. describe simultaneous localization and mapping (SLAM) asfollows: "SLAM [...] is the process whereby an entity (robot, vehicle or even a centralprocessing unit with sensor devices carried by a person) has the capacity for buildinga global map of the visited environment and, at the same time, utilizing this map todeduce its own location at any moment."[19] As the name and definition imply, SLAMconsists of two separate tasks: localization and mapping. Localization describes theproblem of determining the system’s position in the environment. Mapping is theprocess of creating a representation, often referred to as a map, of the environment.The two tasks are dependent on each other. In order to localize the system in theenvironment, a map of the environment is required and in order to build a map of theenvironment the position of the system has to be known. Therefore the localizationand mapping tasks need to be carried out in parallel on separate threads. The highavailability of multi-core processors even in mobile devices makes it possible to useSLAM in real-time applications.

To perform these tasks, systems utilize tracking techniques using different kindsof sensors. There are exteroceptive sensors like sonar, range lasers and cameras andproprioceptive sensors like accelerometers and gyroscopes. A special form of SLAMthat uses cameras as the only exteroceptive sensor is known as visual SLAM. Whenonly one camera is used in a visual SLAM system, it is also referred to as single cameraSLAM or MonoSLAM. MonoSLAM is a popular technique, probably due to the highavailability of single camera systems through smartphones.

There are multiple problems that a SLAM implementation has to solve. Salientfeature detection describes the task to extract meaningful features from image data. In

7

2 Background Theory

the image matching the system tries to find the corresponding features of one imagein another image. Since small errors in sensor measurements and approximations canlead to drift, the map has to be corrected when already visited areas are measuredagain. This process is called loop closure. Kidnapped robot describes the case, when tosystem loses tracking for some reason and has to identify an already visited area again.

Various solutions for SLAM have been proposed. Most techniques use a probabilisticfilter approach, with the Extended Kalman Filter (EKF) [45] and Factored Solution toSLAM (fastSLAM) [47] being the two most commonly used. Structure from motion isanother approach with the goal to compute the 3D structure of the environment andthe camera position from images [40]. Bio-inspired models take a different approach.They try to model the navigation techniques used by animals, for example rats [34].

2.2 Visual Guidance

Navigating augmented scenes is a complex task. Virtual objects can be placed in thethree-dimensional space around the user. Therefore it is likely that the user is notable to see objects that are not positioned in front of him. This problem is furtherenhanced by the fact that the field of view provided by state-of-the-art MR headsets issignificantly smaller than the human field of view. Some form of guidance is requiredin order to facilitate effective navigation in augmented scenes, locating so far unseenobjects and relocating already observed objects.

Guidance as a general concept is defined in the Cambridge Dictionary as "help andadvice about how to do something". In our case, how to navigate augmented scenesand locate potentially out-of-view objects in three-dimensional space. There are variousforms of guidance utilizing different modalities. Acoustic guidance uses auditory cuesin order to convey information to the user. In our case, using a sound that originatesat the location of the target object could help the user to locate it. Verbal guidanceuses more explicit auditory cues in the form of words and sentences, telling the userwhat to do. We could use this form of guidance to tell the user in which directionhe has to turn in order to find the target object. Manual guidance involves physicalsupport by an instructor, for example turning the user’s head in the direction of thetarget object. Mechanical Guidance involves the use of devices. In our scenario, wecould ,for example, use an armband that vibrates depending on whether the user is farfrom, or close to the target location. In this thesis, we focus on visual guidance, wherevisual cues are used in order to convey information. An example of visual guidancewould be arrows that point towards the location of the target object.

We display virtual objects on the screen of AR headsets in order to guide the user.Since MR devices inherently require a display to show augmentations, this seems like

8

2 Background Theory

a natural way to convey additional information to the user and is likely to apply todifferent MR applications as well. We leave using different kinds of guidance andhybrid approaches for future work.

2.3 Microsoft Hololens

To provide MR experiences to the user, a multitude of technologies including sensors,displays, tracking and SLAM implementations are combined in MR devices. Thedevice used by all participants in the user study in this thesis is the Microsoft HololensHMD [32]. Detailed information about the hardware specifications can be found onMicrosoft’s website [33]. Multiple sensors in the Hololens including inertial sensors,cameras, and depth sensors are used to implement one of the best working SLAMalgorithms in currently available MR devices. An advantage of the Hololens comparedto other MR devices is that it is completely untethered and therefore provides greatmobility during usage. However, a problem that is shared to some degree by all MRdevices and that Hololens suffers heavily from, is the small field of view. The Hololensfeatures a diagonal field of view of only approximately 35 degrees. Therefore virtualobjects that are not located directly in front of the user, are likely to be out of view.Even if he has seen an object before, he might not remember it’s location. In order tobring the unseen virtual objects to the user’s attention and make searching for alreadyseen objects less tedious, visual guidance techniques can be used. Cues indicating thelocation of out-of-view objects can be displayed in screen space on the display of theHololens so that they are always in view. Multiple approaches to providing visualguidance in traditional as well as MR applications have been developed. However,none of the proposed solutions solves the problem universally. An overview of thesemethods is given in the next chapter.

9

3 Related Work

3.1 Visualization Techniques for Out-of-View Objects

Existing techniques for visualizing out-of-view objects in MR environments can begrouped into the following three categories: Overview & detail describes a category ofapproaches where two separate windows are used, which are usually displayed ontop of each other [39]. The overview window typically conveys information aboutthe user’s surroundings, e.g. in form of a map. The detail window provides specificinformation about a local sub-space, typically what the user is currently looking at.In Focus & Context approaches there is only one window which provides a distortedview of the user’s surrounding, e.g. using a fisheye projection [12, 23]. The transitionbetween the non-distorted focus area, typically in the middle of the window, and thedistorted context area is smooth. Contextual Views [23, 26] as the third category ofvisualization approaches overlay the detail window with abstract indicators, such asarrows, that provide information about the location of out-of-view objects.

Out-of-view object visualization techniques can be further classified based on thedimensionality of their information encoding. While 3D techniques provide informationthat enables the user to infer the location of virtual objects in 3D space, information con-veyance for 2D techniques is limited to a left-or-right and above-or-below discriminationand lacks information about the depth of objects relative to the user.

3.1.1 2D Techniques

Numerous 2D techniques have been proposed in the past, most of which were designedspecifically for desktop or mobile applications. Figure 3.1 shows examples of thereviewed techniques. Zellweger et al. [55] propose City Lights. This technique is usedto facilitate navigation between windows and nested window spaces on a monitor. Off-screen windows are indicated by thick, colored lines that are displayed on the bordersof on-screen windows via orthogonal projection. The position of the line indicatesthe direction in which the off-screen windows are located. Distance and size of theoff-screen windows can be encoded by the color and length of the lines respectively.

Halo, introduced by Baudisch et al. [6], is a visualization technique that uses circlesto visualize off-screen objects. In their work they use it to display points of interest on a

10

3 Related Work

map, however, the technique can be used to display arbitrary off-screen objects. Objectsthat are not on screen are used as the center of a circle. The radius of that circle is setto be big enough, so that the circle intersects with the screen, resulting in an arc that isdisplayed at the border of the screen. From the size and curvature of this arc, the usercan estimate the center of the circle and therefore the object location. A bigger circlewith a smaller curvature indicates a far away object, whereas a smaller circle with ahigher curvature indicates a nearby object. However, both City Lights and Halo aredifficult to interpret in the presence of multiple objects.

With the goal to improve overlapping issues for multiple objects Gustafson et al.introduce Wedge as an improvement over Halo [23]. Off-screen objects are representedby so-called wedges, which are basically triangles. An off-screen object’s location servesas the tip of the triangle. Two legs of equal length are connected to the tip. The legs arelong enough so that they both intersect with the screen. The base connects the endsof the two legs, completing the wedge. The parts of the legs that are shown on-screenallow the user to mentally extrapolate the legs beyond the screen and determine theobject’s position. The base identifies two legs as belonging to the same object, which isuseful when there are multiple wedges on screen. The technique can be used to furtherimprove overlap issues by adjusting three parameters of the wedges: Rotation describesthe rotation of the wedge around the target object’s location. Aperture describes theangle between the two legs of the wedge. Intrusion describes how much of the two legsis visible on-screen.

EdgeRadar [22] makes use of the screen edges by creating a small overlay area atall four edges of the screen. This divides the edge area into eight sections: left, right,top, bottom and the four corners where two sections overlap. Off-screen objects arerepresented by miniature icons of the objects, which are placed inside of the edgesections. Objects that are to the left of the screen, but neither above or below, havetheir corresponding icon located in the left section; objects that are to the left of thescreen and above the screen, have their corresponding icon located in the top left cornersection; and so forth. The position of an icon within an area further indicates the moredetailed location of the corresponding off-screen object.

Stretched Arrows and Scaled Arrows are both arrow based methods [11]. Whilearrows at the edges of the screen are used in both approaches to indicate the directiontowards off-screen objects, the methods differ in the way they encode the distance tothe target. The arrows are scaled inverse-linearly proportional to the distance to thedistance to the target object in both techniques. However, for Stretched Arrows onlythe length of the arrows is scaled, while for Scaled Arrows the arrows are scaled inall dimensions. So short or small arrows indicate far away objects, while long or bigarrows indicate nearby objects.

Siu and Herskovic picked up on these ideas and introduced the sidebARs technique,

11

3 Related Work

which displays vertical bars at the left and right edges of the screen [44]. Small proxyicons representing the out-of-view objects are placed inside those bars and their positionwithin the bars indicates the position of the objects. Objects whose icons are locatedin the left bar can be found faster by rotating left than by rotating right. The distancetowards off-screen objects is given by a number next to the icon.

In case multiple objects are positioned in close proximity to each other, occlusionsand edge clutter are two common disadvantages of the previous techniques.

a) b)

d) g)e) f)

Figure 3.1: Example pictures of different 2D off-screen visualization techniques. a) CityLights; b) Halo; c) Wedge; d) Edge Radar; e) Scaled Arrows; f) StretchedArrows; g) SidebARs

3.1.2 3D Techniques

In HMD-based MR, especially for tasks such as exploring, navigating, and achievingcertain goals inside an augmented environment, 2D techniques for visualizing out-of-view objects are insufficient. Therefore, a number of techniques have been introducedthat extend the information conveyance about the location of such objects to 3D space.Figure 3.2 shows examples of the reviewed techniques.

Trapp et al. transfer the concept of the 2D Halo technique to three-dimensional space[50]. They introduce two off-screen visualization techniques: 3D Halo and 3D Halo

12

3 Related Work

Projection. In 3D Halo out-of-view objects are represented by two rings, that lie inparallel planes, forming a cylinder with transparent walls around the object. The radiiof the rings are equal and are set big enough to intersect with the screen. From thepartially visible cylinder, the user can infer the center, and therefore the location ofthe off-screen object. 3D Halo projection first projects the off-screen target locationinto a plane that is parallel to the near plane of the camera. This way, the problemis basically reduced from 3D to 2D. At this point, the 2D Halo technique is appliedstraightforwardly to points on the projection plane, except for points that are behindthe user, which are represented as straight lines on the edges of the screen.

Another extension from 2D to 3D is employed by the 3D Arrows technique [43].Three-dimensional arrows that directly point at off-screen objects are used. The arrowsare arranged on an invisible circle that is located in the lower part of the screen. Thering is slightly tilted towards the user to reduce occlusion. When the user rotates, thearrows move along the circle, resembling a compass with multiple needles.

For these techniques, the same drawbacks which are present in the 2D case (clutterand occlusions) also apply in 3D.

Parafrustum [48] focuses not only on finding a specific target object but also to viewthe target from a specific viewpoint. To achieve this two volumes are defined: thehead volume and the tail volume. The head volume contains the set of acceptableviewing positions. The tail volume contains a set of points around the target object.The user’s viewpoint is considered to be correct if it is inside the head volume and thetail volume is completely visible. Two different approaches are used to guide the usertowards the correct viewpoint. The first one makes the head and tail volumes moreand more transparent as the user gets closer to the correct position. The second oneuses radar-like visualizations to guide the user towards the correct viewpoint.

Biocca et al. introduce the Attention Funnel [8]. This technique calculates a smooth,invisible path from the user’s location to the target location. Along this path equidistant,rectangle-like proxies are placed repeatedly, normal to the path. Together these proxiesform a tunnel-like structure that guides the user towards the target object.

Renner and Pfeiffer introduce the SWAVE technique [41]. It is inspired by thebehavior of waves when for example throwing a rock into a lake. Waves emerge inconcentric circles from the location where the rock hits the water. The same principle isused in the inverse direction so that concentric circles or waves do not emerge from, butcenter in on the target location. The waves are either visualized on a sphere around theuser or as a screen space effect. Either way, their visualization is displayed across thewhole display of the AR device. Eye-tracking techniques to locate an object once it is onscreen are explored as well and can be used to enhance existing off-screen visualizationmethods.

Matsuzoe et al. propose a technique that uses flickering icons at the edges of the

13

3 Related Work

screen to visualize off-screen objects [30]. The position of the icon indicates the directiontowards the target object. The direction is calculated using the intersection of the linethat connects the center of the screen with the target location and the edges of thescreen. The amount by which the target moves when vibrating and the frequency of thevibration indicate the distance to the target. Larger movements with higher frequencyindicate a far away object, whereas small movements with lower frequencies indicatenearby objects.

However, most of these techniques perform best for single objects and would sufferfrom severe clutter in case of more complex MR environments.

At first sight, AroundPlot [26] looks very similar to the EdgeRadar technique. Around-Plot also uses a rectangle to mark the focus area. The context area, consisting of thearea outside of the rectangle up to the edges of the screen, is however used to encodethe entire 3D space around the user. Like for the EdgeRadar the context area can bedivided into 8 sectors: the four sectors directly to the left, right, top and bottom ofthe focus area and the 4 corners. Out-of-view objects are represented by dots locatedin the context area. The position of the dots indicates how far the user has to turnin order to be able to see the target object. The sections left and right of the focusarea encode the space 180 degrees to the left and right of the user, which the user canpotentially see by only turning left and right. The sections above and below the focusarea encode the space a little more than 90 degrees above and below the user, whichhe can potentially see by only looking up or down. The space that requires the userto change his orientation horizontally and vertically is encoded in the corners of thecontext area. A dynamic magnification mechanism that enlarges the context area inthe direction that the user is rotating to is also implemented in order to increase theresolution of the encoding in areas of interest.

This technique suffers from the corner-density problem, as a large 3D space isencoded in the areas close to the four corners of the rectangle.

Gruenefeld et al. introduce a similar approach with EyeSee360 [20, 21]. An ellipse isused that spans across the entire screen in addition to a second smaller ellipse in themiddle of the screen, which marks the focus area. The area between the bigger andthe smaller ellipse marks the context area, which is used to indicate the position ofout-of-view objects. These objects are represented by dots. The location of the dotsencodes the position of the corresponding object. The horizontal position indicates thehorizontal angle from the user’s current viewing direction to the out-of-view objectand the vertical position indicates the vertical angle respectively. The ellipse encodesthe space 180 degrees to the left and right of the user and 90 degrees above andbelow the user, therefore covering the space 360 degrees around him. To facilitatedirection estimation, helplines to indicate certain angle values, such as zero degreesand optionally 45, 90 and 135 degrees, are displayed. Distance towards out-of-view

14

3 Related Work

objects is indicated either by the size of the dots, the color of the dots or a combinationof both. For an implementation on the Google Cardboard, the inner ellipse is replacedby a rectangle, because a device’s screen, unlike the human field of view, is usuallyrectangular.

EyeSee360 takes up a large portion of the field-of-view, which potentially degradesthe perception of real-world objects. Additionally, the resolution of the horizontalencoding decreases for objects that are located far above or below the user because theellipse gets narrower in those areas.

Using a Mirror Ball to visualize objects that are not covered by the field-of-view hasbeen used previously in traditional 3D modeling applications by McCrae et al. [31].In their approach, objects are clustered based on their distance to the ball, which isdivided into separate areas, similar to a Voronoi diagram. Each area of the ball reflectsa group of objects that have a similar distance to the ball and are close to each other.This way, all objects appear at about the same size on the ball, independent of theirdistance. A drawback of this method is that the distance information between theobjects and the mirror ball is lost. In order to address this issue different techniquesthat place cones on top of the ball are developed. A cone is placed on each area of theball, which points in the direction of the objects that are visible in the respective area.The length of the cone indicates the distance towards the objects.

The 3D Radar is a minimap-like visualization technique that is heavily inspired bycommercial computer games. Especially space simulator games such as Elite: Danger-ous[15], Eve: Valkyrie[17] and Star Citizen[46] employ this technique for providing theplayer with an overview of the 3D space around him. It extends the traditional 2D ortop-down radar with the ability to visualize objects anywhere in 3D space. While radarvisualizations have been proven to be useful in 2D applications [10], to the best of ourknowledge the extension to 3D space has not been evaluated in the literature yet.

3.2 Evaluation of Out-of-View Visualization Techniques

Many of the previously introduced techniques feature an evaluation of the newlyintroduced method against existing ones. Common metrics used for the evaluationare completion time, error rate and accuracy. Accuracy is sometimes further split intodirection and distance estimation. In this section, we present the evaluation methodsused in previous approaches.

3.2.1 Evaluation of 2D Techniques

Baudisch et al. evaluate their Halo technique against a 2D Arrow approach [6]. Comple-tion time, error rate and user preference in different tasks are compared. The definition

15

3 Related Work

a) b) c)

d) e) f)

g) h) i)

j) k)

Figure 3.2: Example pictures of different 3D off-screen visualization techniques. a) 3DHalo; b) 3D Halo Projection; c) 3D Arrows; d) Parafrustum; e) AttentionFunnel; f) SWAVE; g) Vibrating icons; h) Aroundplot; i) EyeSee360; j) MirrorBall; k) 3D Radar

of an error is different for each task. User preference is captured by asking the usersexplicitly whether they prefer one method above another. In all of the four differenttasks, the users are presented with a static image of a map. Off-screen items are eitherrepresented by circles or arrows, depending on which visualization method is used. Inthe Locate task users have to estimate the exact position of off-screen objects based onthe static image. The error in this task is the Euclidean distance between the estimatedand the actual position of the off-screen object. The Closest task requires the usersto select the closest off-screen object to a specified on-screen object. Choosing thewrong off-screen object is classified as an error. In the Traverse task users have to selectoff-screen objects in the order that connects them with the shortest path. The error isthe difference between the selected and the optimal path. The last task Avoid requires

16

3 Related Work

the user pick an off-screen target location so that specific on- and off-screen points areavoided. Like in the closest task, selecting a wrong location is classified as an error.

In their paper introducing Wedge, Gustafson et al. compare the technique to Halo[23]. The same metrics and tasks are used except for the Traverse task. Performanceis measured during a scarce and a dense scenario, featuring five and ten to twentyoff-screen objects respectively. This way they want to capture in how far a largernumber of off-screen objects leads to occlusion for the different techniques.

Stretched and Scaled Arrows are also compared to Halo [11]. Performance is mea-sured using the same metrics, however in addition to the Closest and Locate task, twoadditional tasks are introduced. The Estimate task requires the user to pick the twooff-screen objects that are closest to each other. A wrong selection is classified as anerror. In the Order task, the users have to order the off-screen objects with increasingdistance to the center of the screen. Wrong orders are recorded as an error. Thedifferent tasks are evaluated in two scenarios, including five and eight off-screen objectsrespectively. None of the aforementioned evaluations mentions how exactly off-screenobjects are positioned, other than that they are randomly distributed.

The EdgeRadar technique is also compared to Halo, however, the evaluation specifi-cally aims to capture the performance of users when tracking moving off-screen objects[22]. Multiple objects are placed randomly on and off-screen. At the start of the task,some objects are highlighted as target objects for two seconds. Then the objects startto move for ten seconds. When objects leave the screen, they are visualized as dots orcircles, depending on the visualization technique. After another ten seconds, a randomobject is highlighted and the user has to determine whether the highlighted object is atarget object or not. A wrong decision by the user is classified as an error. No explicitinformation is given about the movement pattern of the objects.

Siu and Herskovic evaluate their SidebARs technique by using heuristic questionsand interviews with potential users of their application [44].

3.2.2 Evaluation of 3D Techniques

3D Halo and 3D Halo Projection [50] are only evaluated in terms of system performancelike frames per seconds. User performance is not explicitly evaluated in this work.

Schinke et al. [43] evaluate their 3D Arrow technique against a top-down two-dimensional radar. Completion time, user preference on a six-point scale and accuracyare measured in a user study consisting of two different tasks. In the first task, fourobjects are randomly positioned around the user. The objective is to read the labelsfor the objects out loud. Accuracy is not measured for the first task. An object’s labelis displayed, when the object is in the center of the display. In the second task, usershave to memorize four out-of-view objects. After memorizing the objects, the device is

17

3 Related Work

taken away and users have to estimate the location of the off-screen objects. Accuracyis determined by the number of correctly estimated objects and the difference betweenthe estimated and the actual location of the objects.

Parafrustum [48] is only evaluated against different versions of Parafrustum. For theevaluation, two different Parafrustum approaches are used to guide the user towardsthe target. Users have to correctly align their position and orientation towards six targetlocations per approach. For each target location, users have to align for 9 differentcombinations of head and tail volume shapes.

The evaluation for the Attention Funnel [8] compares the technique against twomethods using either visual highlighting or verbal cues. For the visual highlights, avirtual bounding box is displayed on the target object in order to make it stand out.The verbal cues are pre-recorded verbal descriptions of the target object. The task is tofind and select one of 48 objects. A detailed description of the object placement is notgiven. Completion time, error rates and mental workload are measured. Error rates arecalculated based on the number of wrongly selected objects and the mental workloadis measured using the NASA Task Load Index (NASA TLX) [24].

Renner and Pfeiffer [41] evaluate their SWAVE technique against a static display ofthe target objects, an arrow-based approach, and attention funnel. Unlike the othertechniques that use actual augmented reality devices for their evaluations, they usesimulated augmented reality in virtual reality. The users wear a virtual reality headsetand a virtual scene is displayed. To simulate augmented reality they place a virtualscreen in front of the user, that follows the user when he rotates, resembling the displayof an actual augmented reality headset. Seven target objects are randomly positionedon 48 predefined positions. The remaining 41 positions are filled with different objects.The target objects are positioned in a way so that the full path length, consisting ofthe sum of the angles between the target objects, is between 500 and 600 degrees. Theobjects are not randomly distributed around the user. Instead, half of the objects arepositioned on a table and the other half is located on a shelf. The user’s objective isto build a birdhouse by selecting the seven target objects in the correct order. Thesubject’s subjective impressions concerning speed, accuracy, learnability, and usefulnessare measured. Objective measurements are taken, capturing completion times, headrotation in terms of angles traveled and the ratio of time spent focusing on the ARscreen to time spent focusing on the scene.

The vibrating icon technique proposed by Matsuzoe et al. [30] is evaluated againstarrow-based approaches and an approach using text-based instructions such as "Moveright X degrees and up Y degrees". In addition to locating out-of-view objects, usershave to memorize random six- or eight-digit numbers. First, the random number isdisplayed. After two seconds the visual guidance technique is displayed and the userhas to locate the target object. No further information about the positioning of off-screen

18

3 Related Work

objects is given. When the user has located the target object, he has to write downthe number he memorized at the start of the task. Task completion times, errors inmemorizing the random numbers and user satisfaction are measured. User satisfactionis captured using a questionnaire including questions from the System Usability Scale(SUS) [9].

Aroundplot [26] is compared to a 2D radar and a 3D arrow approach. In their userstudy users have to complete two tasks. In the first task, one of multiple out-of-viewobjects is highlighted. After two seconds the highlight disappears and the user needs tofind the no-longer-highlighted target object. The second task is very similar. The onlydifference is that the target stays highlighted until the user has found it. Out-of-viewobjects are randomly placed around the user with distances to the user between 1 and100 meters. Completion time, error rate and subjective task load are measured. Errorrate is determined by the number of objects that were not found within a certain timein the first task. Subjective task load is measured using the NASA Task Load Index.

Gruenefeld et al. compare their EyeSee360 approach against adapted versions of 2DArrows, Halo and Wedge [20]. One task is used to measure direction estimation and asecond task is used to measure distance estimation and completion time. In the firsttask out-of-view object are positioned randomly around the user. The visualizationtechnique, however, is only visible in front of the user’s starting pose and does notmove along when the user rotates. Off-screen objects are then highlighted one at atime. Using the static visualization technique, the user has to estimate the off-screenlocation of the objects by selecting the estimated location with a cursor. The directionaccuracy is measured by the angular difference between the estimated position andthe actual position of the target object. In the Second task, out-of-view objects are alsodistributed around the user and highlighted one at a time. First, the user has to estimatethe distance of the highlighted object and then use the no-longer-static visualizationtechnique to locate the object. Distance accuracy is measured using the differencesbetween estimated and actual distance to the target object. Usability is captured usingSystem Usability Scale and NASA Task Load Index questionnaires.

3.3 Head Rotation Trajectory Analysis

In order to get a better understanding of how the different visualizations are utilized forthe task of targeting virtual objects, we were interested in analyzing the head rotationtrajectories. In a traditional sense, a trajectory is defined as the path of a movingobject, consisting of a set of consecutive positions in space as a function of time. In ourscenario, we also use the notion that a given head rotation path defines a trajectoryby transforming the data into spherical coordinates and referring to the individual

19

3 Related Work

positions of the trajectory as timestamped tuples of both the horizontal and verticalangle.

The analysis of trajectory data is a broad research topic with applications in manydifferent domains, such as activity recognition, surveillance security, anomaly detectionor traffic monitoring. A recent survey of trajectory analysis techniques, covering bothsupervised and unsupervised techniques for trajectory clustering and classification,is provided by Bian et al. [7] To compare two trajectories, multiple similarity mea-sures with varying complexity have been proposed in the past, most of which aredistance-based measures. The most common metrics include the Euclidean distance[27], Dynamic Time Warping [37], Least Common Sub-Sequences [51], the Fréchetdistance [1], and Edit distances [14]. Anagnostopoulos et al. introduced the conceptof Minimum Bounding Rectangles (MBRs) to approximate the distance calculationsfor sub-trajectories [2]. A comparative study on the effectiveness of some of thesemeasures is provided by Wang et al. [52]. Distance-based measures are a powerful toolfor comparing similar trajectories over time, but they define only a global measure anddon’t take into account, that trajectories which have the same distance measure candescribe very different targeting approaches towards a virtual object. Region-basedmeasures split the domain into a grid structure and try to classify trajectories basedon the identification of homogeneous grid cells, which contain only a certain class oftrajectories [29]. Such region-based approaches can be very robust for classes of trajec-tories that are very similar, but their performance decreases if there is a large variabilityin the trajectory data, as expected for our head rotations. Other comparison metricsinclude curvature [18], significant changing points [4, 5], and discontinuities [16]. Forour head rotation analysis, we were especially interested in discriminating betweentwo distinct types of trajectories: i) those that correspond to a direct, one-way targetingapproach, which is characterized by a straight line between a starting head rotationand the orientation of the target point in spherical coordinates; and ii) trajectories thatdescribe an indirect, two-way targeting approach, which is defined by a (potentiallyrotated) L-shaped path in spherical coordinates. We were therefore able to define theoptimal trajectories in both cases and applied an area-based trajectory classificationapproach, which is explained in more detail in section 4.4.

20

4 Methods

We want to compare the performance of our two proposed visualization techniquesto the most promising approaches from the literature in the context of HMD-basedMR environments. For this purpose, we chose three different object collection tasksthat reflect real-world exploratory and goal-oriented search scenarios. We implementedan application for the Microsoft HoloLens, where participants were asked to collect anumber of virtual objects, positioned around them in 3D space and outside of theirfield-of-view, as fast as possible for all three scenarios. The different visual guidancetechniques were used to help participants find the virtual objects. In this section, wepresent a description of the process of selecting the set of visualization techniqueswhich was included in our comparison. Furthermore, we provide implementationdetails for all techniques as well as a detailed description of our head rotation trajectoryclassification algorithm.

4.1 Selecting Techniques for Comparison

Since we want our subjects to collect multiple out-of-view objects during the experi-mental user study, Parafrustum and Attention Funnel are not applicable due to theirlimitation of only supporting the attention guidance for a single object. The sameapplies to SWAVE and the two screen flickering approaches [30, 41]. We also discard3D Halo and 3D Halo Projection since they suffer from severe visual clutter in thepresence of multiple objects [50, 20]. EyeSee360 was developed with EdgeRadar as aninspiration and has been shown to outperform Halo, 2D Arrows and Wedge. Thus,we only include the superior EyeSee360 technique in our comparison and discard theother 2D methods. From the remaining 3D visualizations, we include both AroundPlotand 3D Arrows. Additionally, we include a slightly modified version of the sidebARstechnique, which allows the encoding of locations in 3D space by horizontally movingthe proxy icons according to their location. To the best of our knowledge, there existsno comparison between these techniques, except for AroundPlot and 3D Arrows, whichhave been shown to perform comparably for a small number of objects [26].

21

4 Methods

4.2 Method Implementation Details

All six visualization techniques were implemented as part of a Unity 3D application torun on the HoloLens. In case modifications were made to the original implementations,they are highlighted and motivated in the following paragraphs. Screenshots from ourapplication

4.2.1 3D Arrows

We implemented the 3D Arrows visualization similar to previous works [43]. All arrowsare placed around a sphere that is positioned in front of the user. Each arrow points inthe direction of a virtual object, with the length of the arrow scaled according to thedistance of the object. The only difference to the original implementation is that wecolor the arrows based on the color of the target objects.

4.2.2 sidebARs

In the original 2D sidebARs method, the position of small proxy icons inside the twovertical bars at the edges of the screen indicates where an object is placed relative to thecurrent position of the user [44]. While the vertical position inside the bars indicateswhether an object is above or below the user, an object can only be placed either insidethe left or right bar. In our implementation, we also vary the horizontal position ofproxy icons inside the left and right sidebar to indicate the horizontal angle of a targetobject. Additionally, we removed the numeric distance indicator which is overlaid ontop of the proxy icons in the original method.

4.2.3 AroundPlot

Our implementation of AroundPlot is very similar to the original method [26]. Sphericalcoordinates are mapped to orthogonal fisheye coordinates, such that the visible field-of-view corresponds to a rectangular area on the screen. Virtual objects inside theMR scene are indicated as colored dots positioned around the sides of the rectangleaccording to their horizontal and vertical angle. Compared to the original method,we exclude the dynamic magnification feature, which magnifies the rectangle in thedirection the user is moving. Instead, the size of the rectangle is fixed and covers almostthe same area as the field-of-view of the HoloLens.

22

4 Methods

4.2.4 EyeSee360

For our EyeSee360 implementation, we used a version with helplines and a rectangularinner window that the authors also proposed for their implementation on GoogleCardboard [20]. However, in our implementation, the dots representing out-of-viewobjects have the same color as the objects that they correspond to as opposed to distance-encoding colors in one variant of the original method. Dots positioned inside the innerrectangle indicate objects which are covered by the field-of-view. All other objects arepositioned according to their horizontal and vertical angle inside the ellipse. This way,dots on the left and right edges of the ellipse correspond to objects directly behind theuser, while dots on the upper and lower edges refer to objects directly above or belowthe user.

4.2.5 Mirror Ball

The implementation of the Mirror Ball technique is aimed at mimicking the natureof a real-world spherical mirror. Unlike McCrea et al., [31] who use a mirror ball fornavigation purposes in a traditional virtual 3D environment, we do not divide thesphere into different regions that correspond to clusters of objects at different distances.We position a virtual sphere in front of the user that has a reflective shader attachedto it. In order to provide the user with a better sense of depth, we employ a cubemap that has a grid texture on each side as a skybox and reflects it on the sphere.Similar to a real-world spherical mirror, the reflections visible on our Mirror Ball of allvirtual objects as well as the skybox are spherically distorted. These distortions providepositional hints to the user.

4.2.6 3D Radar

The design and implementation of our 3D Radar are heavily inspired by commercialvideo games. The 3D Radar consists of multiple concentric circles that are placed infront of the user and tilted slightly in order to prevent perceptual occlusion issues.A small triangle is placed in the center of the radar to indicate the users’ position,while a forward-facing cone represents the field-of-view. Every virtual object insidethe MR scene is depicted on the radar as a combination of a proxy icon and a circlecorresponding to the objects’ projection in the 2D radar plane. Similar to traditional2D (top-down) radars, the projection indicates the horizontal angle between the targetobject and the user. To provide the user with additional information about the verticalangle, proxy icons in the form of colored triangles located directly above or belowthe projection circles are used. To quickly convey information about correspondencesbetween proxy icons and their projection circle, a line connecting the two is employed.

23

4 Methods

The color of both the line and the projection circle depends on whether the virtualobject is located above or below the user. Figure 4.1 illustrates how the position ofoff-screen objects is encoded on the 3D radar.

4.3 Challenges

Many of the implemented techniques require the measurement of the horizontal andvertical angle between the user’s viewing direction vector and the vector from the userto the target object. Measuring the horizontal angle between the two vectors around theglobal y-axis works as expected with the functions provided by Unity’s Vector3 class.Measuring the vertical angle between the two vectors around the global x-axis, however,yields inconsistent results when the is rotating. Therefore we rotate the viewing vectorand the vector between the user’s location and the target object into the y-z-plane beforemeasuring the vertical angle between the two vectors around the global x-axis. Mostof the implemented techniques encode 180 degrees on both sides in horizontal space,but only 90 degrees up and down for the vertical space. The SignedAngle functionprovided by Unity’s Vector3 class returns the smallest angle between two vector in therange from -180 to 180 degrees. This angle can be mapped as is to our horizontal angle.The vertical angle, however, needs to be adjusted so that it starts decreasing if it exceeds90 degrees until it hits 180 degrees and vice versa for when it falls below -90 degrees.

In our EyeSee360 implementation, we use a simple texture that is scaled to the screensize for the ellipses and helplines. When positioning proxies in the ellipse we haveto take the width of the ellipse at the different vertical angles into account, since theproxies must not be displayed outside of the ellipse. For example, an object that islocated 180 degrees to the left of the user and zero degrees above and below the user,will have it’s proxy further on the left of the screen than an object that is located 180degrees to the left of the user and 45 degrees above or below the user. Since we simplyuse a texture for the ellipse and the helplines, the width of the ellipse at different heightscannot be easily calculated. Therefore a bezier curve is manually fitted to one-quarterof the ellipse. Since the ellipse is symmetrical both horizontally and vertically, a quarterof the ellipse is sufficient. At the start of the scene, we sample the bezier curve at100 equidistant points and store the 3D points along the curve in a look-up table. Atruntime, we determine the y-position of the dot and perform a binary search in ourlook-up table for the same y-value. We perform bilinear interpolation between the twoclosest neighbors and return the corresponding x-value. this x-value is then used todetermine the width of the ellipse with respect to the screen width. The horizontalangle is then used to determine the horizontal position within the appropriately scaledwidth.

24

4 Methods

Another challenge is that objects, especially thin lines and text, that are fixed inscreen space, cause severe tearing while moving. For example, when rotating quickly,the RGB channels that form a certain color can be seen separately. This effect wasenhanced for the mirror ball, since calculating the reflections of the objects and the ismore computationally expensive compared to the other techniques. Unfortunately, wedid not find a satisfying solution to this problem. While the effect was clearly noticeableall of the methods are still usable.

4.4 Head Rotation Trajectory Classification

Our analysis for comparing the head rotation trajectories of the different visualizationsis formulated as a classification problem. Our approach is driven by the hypothesis,that there are two very distinct ways of targeting a virtual object: the first one canbe described as a direct, one-way targeting approach (TOneWay), where the trajectorymatches the horizontal and vertical angle of the target object simultaneously. Thesecond way is an indirect, two-way targeting approach (TTwoWay), such that the trajectorymatches the two angles sequentially in an L-shaped fashion, i.e. the horizontal angleof the target object is matched first, then the vertical one (or vice versa). In sphericalcoordinates, a perfect one-way targeting approach towards a target point t is givenby the straight line lt connecting both the target point and the starting point s (i.e.initial head rotation). On the other hand, a perfect two-way approach is given by twoconnected straight lines forming the two legs of a right triangle defined by s, t, andthe point on the x-axis q = (tx, 0) (or alternatively the point on the y-axis q′ = (0, ty)).In this way, the first line matches the horizontal angle of t, while the second linesubsequently matches the vertical angle (or vice versa). Figure 4.3 illustrates these twotargeting approaches in the first quadrant. The blue trajectory resembles a one-waytargeting approach closely following the line between the origin and tred, while thegreen trajectory resembles a two-way targeting approach.

A given head rotation trajectory can be described as a polygonal path V, consistingof a set of vertices vi which correspond to discrete angular measurements, such thatV = {s, v1, v2, ..., t}. Our classification approach for such polygonal paths V is based onthe hypothesis, that the previous two object targeting approaches TOneWay and TTwoWaycan be discriminated based on the area of the region enclosed by lt and V. Assumingthat the starting point s is translated to the origin, a one-way approach is characterizedby an area AV ≈ 0, while for a two-way approach AV ≈

ARt2 , with ARt as the area of

the rectangle Rt defined by the origin and t. To avoid misclassifications of polygonalpaths with large portions of their enclosing area AV lying outside of Rt, we furthersplit AV into the two parts Ain

V and AoutV , such that AV = Ain

V + AoutV and Ain

V = V ∩ Rt.

25

4 Methods

Two percentage thresholds α and β for AinV and Aout

V respectively can be defined toallow for some marginals. The final trajectory classification with two more classes forunclassifiable trajectories can now be defined as follows:

f (V, t) =

TOneWay, for Ain

V < α, AoutV < β

TTwoWay, for AinV > 100− α, Aout

V < β

TInBetween, for 100− α ≥ AinV ≥ α, Aout

V < β

TUnclassi f iable, for AoutV ≥ β

(4.1)

Figure 4.3 illustrates the different area definitions by depicting six polygonal pathsthat define different trajectories towards three target points, as well as their classificationresults.

The computation of the individual areas is based on splitting a polygonal path intoseveral sub-polygons Vi, each limited by two consecutive intersections with the line lt.The different line segments of the path are traversed and tested for intersection withlt. In case of an intersection, all vertices starting from the previous intersection pointto the current one define a sub-polygon Vi (for the first sub-polygon V1, the origin isset to be the initial intersection point). Each time a sub-polygon is closed this way, thetotal area AVi of the sub-polygon is computed. Subsequently, the intersection area Ain

Vbetween the sub-polygon and the rectangle Rt is computed via polygon clipping [49,53]. Algorithm 1 provides more details on the implementation.

One minor limitation of our classification algorithm is that for the two-way targetingapproach, no distinction is currently made whether the vertical or the horizontal angleis matched first. Furthermore, the classification might be imprecise for target pointslocated very close to the coordinate axes x = 0 or y = 0, where the area of Rt is verysmall and can even be 0 for target points lying exactly on one of the axes. In these cases,a classification is not meaningful since one-way and two-way targeting approaches arealmost the same and cannot be distinguished. In addition to that, large values for Aout

Vand therefore misclassification are very likely to occur. To account for these cases, it ispossible to simply discard these trajectories with target points close to one of the axesand classify them as outliers. The drawback of this approach is that for these cases,target-oriented trajectories are treated the same as random trajectories. However, weargue that for our research question, which is to identify patterns in the head rotationdata, this can be tolerated. Another set of outliers are those trajectories, whose lengthexceeds a certain percentage of lt, i.e. a trajectory that is three times longer than theoptimal path is discarded and classified as an outlier.

26

4 Methods

Algorithm 1 Computation of the areas AinV and Aout

V enclosed between a given polygonalpath and the line towards the target point.

1: procedure calculateEnclosingAreas

2: Input:3: t← target point4: V ← list of vertices of polygonal path towards target point5: Output:6: Ain

V ← total area inside Rt

7: AoutV ← total area outside of Rt

8: Algorithm:9: Rt ← rectangular polygon defined by origin and t

10: lt ← line segment between origin and t11: Vc ← empty list of vertices for current sub-polygon12: for all vertices v in V do13: l ← current line segment defined by v and v + 114: intersects, p← testIntersection(l, lt)15: if intersects then16: Vc ← append intersection point p17: AVc ← calculatePolygonArea(Vc)18: Ain

Vc← calculatePolygonIntersectionArea(Vc, Rt)

19: AoutVc← AVc − Ain

Vc

20: AinV ← Ain

V + AinVc

21: AoutV ← Aout

V + AoutVc

22: Vc ← clear23: Vc ← append intersection point p24: Vc ← append next vertex v + 125: else26: Vc ← append current vertex v

27

4 Methods

Figure 4.1: Schematic drawing illustrating how objects are positioned on the 3D Radar.Note that this is not the perspective seen by a user in an actual application.For an actual user’s point of view, we refer to figure 4.2. Five objectsare positioned in the scene. A vertical line connects the objects to theirprojections on the radar plane. The encoding of an object’s location isillustrated with the purple cube. The green area marks the horizontalangle between the user’s viewing direction and the object and the red areamarks the vertical angle between the user’s viewing direction and the object.Whether an object is above or below the user is additionally indicated by thecolor of the object’s projection on the radar plane. A blue projection indicatesthat the object is above the user and a transparent projection indicated thatthe object is below the user.

28

4 Methods

Figure 4.2: Screenshots of the six different visual guidance techniques which werecompared with respect to their potential for displaying out-of-view objects inMixed Reality environments: 3D Arrows (top-left), AroundPlot (top-center),3D Radar (top-right), EyeSee360 (bottom-left), sidebARs (bottom-center),and Mirror Ball (bottom-right).

29

4 Methods

-30

-30

-20

-20

0

0

-10

-10

5040

50

40

10

10

20

20

30

30

-40

-40

-50Horizontal Angle

VerticalAngle

Figure 4.3: Visual explanation of the areas AinV and Aout

V for six exemplary targetingtrajectories V1 (blue, classified as TOneWay), V2 (green, classified as TTwoWay),V3 (orange, classified as TInBetween), V4 (pink, classified as TUnclassi f iable), V5

(brown, classified as Outlier due to long path), and V6 (yellow, classified asOutlier due to horizontal angle of tblue ≈ 0).

30

5 User Study

To investigate the potential of our proposed visualization techniques for visuallyguiding the view of HMD users towards out-of-view objects, we designed a user studythat reflects important visual search and navigation scenarios in MR environments.Following an interactive tutorial, participants performed three different object collectiontasks, during which completion time and the head rotation trajectories were recorded.Mental effort levels and overall usability scores were measured for all six visualizationsin post-experiment surveys.

5.1 Subjects

A total of twenty-four subjects aged from 22 to 47 (19 male and 5 female) with amean age of 26.63± 7.05 years were recruited for the experiment. All of them wereunpaid volunteers and provided their informed consent to the experiment protocol. Aprerequisite for participation was that all subjects had proper color vision, as differentlycolored virtual objects had to be distinguished quickly. All participants passed thescreening using the standard Ishihara color vision deficiency test [25], consisting ofvarious color plates which were shown to the subjects on a smartphone application.

5.2 Task & Procedure

The experiment was conducted in an open office space. A Microsoft HoloLens HMDwas used by the participants to view the MR environment. For collecting virtual objects,participants were asked to use the HoloLens clicker as the input device, which provedto be more robust and novice-friendly than the tap gesture in a pilot study. Figure 5.1shows a participant in our user study.

5.2.1 Calibration & Interactive Tutorial

At the beginning of the session, each participant went through the calibration appli-cation that is pre-installed on the Microsoft HoloLens. The application calibrates thedisplay of the HMD according to the users’ individual inter-pupillary distance. By

31

5 User Study

going through the calibration, all users were familiarized with the basic controls of theHoloLens. Subsequently, every participant went through a tutorial application, whichintroduced the different visualization techniques. For each method, the tutorial offeredtext instructions, a visual demonstration, and the collection of three demo objects.During the tutorials, the participants had no time constraints and no measurementswere taken.

5.2.2 Object Collection Scenarios

After successful completion of the tutorial, the main part of the user study followed.During three different scenarios, participants were asked to use the six visualizationtechniques for the task of collecting a total of eight virtual objects located in the spacearound them as fast as possible. Prior to each visualization technique, a start buttonand the name of the current technique were displayed. The experiment, as well as themeasurement of completion times and head rotation trajectories, began as soon as theuser clicked on the start button. After collecting all objects in all six techniques, theapplication moved on to the next scenario.

The first scenario was a Sequential object collection where participants had to collectthe eight objects one-by-one in sequential order. Only one object was displayed at atime and the visualization technique only showed hints for this object. After collectingan object, the next one appeared until all eight objects were collected. This scenarioresembles use cases where the user is looking for a specific virtual object and is ableto pass this information on to the application. This way, it is possible to limit thevisualization technique to only display hints for the desired object.

In the second scenario (Random), the eight objects could be collected in an arbitraryorder. All the objects were displayed at the same time and the visualization techniqueshowed hints for all of them. This scenario is inspired by a more exploratory searchtask, where the user tries to quickly understand the MR environment and where allobjects are potentially interesting for the user.

The third and last scenario required the user to collect the virtual objects in a Specificorder. All eight objects were displayed at the same time and the visualization techniquedisplayed hints for all objects. The user could, however, only collect one specific object,for which a visual hint in the form of a text message was displayed on the screen (e.g."Collect the yellow sphere!"). All other objects did not react when the user tried tocollect them. Once the user collected the correct object, the next hint for the next objectwas displayed on-screen until all objects were collected. This scenario resembles asituation where the user is looking for something specific in a potentially clutteredenvironment but does not pass the information to the application. Therefore, it isnot possible to limit the number of hints that are displayed. We artificially create the

32

5 User Study

incentive to search for a specific object by requiring the user to collect this object.

5.2.3 Post Experiment Surveys

After finishing all scenarios, participants were asked to fill out a survey. We collecteddata in form of their demographic information, experience with virtual and augmentedreality and their experience with gaming in general. The subjects continued on toreport the mental effort experienced during the different visualization techniques basedon the 9-point scale of Paas [38]. Eventually, users had to fill out a SUS survey, resultingin a SUS score for every visualization technique.

The total duration of the experiment, including tutorial and post-experiment surveys,was between 30 and 50 minutes for each participant.

5.3 Object Placement

The objects that participants had to collect were positioned in a pseudo-random wayaround the user. We created a cube at the position of the users’ head, which wassubdivided into 3× 3× 3 inner cubes. We then randomly positioned our virtual objectsat the centers of these 27 inner cubes. Objects could not be placed into cubes directlyinside, below, above, and in front of the users, resulting in 23 potential object positions.In every scene, we placed eight objects with different shapes and uniquely identifiablecolors. All objects followed the translational movements of the user, such that it wasnot possible to move towards or away from an object.

5.4 Design

There were two independent variables, which were controlled during the user study.The first variable had three levels and corresponded to the object collection scenario, i.e.the order, in which virtual objects had to be collected (sequential, random, or specific).With the second independent variable, we controlled the visualization technique whichwas used to convey the 3D position of out-of-view objects. Consequently, our userstudy had a 3× 6 within-subjects design. A balanced Latin square matrix was employedduring both the tutorial and the three object collection scenarios for randomizing theseconditions across study participants [54]. The two dependent variables we measuredduring the experiment were task completion time and the head rotation trajectory. Theformer defined the time necessary to collect all virtual objects for a given visualizationmethod and collection scenario. The head rotation trajectory, expressed in sphericalcoordinates, corresponds to the path from the initial head rotation of a participant to

33

5 User Study

the last target object and was recorded as the rotational component of the HoloLens’3D pose at isochronal intervals.

5.5 Hypotheses

With our experimental setup in mind, we formulated the following hypothesis whichwas subject to an extensive statistical evaluation:

H 1. Mean task completion times with our newly introduced techniques are lower orequal to existing state of the art techniques.

Furthermore, we expected that the different visualization techniques can be classifiedaccording to the object targeting approaches employed by participants. Especiallyfor the 3D Radar, we predicted the majority of head rotation trajectories to follow anindirect, two-way targeting approach.

34

5 User Study

Figure 5.1: Image of a participant in the user study. The shapes are usually onlyvisible when wearing a head-mounted display. In the image the shapes aresuperimposed onto the original image to illustrate the collection task froman external point of view.

35

6 Results

In this section, we provide a detailed analysis of the results obtained during the courseof our user study. Overall, the best performing visualizations were EyeSee360 and ourproposed 3D Radar, both in terms of completion times and overall usability. Further-more, our results clearly demonstrate that participants preferred to choose the indirect,two-way object targeting approach for the 3D Radar. For all other visualizations,especially for EyeSee360, the direct, one-way targeting approach was predominantlyused.

6.1 Completion Times

For comparing the time necessary to collect all virtual objects in the three differentscenarios, we employed a univariate analysis of variances (ANOVA) with repeatedmeasures in conjunction with Tukey post-hoc tests to reveal significant differencesbetween the six visualization techniques. The results are summarized in Figure 6.1.

6.1.1 Sequential Collection Scenario

For the first scenario, in which the virtual objects were collected in a sequential order,significant differences were recorded (F5,138 = 8.22, p < 0.001, η2 = 0.23). Participantsachieved significantly lower mean task completion times using the 3D Arrows (p =

0.015), the 3D Radar (p = 0.002), AroundPlot (p < 0.001), and EyeSee360 (p < 0.001)compared to the sidebARs visualization. Furthermore, mean completion times weresignificantly lower for the 3D Radar (p = 0.013), AroundPlot (p = 0.002), and EyeSee360(p < 0.001) compared to the Mirror Ball technique. Overall, EyeSee360 had the lowestmean completion time (24.76± 4.04s), followed by AroundPlot (26.40± 6.23s) and the3D Radar (27.76± 5.94s), cf. Table 6.1.

6.1.2 Random Collection Scenario

In the second scenario, virtual objects were collected in an arbitrary, subject-determinedorder. Results were comparable to the sequential scenario with significant differences atthe p < 0.001 level (F5,138 = 10.35, η2 = 0.27). Mean completion times of 3D Arrows (p <

36

6 Results

0.001), 3D Radar (p < 0.001), AroundPlot (p = 0.001), and EyeSee360 (p < 0.001) wereagain significantly lower than for sidebARs. However, only the 3D Radar (p = 0.014)and EyeSee360 (p = 0.002) had significantly lower mean completion times comparedto the Mirror Ball condition. EyeSee360 was again the fastest method (18.58± 3.63s),closely followed by 3D Radar (19.85± 3.87s) and 3D Arrows (21.86± 4.68s).

6.1.3 Specific Collection Scenario

The third and final scenario was similar to the random scenario in a sense that all virtualobjects were visible simultaneously. However, only one specific object could be collectedat a time. Similar to the previous two scenarios, an ANOVA revealed significantdifferences for mean task completion times (F5,138 = 3.94, p = 0.0023, η2 = 0.13).Interestingly, mean completion times of EyeSee360 were not only significantly lowerthan sidebARs (p < 0.033), but also compared to AroundPlot (p < 0.003), which was theslowest visualization technique for this scenario. Besides this, participants performedcomparably with all other techniques, such that no other significant differences couldbe revealed. Table 6.1 again lists all mean completion times for the three differentscenarios.

6.1.4 Augmented Reality and Gaming Experience

In order to evaluate whether previous experiences with AR and gaming had any effecton the completion times, participants were split into two groups: one for low andone for high experience with AR and gaming respectively. Both experience levels Ewere reported in the post-experiment survey and followed a seven-level Likert scale.For AR experience, the two groups had mean experiences EHigh

AR = 4.58 ± 1.00 andELow

AR = 1.25 ± 0.45, while for gaming the mean experiences were EHighGaming = 6.6 ± 0.49

and ELowGaming = 3.17 ± 0.83. In both groups with high experience levels, the mean

completion times were slightly faster for each visualization technique and across allthree scenarios. However, for none of these techniques, the differences were statisticallysignificant.

6.2 Head Rotation Trajectory Analysis

For analyzing how people targeted the virtual objects with the different visualizationmethods, we employed our previously discussed classification algorithm. As objecttargeting trajectories could vary greatly in the random scenario and an optimal pathfor collecting all objects is difficult to define in this scenario, we restricted our analysis

37

6 Results

*

**

****

****

** **

******

***

*********

MeanCom

pletionTimes

(ins)

3D Arrows

20

30

40

50

60

70

RandomSequentialSpecific

80

3D Radar AroundPlot EyeSee360 Mirror Ball sidebARs

Figure 6.1: Combined mean completion time results for the object collection study. Inthree different scenarios (sequential, random, specific), participants wereasked to collect virtual shapes as fast as possible using the six differentvisualization techniques. Significant time differences are indicated as *(p < 0.05), ** (p < 0.01), and *** (p < 0.001).

38

Table 6.1: Comparison of the mean completion times for all three scenarios of the object collection study, as well asaverage SUS-scores and cognitive load results for all six visualization techniques.

Conditions Completion Time SUS-Score Cognitive Load

Random Sequential Specific

Mean µ SD σ Mean µ SD σ Mean µ SD σ Mean µ SD σ Mean µ SD σ

3D Arrows 21.86s 4.68s 29.30s 7.04s 34.81s 9.15s 71.29 17.34 3.87 1.96

3D Radar 19.85s 3.87s 27.76s 5.94s 35.69s 9.82s 83.63 17.10 2.71 1.74

AroundPlot 23.17s 9.40s 26.40s 6.23s 43.03s 13.45s 79.19 12.82 3.13 1.23

EyeSee360 18.58s 3.63s 24.76s 4.04s 31.84s 9.38s 89.27 10.31 1.55 0.68

Mirror Ball 26.61s 9.38s 37.23s 7.40s 39.88s 15.29s 44.35 19.87 5.45 2.11

sidebARs 31.32s 10.01s 38.66s 10.76s 40.91s 14.95s 51.05 19.67 5.26 2.35

6 Results

of head rotation data to the other two scenarios with sequential and specific objectcollection orders. In Figure 6.2, all 48 targeting trajectories (8 target objects × 6visualization techniques) of one particular participant for the sequential scenario aredepicted.

-180-90

-60

-30

0

30

60

90-90

-60

-30

0

30

60

90-90

-60

-30

0

30

60

90

-120 -60 0 60

Horizontal Angle

VerticalAngle

120 180 -180 -120 -60 0 60 120 180

AroundPlot

3D Arrows

Mirror Ball

EyeSee360

3D Radar

sidebARs

Figure 6.2: All trajectories from one participant for the six visualization techniquesduring the sequential collection scenario. The different colors correspondto the colors of the virtual objects that were collected with the respectivetrajectory. Note: No distinction between Ain

V and AoutV is made.

6.2.1 Outlier Removal

Prior to classification, outlier trajectories were identified and removed from the data.There are two types of outliers: the first one is characterized by a trajectory lengththat exceeds the length of lt (i.e. the line from the origin to the target point t) by

40

6 Results

a certain percentage. For our analysis, this percentage was set to 300, such that alltrajectories whose length was 3-times longer than that of lt were removed. Furthermore,we classified all those trajectories as outliers, where the target virtual object was closeto one the axes x = 0 or y = 0. In these cases, it is not possible to distinguish betweena one-way approach and a two-way approach. For our analysis, this threshold wasset to 10◦ for both the horizontal and vertical angle. For the sequential scenario, thepercentages of trajectories classified as outliers using these two criteria were 19.26%,distributed in the following way: 3D Arrows (3.04%), 3D Radar (2.60%), AroundPlot(2.69%), EyeSee360 (2.86%), Mirror Ball (4.51%), and sidebARs (3.56%). In the specificscenario, the percentage of outliers was almost the same with 19.10% of all trajectories:3D Arrows (3.56%), 3D Radar (3.13%), AroundPlot (2.95%), EyeSee360 (1.91%), MirrorBall (3.47%), and sidebARs (4.08%).

6.2.2 Class Definitions & Thresholds

For all remaining trajectories (inliers), Algorithm 1 is applied to calculate the ar-eas Ain

V and AoutV . Based on these two metrics, the trajectories are classified as

TOneWay, TTwoWay, TInBetween, or TUnclassi f iable, according to equation 4.1. As describedin section 4.4, the classes are used to describe which object targeting approach was usedby the participant. For the thresholds α and β in equation 4.1, we set the values to 20%and 30% respectively to have a rather conservative estimate and allow for moderate

marginals. Figure 6.3 illustrates the distribution of ( AinV

ARt, Aout

VARt

) ratio pairs as well as theclassification boundaries defined by the two thresholds α and β for both the sequentialand specific scenario.

6.2.3 Sequential Scenario

After the removal of outliers, there was still a moderate percentage of trajectories thatcould not be classified due to large portions of the area contributing to Aout

V . Interest-ingly, the visualizations with the smallest percentages of unclassifiable trajectories werealso the ones with the lowest mean completion times in the sequential scenario, namelyEyeSee360 (11.61%), AroundPlot (15.93%), and 3D Radar (16.67%). The Mirror Ball hadthe largest percentage of trajectories classified as TUnclassi f iable with 28.16%.

The largest percentages of trajectories for all six visualizations was classified asTInBetween. For these trajectories, the relative percentage of Ain

V was in the range of[α, 100− α] and thus could not be assigned clearly to one of the two targeting ap-proaches, cf. Figure 6.3.

From the remaining trajectories, between 30%− 40% were classified as either TOneWayor TTwoWay, which is still an acceptable percentage considering the large inter-operator

41

6 Results

variabilities of targeting virtual objects in AR. Our results demonstrate that participantsclearly preferred a one-way targeting approach for all visualization techniques, withthe exception of the 3D Radar. This was especially evident for the EyeSee360 method,where only 2.68% of all trajectories were classified as TTwoWay, compared to 34.82%classified as TOneWay. The 3D Radar was the only method which favored a two-wayapproach, with 26.47% of all trajectories classified as TTwoWay, compared to 10.78% forTOneWay. Table 6.2 provides the exact percentages for all six visualization techniques.

6.2.4 Specific Scenario

In the specific collection scenario, we observed the same overall trends, reaffirmingour results from the sequential scenario. Participants predominantly used the two-waytargeting approach with the 3D Radar visualization (24.77% compared to 9.73% forTOneWay). For all other techniques, the one-way targeting approach was clearly preferredby the majority, again most notably for EyeSee360 (TOneWay = 40.37%, TTwoWay = 7.34%).Surprisingly, the overall percentages of trajectories that could not be classified wereslightly lower compared to the (theoretically easier) sequential collection scenario.

6.2.5 System Usability Scale

The perceived usability of all six visualization techniques was studied by means of aSUS questionnaire, cf. Table 6.1. Very high scores were recorded for both EyeSee360(89.27± 10.31) and the 3D Radar (83.63± 17.10), followed by AroundPlot (79.19±12.82) and 3D Arrows (71.29± 17.34). The Mirror Ball and sidebARs were clearlyoutperformed and received much lower SUS-scores with 44.45± 19.87 and 51.05± 19.67respectively.

6.2.6 Mental Effort

For comparing the mental effort experienced by the subjects during the three differentcollection scenarios, non-parametric Kruskal-Wallis tests were employed. They showeda significant effect of the visualization technique on the experienced mental effort(H(5) = 60.06, p < 0.001, η2 = 0.40). Post-hoc tests revealed significantly lower mentaleffort levels for EyeSee360 compared to 3D Arrows (p < 0.001), AroundPlot (p = 0.007),the Mirror Ball (p < 0.001), and sidebARs (p < 0.001). Similarly, the experiencedmental work levels were significantly lower for the 3D Radar compared to the MirrorBall (p < 0.001) and sidebARs (p < 0.001). Table 6.1 provides the mean mental effortscores for all visualization techniques, which were reported according to a 9-point scaleof Paas [38].

42

6 Results

Specific Scenario

0 20

20

40

40

60

60

80

80

1000 20 40 60 80 100

100

120

140

160

180

200

220

240

20

40

60

80

100

120

140

160

180

200

220

240

Sequential Scenario

Frequency Frequency

Frequency

Frequency

Relative Area Ratio Relative Area Ratio

RelativeAreaRatio

RelativeAreaRatio

Figure 6.3: Distribution of head rotation trajectories for the sequential and specificcollection scenario without outliers. Each point corresponds to one trajectory,which is described in terms of the two ratios Ain

V /ARt and AoutV /ARt . The

classification boundaries are indicated by three colored rectangles: TOneWay(red), TInBetween (yellow), TTwoWay (green). All other trajectories (blue) areassigned to TUnclassi f iable.

43

Table 6.2: Results of the head rotation trajectory classification. For both the sequential and specific collection scenario,the percentages of all four classes indicating different object targeting approaches are summarizedindividually for all six visualization techniques.

Conditions Sequential Specific

One-Way Two-Way In-Between Unclassifiable One-Way Two-Way In-Between Unclassifiable

3D Arrows 25.69% 8.26% 41.28% 24.77% 26.55% 10.62% 43.36% 19.47%

3D Radar 10.78% 26.47% 46.08% 16.67% 9.73% 24.77% 46.28% 19.20%

AroundPlot 21.24% 11.50% 51.33% 15.93% 32.08% 20.75% 33.96% 13.21%

EyeSee360 34.82% 2.68% 50.89% 11.61% 40.37% 7.34% 41.28% 11.01%

Mirror Ball 29.13% 12.62% 30.10% 28.16% 26.61% 11.93% 32.11% 29.36%

sidebARs 21.37% 17.95% 36.75% 23.93% 21.90% 14.29% 42.86% 20.95%

7 Discussion

Three main observations may be inferred from the results of our user study. First,the performance of our proposed 3D Radar is on par with state-of-the-art techniquesfor visualizing out-of-view objects. Even if not outperforming EyeSee360, our resultsindicate that the 3D Radar provides an intuitive way for quickly navigating MRenvironments. In contrast to other techniques, the 3D Radar offers the potential toencode a variety of information in a small portion of the users’ view, including boththe direction towards a target object as well as its distance. By varying the number andradii of the concentric circles, the 3D Radar additionally allows for encoding variousranges of distances, which is not possible with techniques such as EyeSee360 or 3DArrows. This way, the inner circles can be used to convey information about objects inclose proximity, while the outer circles display objects that are possibly far away fromthe user. In our study, we used up- and downward facing triangles as proxy icons toindicate whether an object is above or below the user. However, this information is alsoencoded in the color of both the projection circle and the line connecting the proxies andthe projections. Therefore, it is possible for other applications to replace the triangleswith small pictures of the individual objects to gain an even better understandingof the environment. While the proxies in Aroundplot and EyeSee360 could also bereplaced by more specific icons, the 3D Arrows can hardly support this type of encoding.Additionally, the 3D Radar takes up significantly less screen estate than the slightlysuperior EyeSee360 method. For certain applications, that continuously require anunobstructed view onto the real scene, the 3D Radar might be the preferred choice.These potential advantages need to be further investigated in future experiments.

The second observation concerns the inferior performance of the Mirror Ball, bothin terms of collection times of virtual objects and overall usability. We think there areseveral factors that contributed to these results: first of all, interactions with sphericalmirrors hardly occur in our everyday life. Therefore, interpreting the reflections onthe Mirror Ball is not intuitive and requires high mental effort. This is reflected in thehighest cognitive load scores during our questionnaires for the Mirror Ball. Secondly,inferring the location of objects by looking at their reflections on the Mirror Ball iscomplicated by the fact, that the physical environment is not reflected on the MirrorBall. This could potentially be achieved by having either a dynamic 3D model ofthe environment or a 360◦ video from the users’ perspective. However, the resulting

45

7 Discussion

visualization would no longer correspond to a visual attention guide towards virtualobjects but correspond to an alternative, distorted representation of the world. Weattempted to improve the perception of the Mirror Ball by reflecting a skybox in form ofa spherical grid, which only partially fulfilled its purpose. Additionally, users reportedthat it is difficult to recognize whether an object is behind or in front of the user becausethe user’s body is not reflected by the Mirror Ball. This issue could potentially besolved by placing a semi-transparent representation of the users’ body in the virtualworld. A technical issue that was specifically evident for the Mirror Ball was screentearing due to the limited computing power of the HoloLens. This could possibly beimproved by employing a custom shader that handles reflections more efficiently orrepresenting virtual objects in form of their projections on the Mirror Ball rather thanreflections. Considering the first two observations, we can only accept our hypothesisH1 for the 3D Radar.

A third important observation was the tendency of participants to favor indirect,two-way targeting trajectories when using the 3D Radar, which confirmed our initialassumption. The comparable completion time results between the 3D Radar andEyeSee360 are especially interesting in the light of this observations, as users generallyfollowed longer trajectories with the former. Our area-based trajectory classificationalgorithm proved to be a suitable tool for discriminating between the different objecttargeting approaches. While our analysis of the head rotation data presents a firststep in understanding the rotational movement patterns in MR environments, furtherinvestigation is required. A common phenomenon which frequently occurred forall types of visualizations were overshoots, i.e. moving too far in the direction of thetarget objects with a subsequent reversal. Similarly, directional corrections where usersinitially moved to a wrong direction could be observed especially at the beginning ofhead trajectories, but were not included in our analysis. Investigating the trajectorydata with respect to these two patterns and developing mitigation strategies will bean interesting area for future research. Another potential direction for future researchis the integration of eye tracking. Examining how often and for how long users focuson a certain visualization could provide further insights into the perceptual aspects ofvisualizing out-of-view objects.

46

8 Conclusion

In this thesis, we presented the Mirror Ball and the 3D Radar as two novel approachesfor visualizing out-of-view objects in MR environments. We evaluated their potentialto convey information about the location of virtual objects in a user study whichreflected three scenarios of real-world exploratory and goal-oriented search tasks. Inour experiments, performance for the 3D Radar was comparable to the best state-of-the-art techniques, while at the same time offering several advantages such as smallerfield-of-view coverage and additional information encoding potential. For the MirrorBall, several perceptual challenges have to be further investigated before comparableresults can be expected. In addition to the two novel visualization approaches, weproposed a new area-based method for analyzing and classifying head rotation trajecto-ries. We demonstrated that different visualizations encourage distinct object targetingapproaches in MR.

Our research has a few limitations. Our results concerning the performance of thedifferent visualization techniques are only representative for the scenarios we tested.The task consisted of collecting eight objects at the distance of about one to two metersand was conducted in an office space. Changing parameters like the number of objects,the distance to the objects and the environment could potentially benefit some methods,while putting others at a disadvantage. However, all attempts at evaluating out-of-viewvisualization techniques suffer from this problem to some degree, since the potentialuse cases for MR applications are numerous and have highly variable requirements.Apart from that we do not measure the additional mental effort introduced by thevisualization techniques on top of other tasks. Our task only consists of object collection.However, a common use case for such techniques could be to use them in alreadyexisting application to help with the localization of off-screen objects, which wouldlikely increase the mental effort when using those applications to some degree. Wedo not measure these effects in our study. Measuring the head trajectory data of theparticipants yields a lot of raw data. Our approach to classify head trajectories is onlyone of potentially many metrics one could derive from the data. Furthermore, ourunderstanding of our classification is still limited. More research will be required todetermine whether a certain kind of trajectory is desirable and which features of avisualization technique are likely to produce which trajectory.

On a personal note it was a valuable experience to conclude my first user study.

47

8 Conclusion

Developing an application that is going to be used by participants who have neverinteracted with the system before requires a lot more attention and care than anapplication that is only used by developers that know exactly how to operate thesystem. Apart from that it turned out to be challenging to interpret the raw data resultsfrom the user study, especially for the head trajectory data. Processing and visualizingthe data, as well as eventually identifying interesting findings required a lot of time,effort and some trial and error.

To conclude this thesis, we hope that the techniques we introduced prove usefulbeyond our experiment and that this work opens the path for further research in thearea of head rotation analysis in MR environments in order to deeply understand theunderlying perceptual effects of different out-of-view object visualization techniques.

48

List of Figures

2.1 RV continuum illustrated as a one-dimensional line. . . . . . . . . . . . 4

3.1 Example pictures of different 2D off-screen visualization techniques. a)City Lights; b) Halo; c) Wedge; d) Edge Radar; e) Scaled Arrows; f)Stretched Arrows; g) SidebARs . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 Example pictures of different 3D off-screen visualization techniques.a) 3D Halo; b) 3D Halo Projection; c) 3D Arrows; d) Parafrustum;e) Attention Funnel; f) SWAVE; g) Vibrating icons; h) Aroundplot; i)EyeSee360; j) Mirror Ball; k) 3D Radar . . . . . . . . . . . . . . . . . . . 16

4.1 Schematic drawing illustrating how objects are positioned on the 3DRadar. Note that this is not the perspective seen by a user in an actualapplication. For an actual user’s point of view, we refer to figure 4.2. Fiveobjects are positioned in the scene. A vertical line connects the objects totheir projections on the radar plane. The encoding of an object’s locationis illustrated with the purple cube. The green area marks the horizontalangle between the user’s viewing direction and the object and the redarea marks the vertical angle between the user’s viewing direction andthe object. Whether an object is above or below the user is additionallyindicated by the color of the object’s projection on the radar plane. A blueprojection indicates that the object is above the user and a transparentprojection indicated that the object is below the user. . . . . . . . . . . . 28

4.2 Screenshots of the six different visual guidance techniques which werecompared with respect to their potential for displaying out-of-viewobjects in Mixed Reality environments: 3D Arrows (top-left), AroundPlot(top-center), 3D Radar (top-right), EyeSee360 (bottom-left), sidebARs(bottom-center), and Mirror Ball (bottom-right). . . . . . . . . . . . . . . 29

4.3 Visual explanation of the areas AinV and Aout

V for six exemplary tar-geting trajectories V1 (blue, classified as TOneWay), V2 (green, classifiedas TTwoWay), V3 (orange, classified as TInBetween), V4 (pink, classified asTUnclassi f iable), V5 (brown, classified as Outlier due to long path), and V6

(yellow, classified as Outlier due to horizontal angle of tblue ≈ 0). . . . . 30

49

List of Figures

5.1 Image of a participant in the user study. The shapes are usually onlyvisible when wearing a head-mounted display. In the image the shapesare superimposed onto the original image to illustrate the collection taskfrom an external point of view. . . . . . . . . . . . . . . . . . . . . . . . . 35

6.1 Combined mean completion time results for the object collection study. Inthree different scenarios (sequential, random, specific), participants wereasked to collect virtual shapes as fast as possible using the six differentvisualization techniques. Significant time differences are indicated as *(p < 0.05), ** (p < 0.01), and *** (p < 0.001). . . . . . . . . . . . . . . . . 38

6.2 All trajectories from one participant for the six visualization techniquesduring the sequential collection scenario. The different colors correspondto the colors of the virtual objects that were collected with the respectivetrajectory. Note: No distinction between Ain

V and AoutV is made. . . . . . 40

6.3 Distribution of head rotation trajectories for the sequential and specificcollection scenario without outliers. Each point corresponds to onetrajectory, which is described in terms of the two ratios Ain

V /ARt andAout

V /ARt . The classification boundaries are indicated by three coloredrectangles: TOneWay (red), TInBetween (yellow), TTwoWay (green). All othertrajectories (blue) are assigned to TUnclassi f iable. . . . . . . . . . . . . . . . 43

50

List of Tables

6.1 Comparison of the mean completion times for all three scenarios of theobject collection study, as well as average SUS-scores and cognitive loadresults for all six visualization techniques. . . . . . . . . . . . . . . . . . 39

6.2 Results of the head rotation trajectory classification. For both the se-quential and specific collection scenario, the percentages of all fourclasses indicating different object targeting approaches are summarizedindividually for all six visualization techniques. . . . . . . . . . . . . . . 44

51

Bibliography

[1] H. Alt and M. Godau. “Computing the Fréchet distance between two polygonalcurves.” In: International Journal of Computational Geometry & Applications 5.01n02(1995), pp. 75–91.

[2] A. Anagnostopoulos, M. Vlachos, M. Hadjieleftheriou, E. Keogh, and P. S. Yu.“Global distance-based segmentation of trajectories.” In: Proceedings of the 12thACM SIGKDD international conference on Knowledge discovery and data mining. ACM.2006, pp. 34–43.

[3] R. T. Azuma. “A Survey of Augmented Reality.” In: Presence: Teleoper. VirtualEnviron. 6.4 (Aug. 1997), pp. 355–385. issn: 1054-7460. doi: 10.1162/pres.1997.6.4.355.

[4] F. I. Bashir, A. A. Khokhar, and D. Schonfeld. “Object trajectory-based activityclassification and recognition using hidden Markov models.” In: IEEE transactionson Image Processing 16.7 (2007), pp. 1912–1919.

[5] F. I. Bashir, A. A. Khokhar, and D. Schonfeld. “Real-time motion trajectory-basedindexing and retrieval of video sequences.” In: IEEE Transactions on Multimedia9.1 (2007), pp. 58–65.

[6] P. Baudisch and R. Rosenholtz. “Halo: A Technique for Visualizing Off-screenObjects.” In: Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems. CHI ’03. Ft. Lauderdale, Florida, USA: ACM, 2003, pp. 481–488. isbn:1-58113-630-7.

[7] J. Bian, D. Tian, Y. Tang, and D. Tao. “A survey on trajectory clustering analysis.”In: arXiv preprint arXiv:1802.06971 (2018).

[8] F. Biocca, A. Tang, C. Owen, and F. Xiao. “Attention Funnel: Omnidirectional 3DCursor for Mobile Augmented Reality Platforms.” In: Proceedings of the SIGCHIConference on Human Factors in Computing Systems. CHI ’06. Montr&#233;al,Qu&#233;bec, Canada: ACM, 2006, pp. 1115–1122. isbn: 1-59593-372-7.

[9] J. Brooke. “SUS: A quick and dirty usability scale.” In: 189 (Nov. 1995).

[10] S. Burigat and L. Chittaro. “Navigation in 3D virtual environments: Effects ofuser experience and location-pointing navigation aids.” In: International Journal ofHuman-Computer Studies 65.11 (2007), pp. 945–958.

52

Bibliography

[11] S. Burigat, L. Chittaro, and S. Gabrielli. “Visualizing Locations of Off-screenObjects on Mobile Devices: A Comparative Evaluation of Three Approaches.” In:Proceedings of the 8th Conference on Human-computer Interaction with Mobile Devicesand Services. MobileHCI ’06. Helsinki, Finland: ACM, 2006, pp. 239–246. isbn:1-59593-390-5.

[12] M. S. T. Carpendale and C. Montagnese. “A Framework for Unifying PresentationSpace.” In: Proceedings of the 14th Annual ACM Symposium on User Interface Softwareand Technology. UIST ’01. Orlando, Florida: ACM, 2001, pp. 61–70. isbn: 1-58113-438-X.

[13] D. Chekhlov, A. P. Gee, A. Calway, and W. Mayol-Cuevas. “Ninja on a Plane: Auto-matic Discovery of Physical Planes for Augmented Reality Using Visual SLAM.”In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.Nov. 2007, pp. 153–156. doi: 10.1109/ISMAR.2007.4538840.

[14] L. Chen and R. Ng. “On the marriage of lp-norms and edit distance.” In: Proceed-ings of the Thirtieth international conference on Very large data bases-Volume 30. VLDBEndowment. 2004, pp. 792–803.

[15] Elite:Dangerous. url: https://en.wikipedia.org/wiki/Elite:_Dangerous(visited on 03/11/2018).

[16] S. R. Ellis, B. D. Adelstein, and K. Yeom. “Human control in rotated frames:anisotropies in the misalignment disturbance function of pitch, roll, and yaw.” In:Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 56. 1.SAGE Publications Sage CA: Los Angeles, CA. 2012, pp. 1336–1340.

[17] Eve:Valkyrie. url: https://en.wikipedia.org/wiki/Eve:_Valkyrie (visited on03/11/2018).

[18] D. R. Faria and J. Dias. “3D hand trajectory segmentation by curvatures andhand orientation for classification through a probabilistic approach.” In: IntelligentRobots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on. IEEE.2009, pp. 1284–1289.

[19] J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rendón-Mancha. “Visual simulta-neous localization and mapping: a survey.” In: Artificial Intelligence Review 43.1(Jan. 2015), pp. 55–81. issn: 1573-7462. doi: 10.1007/s10462-012-9365-8.

[20] U. Gruenefeld, D. Ennenga, A. E. Ali, W. Heuten, and S. Boll. “EyeSee360:Designing a Visualization Technique for Out-of-view Objects in Head-mountedAugmented Reality.” In: Proceedings of the 5th Symposium on Spatial User Interaction.SUI ’17. Brighton, United Kingdom: ACM, 2017, pp. 109–118. isbn: 978-1-4503-5486-8.

53

Bibliography

[21] U. Gruenefeld, D. Hsiao, W. Heuten, and S. Boll. “EyeSee: beyond reality withMicrosoft HoloLens.” In: Proceedings of the 5th Symposium on Spatial User Interaction.ACM. 2017, pp. 148–148.

[22] S. G. Gustafson and P. P. Irani. “Comparing Visualizations for Tracking Off-screenMoving Targets.” In: CHI ’07 Extended Abstracts on Human Factors in ComputingSystems. CHI EA ’07. San Jose, CA, USA: ACM, 2007, pp. 2399–2404. isbn: 978-1-59593-642-4.

[23] S. Gustafson, P. Baudisch, C. Gutwin, and P. Irani. “Wedge: Clutter-free Visualiza-tion of Off-screen Locations.” In: Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems. CHI ’08. Florence, Italy: ACM, 2008, pp. 787–796.isbn: 978-1-60558-011-1.

[24] S. G. Hart and L. E. Staveland. “Development of NASA-TLX (Task Load Index):Results of Empirical and Theoretical Research.” In: Human Mental Workload. Ed. byP. A. Hancock and N. Meshkati. Vol. 52. Advances in Psychology. North-Holland,1988, pp. 139–183. doi: https://doi.org/10.1016/S0166-4115(08)62386-9.

[25] S. Ishihara. “Series of plates designed as tests for colour-blindness.” In: (1936).

[26] H. Jo, S. Hwang, H. Park, and J.-h. Ryu. “Aroundplot: Focus+context interfacefor off-screen objects in 3D environments.” In: Computers & Graphics 35.4 (2011).Semantic 3D Media and Content, pp. 841–853. issn: 0097-8493.

[27] R. Jonker, G. De Leve, J. Van Der Velde, and A. Volgenant. “Rounding symmet-ric traveling salesman problems with an asymmetric assignment problem.” In:Operations Research 28.3-part-i (1980), pp. 623–627.

[28] G. Klein and D. Murray. “Parallel Tracking and Mapping for Small AR Workspaces.”In: 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.Nov. 2007, pp. 225–234. doi: 10.1109/ISMAR.2007.4538852.

[29] J.-G. Lee, J. Han, X. Li, and H. Gonzalez. “TraClass: trajectory classification usinghierarchical region-based and trajectory-based clustering.” In: Proceedings of theVLDB Endowment 1.1 (2008), pp. 1081–1094.

[30] S. Matsuzoe, S. Jiang, M. Ueki, and K. Okabayashi. “Intuitive visualization methodfor locating off-screen objects inspired by motion perception in peripheral vision.”In: Proceedings of the 8th Augmented Human International Conference. ACM. 2017,p. 29.

[31] J. McCrae, M. Glueck, T. Grossman, A. Khan, and K. Singh. “Exploring the DesignSpace of Multiscale 3D Orientation.” In: Proceedings of the International Conferenceon Advanced Visual Interfaces. AVI ’10. Roma, Italy: ACM, 2010, pp. 81–88. isbn:978-1-4503-0076-6.

54

Bibliography

[32] Microsoft Hololens. url: https://www.microsoft.com/en-us/hololens (visitedon 05/10/2018).

[33] Microsoft Hololens Hardware Details. url: https://docs.microsoft.com/en-us/windows/mixed-reality/hololens-hardware-details (visited on 05/10/2018).

[34] M. J. Milford, G. F. Wyeth, and D. Prasser. “RatSLAM: a hippocampal modelfor simultaneous localization and mapping.” In: Robotics and Automation, 2004.Proceedings. ICRA ’04. 2004 IEEE International Conference on. Vol. 1. Apr. 2004,403–408 Vol.1. doi: 10.1109/ROBOT.2004.1307183.

[35] P. Milgram and F. Kishino. “A taxonomy of mixed reality visual displays.” In:IEICE TRANSACTIONS on Information and Systems 77.12 (1994), pp. 1321–1329.

[36] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino. “Augmented reality: aclass of displays on the reality-virtuality continuum.” In: vol. 2351. 1995. doi:10.1117/12.197321.

[37] C. Myers, L. Rabiner, and A. Rosenberg. “Performance tradeoffs in dynamictime warping algorithms for isolated word recognition.” In: IEEE Transactions onAcoustics, Speech, and Signal Processing 28.6 (1980), pp. 623–635.

[38] F. G. Paas. “Training strategies for attaining transfer of problem-solving skill instatistics: A cognitive-load approach.” In: Journal of educational psychology 84.4(1992), p. 429.

[39] C. Plaisant, D. Carr, and B. Shneiderman. “Image-Browser Taxonomy and Guide-lines for Designers.” In: IEEE Softw. 12.2 (Mar. 1995), pp. 21–32. issn: 0740-7459.

[40] M. Pollefeys, L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, andR. Koch. “Visual Modeling with a Hand-Held Camera.” In: International Journal ofComputer Vision 59.3 (Sept. 2004), pp. 207–232. issn: 1573-1405. doi: 10.1023/B:VISI.0000025798.50602.3a.

[41] P. Renner and T. Pfeiffer. “Attention Guiding Techniques using Peripheral Visionand Eye Tracking for Feedback in Augmented-Reality-based Assistance Systems.”In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 2017.

[42] J. P. Rolland, Y. Baillot, and A. A. Goon. A SURVEY OF TRACKING TECHNOL-OGY FOR VIRTUAL ENVIRONMENTS. 2001.

[43] T. Schinke, N. Henze, and S. Boll. “Visualization of Off-screen Objects in MobileAugmented Reality.” In: Proceedings of the 12th International Conference on HumanComputer Interaction with Mobile Devices and Services. MobileHCI ’10. Lisbon,Portugal: ACM, 2010, pp. 313–316. isbn: 978-1-60558-835-3.

55

Bibliography

[44] T. Siu and V. Herskovic. “SidebARs: Improving Awareness of Off-screen Elementsin Mobile Augmented Reality.” In: Proceedings of the 2013 Chilean Conference onHuman - Computer Interaction. ChileCHI ’13. Temuco, Chile: ACM, 2013, pp. 36–41.isbn: 978-1-4503-2200-3.

[45] R. Smith, M. Self, and P. Cheeseman. “Estimating Uncertain Spatial Relationshipsin Robotics.” In: Autonomous Robot Vehicles. Ed. by I. J. Cox and G. T. Wilfong.New York, NY: Springer New York, 1990, pp. 167–193. isbn: 978-1-4613-8997-2.doi: 10.1007/978-1-4613-8997-2_14.

[46] Star Citizen. url: https://en.wikipedia.org/wiki/Star_Citizen (visited on03/11/2018).

[47] A. Stentz, D. Fox, M. Montemerlo, and M. Montemerlo. “FastSLAM: A FactoredSolution to the Simultaneous Localization and Mapping Problem with UnknownData Association.” In: In Proceedings of the AAAI National Conference on ArtificialIntelligence. AAAI, 2003, pp. 593–598.

[48] M. Sukan, C. Elvezio, O. Oda, S. Feiner, and B. Tversky. “ParaFrustum: Visual-ization Techniques for Guiding a User to a Constrained Set of Viewing Positionsand Orientations.” In: Proceedings of the 27th Annual ACM Symposium on UserInterface Software and Technology. UIST ’14. Honolulu, Hawaii, USA: ACM, 2014,pp. 331–340. isbn: 978-1-4503-3069-5.

[49] I. E. Sutherland and G. W. Hodgman. “Reentrant polygon clipping.” In: Commu-nications of the ACM 17.1 (1974), pp. 32–42.

[50] M. Trapp, L. Schneider, C. Lehmann, N. Holz, and J. Döllner. “Strategies forvisualising 3D points-of-interest on mobile devices.” In: Journal of Location BasedServices 5.2 (2011), pp. 79–99.

[51] M. Vlachos, G. Kollios, and D. Gunopulos. “Discovering similar multidimensionaltrajectories.” In: Data Engineering, 2002. Proceedings. 18th International Conferenceon. IEEE. 2002, pp. 673–684.

[52] H. Wang, H. Su, K. Zheng, S. Sadiq, and X. Zhou. “An effectiveness study ontrajectory similarity measures.” In: Proceedings of the Twenty-Fourth AustralasianDatabase Conference-Volume 137. Australian Computer Society, Inc. 2013, pp. 13–22.

[53] K. Weiler and P. Atherton. “Hidden surface removal using polygon area sorting.”In: ACM SIGGRAPH computer graphics. Vol. 11. 2. ACM. 1977, pp. 214–222.

[54] E. Williams. “Experimental designs balanced for the estimation of residual effectsof treatments.” In: Australian Journal of Chemistry 2.2 (1949), pp. 149–168.

56

Bibliography

[55] P. T. Zellweger, J. D. Mackinlay, L. Good, M. Stefik, and P. Baudisch. “City Lights:Contextual Views in Minimal Space.” In: CHI ’03 Extended Abstracts on HumanFactors in Computing Systems. CHI EA ’03. Ft. Lauderdale, Florida, USA: ACM,2003, pp. 838–839. isbn: 1-58113-637-4.

[56] F. Zhou, H. B. L. Duh, and M. Billinghurst. “Trends in augmented reality tracking,interaction and display: A review of ten years of ISMAR.” In: 2008 7th IEEE/ACMInternational Symposium on Mixed and Augmented Reality. Sept. 2008, pp. 193–202.doi: 10.1109/ISMAR.2008.4637362.

57