unit—a modular framework for interaction technique design ...aolwal/text/thesis.pdfvirtual, mixed...
TRANSCRIPT
1
May 13, 2002
Unit—A Modular Framework for Interaction Technique Design, Development and Implementation.
Unit—En modulär plattform för design, utveckling och implementation av interaktionstekniker.
Master’s project at the Department of Numerical Analysis and Computer Science at the Royal Institute of Technology (KTH), Stockholm, Sweden, executed in the Computer Graphics and User Interfaces Laboratory at the Department of Computer Science at Columbia University (CU), New York, USA.
Alex Olwal Author ([email protected])
Steven Feiner Kai-Mikael Jää-Aro Supervisor CU ([email protected]) Supervisor KTH ([email protected])
Lars Kjelldahl Examiner KTH ([email protected])
2
3
A B S T R A C T
This thesis presents the Unit Concept, a framework for the development, design and implementation of interaction techniques. Unit’s main goal is to stimulate the progress of new interfaces that go beyond the traditional keyboard/mouse setup. Virtual, Mixed and Augmented Reality are examples of new technologies that demand advanced interaction techniques with concepts such as proprioception, two-handed interaction and physical interfaces. These are practically impossible to represent within the traditional human-computer interface and new mechanisms are needed to enable their development.
Unit places the interaction techniques in an abstraction layer between the input devices and the application, which allows the application developer to separate application functionality from interaction technique and behavior. Unit’s modular approach leads to the design of reusable interaction control components that are independent of the application. Unit’s nature makes it possible to interactively change, tweak and experiment with the behavior of the interaction techniques, but also to switch among different input device configurations.
4
5
U N I T — E N M O D U L Ä R P L A T T F O R M F Ö R D E S I G N , U T V E C K L I N G O C H I M P L E M E N T A T I O N A V I N T E R A K T I O N S T E K N I K E R S A M M A N F A T T N I N G
Det här examensarbetet presenterar Unit, en plattform för utveckling, design och implementation av interaktionstekniker. Units huvudsakliga mål är att stimulera utvecklingen av nya gränssnitt som går bortom traditionella tangentbord/mus-gränssnitt. Nya tekniker såsom Virtual, Mixed och Augmented Reality kräver avancerade interaktionstekniker med begrepp såsom proprioception, tvåhandsinteraktion och fysiska gränssnitt. Dessa är praktiskt taget omöjliga att representera med det traditionella gränssnittet och nya mekanismer krävs för att möjliggöra deras utveckling.
Unit placerar interaktionsteknikerna i ett abstraktionslager mellan inmatningsenheterna (t.ex. mus och tangentbord) och programmet, vilket låter utvecklaren separera funktionalitet från interaktionsteknikerna och deras beteende. Units modulära lösning leder till design av återanvändbara komponenter för interaktionsteknik som är fristående från programmet. Unit gör det möjligt att interaktivt ändra, finjustera och experimentera med interaktionsteknikernas beteende, men också att byta mellan olika konfigurationer för inmatningsenheterna.
6
7
A C K N O W L E D G E M E N T S
Thanks to:
Steven Feiner, my advisor at Columbia University, for the guidance and inspiration;
Kai-Mikael Jää-Aro, my advisor at the Royal Institute of Technology, for the advice and critique;
All the people working in Columbia University’s Computer Graphics and User Interfaces Laboratory during May 2001 to December 2002, for the valuable discussions and arguments as well as the incredible experiences during the demonstrations at SIGGRAPH 2001 and ISAR ’01;
My family for their support and encouragement;
All my friends, for the hard questions and distractions…
This work was funded in part by ONR Grant N00014-99-1-0394 and N00014-99-1-0249 (S. Feiner, PI).
8
9
T A B L E O F C O N T E N T S
1 Introduction ............................................................................................. 13
2 Human-Computer Interaction .............................................................. 15
2.1 The Traditional 2D Interface— the Keyboard, Mouse and Screen ...........................................15 2.2 2D Interfaces for 3D Interaction ....................................................................................................16 2.3 3D Environments.............................................................................................................................16
2.3.1 Immersive Virtual Reality ....................................................................................................17 2.3.2 Mixed Reality/Augmented Reality .....................................................................................18 2.3.3 The Real Reality—Our Natural Environment ...................................................................18
2.4 Future Interfaces..............................................................................................................................18 2.5 Spatial Input .....................................................................................................................................19
3 Problem..................................................................................................... 21
3.1 The Way of Interacting ...................................................................................................................21 3.2 Building Interfaces ..........................................................................................................................21 3.3 Objective ...........................................................................................................................................22
4 Interaction Exploration .......................................................................... 25
4.1 Equipment ........................................................................................................................................25 4.1.1 Tracking Systems...................................................................................................................25 4.1.2 Input Devices .........................................................................................................................25 4.1.3 Software ..................................................................................................................................26 4.1.4 Display ....................................................................................................................................26
4.2 Tracker Utility Application ............................................................................................................26 4.2.1 Choosing the Tracker Locations ..........................................................................................27 4.2.2 Controlling the View.............................................................................................................27 4.2.3 Conclusions ............................................................................................................................28
10
4.2.4 Future Work ...........................................................................................................................29 4.3 Interaction Techniques— sketches, ideas and inspiration.........................................................29
4.3.1 Pointing...................................................................................................................................29 4.3.2 Virtual Hand ..........................................................................................................................31 4.3.3 Go-go.......................................................................................................................................31 4.3.4 Coarse-grain/Fine-grain Selection.......................................................................................31 4.3.5 Body Related Selection .........................................................................................................31 4.3.6 Speech .....................................................................................................................................33 4.3.7 Manipulation Factors ............................................................................................................33 4.3.8 Low-level Gestural Manipulation .......................................................................................34 4.3.9 Conclusion..............................................................................................................................34
4.4 Selection in Mobile Augmented Reality.......................................................................................34 4.4.1 Original Interface...................................................................................................................35 4.4.2 Proposed Interface.................................................................................................................36 4.4.3 Implementation Problems....................................................................................................38 4.4.4 Conclusions ............................................................................................................................38
4.5 The Cube Interface—Description and Sketches ..........................................................................39 4.5.1 Introduction............................................................................................................................39 4.5.2 Interface Examples ................................................................................................................39 4.5.3 A Cube Interface Scenario—The Creation of a Flashlight ...............................................41 4.5.4 Conclusions ............................................................................................................................43
4.6 Three Significantly Important Interaction Concepts ..................................................................43 4.6.1 Proprioception .......................................................................................................................43 4.6.2 Physical Interfaces .................................................................................................................43 4.6.3 Two-handed Interaction .......................................................................................................44
5 The Unit Concept .................................................................................... 45
5.1 Introduction......................................................................................................................................45 5.2 Related Work ...................................................................................................................................45 5.3 The Unit ............................................................................................................................................46
5.3.1 Properties and Connections .................................................................................................46 5.3.2 Methods ..................................................................................................................................46
5.4 The Unit Network ...........................................................................................................................47 5.4.1 Information Flow...................................................................................................................47
11
5.4.2 Interaction Abstraction Layer ..............................................................................................47 5.5 Making the Real World Interface Natural....................................................................................47 5.6 Issues .................................................................................................................................................47 5.7 Unit Examples..................................................................................................................................47
5.7.1 A Simple Unit Application—A Media Controller ............................................................48 5.7.2 Different Solutions for 3D Manipulation ...........................................................................50 5.7.3 Implementing the Go-go Interaction Technique...............................................................54 5.7.4 An Experimental 3D Unit User Interface ...........................................................................56
6 Conclusions.............................................................................................. 59
7 Future Work............................................................................................. 61
7.1 Unit User Interface ..........................................................................................................................61 7.2 Unit Network Library .....................................................................................................................61 7.3 Composite Input Devices ...............................................................................................................61
7.3.1 The Mouse as an Interaction Component ..........................................................................61 7.3.2 Related Work .........................................................................................................................62 7.3.3 Multiple Vertical and Horizontal Mice ..............................................................................63 7.3.4 Two Fixed Mice......................................................................................................................63 7.3.5 Mouse and Orientation Sensor ............................................................................................63
7.4 Cube Interface ..................................................................................................................................63 7.5 Curved Pointer.................................................................................................................................63
Abbreviations ............................................................................................. 65
References.................................................................................................... 67
12
13
1 I N T R O D U C T I O N
The evolution of new user interface technologies has the potential of introducing revolutionary changes in man-machine interaction. Technologies such as Virtual, Augmented and Mixed Reality (see Section 2.3 3D Environments) demand a different type of intuitive and natural interface, although desktop applications have also been subject to a focus on more advanced interaction with the increasing number of 3D applications. The design and implementation of these interfaces is not an easy task, and the developer is forced to understand the intrinsic mechanisms of human interaction.
The current research in 3D Human-Computer Interaction (HCI) revolves around interfaces, interaction techniques, input devices and software platforms for their implementation. The development of such a software platform requires a theoretical and practical experience of interfaces and interaction techniques.
To better understand the development, design, experimentation and implementation of interaction techniques, I decided to expose myself to the problems by implementing a 3D application with a 2D interface, studying and designing interaction techniques, analyzing and improving an existing 3D interface, and finally conceptually designing an Augmented Reality (AR) interface. I considered these to be essential experiences before beginning the design and implementation of a platform that would simplify such work.
The thesis work began thus with the development of the Tracker Utility, a motion capture application, developed to support the studies of how multiple tracked body parts could be used effectively for interaction in AR. The Tracker Utility helped identifying some practical problems with 2D desktop Graphical User Interfaces (GUIs) for 3D applications. A study of 3D interaction techniques was conducted for a deeper understanding of the issues in 3D HCI and Columbia University’s current AR interface was analyzed and an improved interface proposed.
The continuing study of 3D interaction literature made me discover an increasing number of interesting aspects of HCI. I consider proprioception, two-handed interaction and physical interfaces the most interesting new interaction principles and started to revolve my work around these. This resulted in the design and sketching of a physical interface, the “Cube interface”, which introduces some interesting interaction techniques.
The knowledge acquired from the abovementioned work laid the foundation for the “Unit Concept”, a modular framework that facilitates the design, implementation and experimentation phases in the development of interaction techniques. The interaction techniques are specified within a replaceable abstraction layer that resides between the application and the input devices. This isolates the interaction techniques and behavior from the application, allowing the
14
application to focus strictly on its functionality, while interaction techniques can be treated as modular components.
This approach allows the use of different interaction techniques and input devices in an application, as well as the tuning of and switching among them in runtime.
I will start by briefly reviewing some Human-Computer interfaces in Chapter 2, Human-Computer Interaction. The interaction technique development issues and the objectives of the work are discussed in Chapter 3. Chapter 4, Interaction Exploration, describes the conducted interaction studies, including the implementation of the Tracker Utility, the study of interaction techniques, the proposed redesign of an AR interface and the sketching of the Cube Interface. Chapter 5, The Unit Concept, thoroughly describes the framework. The closing discussion follows in Chapter 6 and Chapter 7 lists future directions for Unit.
15
2 H U M A N - C O M P U T E R I N T E R A C T I O N
2.1 The Traditional 2D Interface— the Keyboard, Mouse and Screen
As we use computers today, the three by far most common human-computer interaction devices are the keyboard, the mouse and the monitor. This interface has been the dominating de facto standard interface for over 20 years and the first prototype appeared already about 35 years ago [English et al. 1967, Engelbart and English 1968] (see Figure 1). The keyboard functions primarily as an obvious text-input device, the mouse as an intuitive 2D input device and the screen as the visual feedback device. The keyboard/mouse combination has proven to be a very powerful input device combination for 2D graphical user interfaces and has been able to maintain its position over the years as the standard interface on most platforms. This 2D GUI is also known as the WIMP interface from its elements—Windows, Icons, Menus and Pointer.
Compared to other aspects of computer hardware, a surprisingly evident lack of evolution is present in the keyboard/mouse interface. The keyboards look pretty much the same, except for some ergonomic enhancements and introduction of application-specific buttons. The mouse has perhaps been subject to most enhancements. It has been equipped with more buttons, scroll wheels, ergonomic design, and higher precision, but is still based on the same 2D input principle. The computer screens have about the same resolution as high-end raster displays of 20 years ago. In contrast to this, the development of memory, secondary storage, removable media and CPU speed have been enhanced in magnitudes of a 100–1000. Is there a reason for this evolutionary freeze in interaction devices?
Figure 1. An early keyboard, mouse and screen interface [English et al. 1967].
16
2.2 2D Interfaces for 3D Interaction The traditional 2D interface is often sufficient, but there is an increasing number of applications where there is a need for or advantage of visualization and/or interaction in 3D. In these cases, the degrees of freedom (DOF) that a standard 2D mouse provides has to be combined with the keyboard in some way in order to control the additional DOF that are usually needed for intuitive 3D interaction (see the examples in Figure 2). The 2D display also affects the interaction, since the user normally has to be able to change view from time to time, for example in order to understand 3D structures or to see obscured objects.
Figure 2. 3D applications. 3D Studio MAX R4 and Virtua Cop 2 [3D Studio Max R4 2000 and Virtua Cop 2 1997].
As Houde [1992] and Chen et al. [1988] show, it might be sufficient to use a 2D input device, such as a mouse, for interaction in 3D. The choice of input device heavily depends on the characteristics of the application. Limiting the available DOF is not necessarily bad—in many cases it helps to limit the number of usable DOF in HCI, depending on the task. For instance, if the current task is to translate an object along the x-axis in space, it might not make sense if it can simultaneously be rotated in all directions as well as translated along the y- and z-axes. It is often useful to restrict the interaction to fewer dimensions than the available DOF.
This is not the same thing as saying that it is sometimes better to use a 2DOF device than a 6DOF device—the more DOF a device has the more information is available. Liang and Green state the problem of insufficient DOF: “In particular, the traditional mouse-based interaction-style is a bottleneck to 3D geometric design, as it forces the user to decompose a 3D task into a series of 1D or 2D tasks.” [Liang and Green 1994]. However, it is sometimes better to use only the information provided by a subset of the available DOF. In general, one must use the available DOF with caution and be careful with extraneous DOF [Hinckley et al. 1994b].
2 .3 3D Environments To overcome the problems evident in 2D interfaces, many techniques have been developed that are more suitable for 3D interaction.
Different 3D input devices have been used to provide more straightforward interaction in 3D environments. These devices use a wide variety of techniques that allow advanced human-computer interaction. The main feature that most manufacturers try to provide is the intuitive and natural interaction in three dimensions. The Z-mouse and Spaceball are examples of such devices for desktop interfaces [Hearn and Baker 1997]. The Z-mouse is a regular three-button
17
mouse, which has been extended with a thumbwheel and a trackball on top to provide additional DOF. The Spaceball is a 6DOF device that does not move—instead the applied force on the ball is measured and interpreted as orientation and translation. Whereas the Spaceball is based on relative changes in acceleration, the mouse on relative changes in position, the Ascension Flock of Birds (FOB) [Flock of Birds 2002] and Polhemus Fastrak [Fastrak 2002] use an absolute reference frame. These systems are based on electromagnetic technology where a transmitter creates an electromagnetic field in which a number of receivers are tracked. The user can freely move and rotate a receiver in space and get information about its position and orientation with respect to the transmitter, which serves as the origin in the coordinate system.
The receivers are usually attached to an input device with buttons and a better ergonomic design, either commercial off the shelf, or custom-made. See the examples in Figure 3, 4 and 5.
Figure 3. The I3Stick. Microsoft
Sidewinder 3D Pro joystick combined with a 6DOF FOB sensor. [Brederson 1999].
Figure 4. The Cricket. A 3D input device
with an embedded 6DOF sensor, thumb joystick, pressure-sensitive buttons and vibrating functionality. Digital Image Design 1993 [Cricket 2002].
Figure 5. A custom-made “FOB-mouse” I
quickly created in 2001. The 2D mechanism is replaced by a 6DOF FOB sensor.
Stereoscopic displays allow the user to experience the scene in 3D by adding stereoscopic depth to the image. Showing each eye a slightly different view of the scene makes our perception use the disparity in the two views for depth clues to create an illusion of real 3D.
2 .3 .1 Immers ive Vir tual Real i ty Immersive Virtual Reality replaces the real world with a synthetic, virtual environment. Stereoscopic displays (see Section 2.3 3D Environments, above) are essential in the process of creating the illusionary immersion in the virtual environment.
There are mainly two methods used for stereoscopy in VR. One method is the use of a Head Mounted Display with two small displays that show each eye an independent image. Another method is the use of a single display where the different images are time-multiplexed. The screen uses alternating refresh cycles for each image. Stereoscopic glasses are synchronized with the cycles and filter out the corresponding image, so that one eye at a time sees the image. So when the left image is displayed on the screen, only the left eye sees through the glasses—the right eye sees nothing and vice versa.
Some sort of tracking is normally used, so that when the user looks around (e.g. by rotating his head), the view changes correspondingly. Tracking is also used for interacting with the virtual world. Attaching a tracker to the real hand, could for instance allow the interaction in the world with a virtual hand.
18
2 .3 .2 Mixed Real i ty /Augmented Real i ty Mixed and Augmented Reality combine real and virtual objects in a real environment. They take Virtual Reality one step further by allowing the user to interact with real and virtual objects simultaneously.
See-through displays are used to project virtual objects onto the real world. There are two main categories of see-through displays, optical and video. Optical see-through displays use a semi-transparent display that virtual objects are projected on, and thus overlaid onto the real world. Video see-through displays use cameras that provide a video stream of the real world. The virtual objects are then combined with the video, and the result is projected onto an opaque display. Video and optical see-through displays both have their respective advantages and disadvantages.
The system needs to know the user’s exact field of view at all times in order to correctly overlay virtual objects onto a real environment. Tracking the userʹs head position and orientation makes it possible to calculate the approximate position of the userʹs eyes and the direction in which they are looking. Finding the field of view, the registration problem, is one of the largest issues in the construction of MR/AR systems [Azuma 1997]. Different technologies are used to keep track of the user; these include electromagnetic, such as the Ascension Flock of Birds [Flock of Birds 2002], hybrid, such as the InterSense IS-600 Mark 2 Plus (IS-600) [IS-600 Mark 2 Plus 2002] and vision-based technologies. The IS-600’s hybrid tracking consists of a combination of accelerometers, gyros and ultrasound. The AR Toolkit, a software library for AR applications uses regular video cameras connected to a PC for tracking [Kato et al. 2000]. Azuma identifies “at least six classes of potential AR applications”: Medical visualization, Maintenance and repair, Annotation, Robot path planning, Entertainment and Military aircraft navigation and targeting [Azuma 1997].
2 .3 .3 The Real Real i ty—Our Natura l Environment Interaction techniques are obviously not only constrained to HCI. The increasing number of electronic devices that we work with in our everyday surroundings all have interfaces as well. Traditionally these have been and are in most cases buttons/switches and dials on a panel or remote control. Why are these interfaces so popular? One reason might be that placing a label on a button makes it pretty self-explanatory, which is something that seems to be applicable in HCI as well. It is also easy to manufacture a physical button with two discrete states. The dial is traditionally used for scaling entities (e.g. increasing/decreasing volume) and can often be replaced with two buttons.
There might be advantages in extending these everyday interfaces to not only use button presses and occasional turnings of a dial, but interaction techniques that are more intuitive, natural and common.
2 .4 Future Interfaces Nielsen lists 12 ways in how future interfaces should differ from the current [Nielsen 1993]:
1) The user’s focus will be to control the task domain, not to control the computer.
2) The computer’s role will be to dynamically interpret the user’s actions instead of blindly executing commands.
3) The computer, not the user, will be responsible for interface control.
19
4) No syntax will be required.
5) Some objects may be implicit and hidden, whereas their visibility is currently essential to direct manipulation.
6) Parallel streams from multiple devices will be possible.
7) The bandwidth will be relatively high compared to the current keyboard/mouse input.
8) Feedback will not be as straightforward to provide, due to higher interaction complexity.
9) Instead of a traditional turn taking, the system and user will work simultaneously.
10) The interface will be ubiquitous instead of just restricted to the keyboard, mouse and screen.
11) User programming will be simpler.
12) Software will be built upon independent plug-and-play modules.
Many issues on this, almost 10 years old, list have indeed been getting attention, some have even been resolved or subject to the attempt thereof. But the way we physically interact with a computer has not changed—the same primitive input techniques are used as when the list was created. Meanwhile, perhaps as compensation, the software has become more user-oriented. This is however not a lasting solution and the hardware interaction will ultimately have to change as well.
2 .5 Spatial Input Hinckley et al. [1994b] identified a set of principles that may facilitate 3D space perception:
• Spatial references
Using real-world objects as a reference to the virtual world make a significant difference for 3D interaction.
• Relative gesture
Hinckley et al. showed that working in a relative reference frame has shown to be more intuitive than in an absolute. A relative reference frame makes use of the relation between interaction elements to support the user. For example, it is easier to move an object a distance with respect to another object, rather than moving it the same distance without any reference.
• Two-handed interaction
This is a case of the above-mentioned relative gesture principle—the body and the two hands form a relative reference frame, which allows the user to make use of proprioception (see Section 4.6.3 Two-handed Interaction and Section 4.6.1 Proprioception, below).
• Multisensory interaction
The stimulation of multiple senses can provide an important aid to 3D perception. Audio feedback is more or less standard, whereas haptic and force feedback is mainly used for entertainment.
20
• Physical constraints
The usage of physical constraints provides a form of spatial reference. The Virtual Notepad uses a pressure-sensitive pen tablet with an attached 6DOF sensor to help the user take notes in 3D [Poupyrev et al. 1998]. Although possible, freehand writing in 3D space is not an intuitive task. This example of a 2D input device tracked by a 6DOF sensor, illustrates the advantage of physical support. The user can take notes by writing on the tablet and still easily position them in the 3D environment by moving the tablet. Proper use of physical support/restriction can in many cases facilitate the interaction.
• Head tracking techniques
Spatial feedback. Head tracking can for instance be used to provide depth cues with motion parallax.
Almost all of these principles relate to one another in some way—especially the first five are tightly coupled. An optimal interface makes use of the same interaction principles as in the real world. Real world objects with physical properties should assist the perception of the virtual environment—it is important to assist our cognition by stimulating the senses with hints in as many channels as possible. Not to forget is that these principles are not only valid for interaction in 3D, but most of them are also applicable to 2D.
21
3 P R O B L E M
3.1 The Way of Interacting The way we interact with machines and especially computers differ in many ways from the way we interact with people. There are many aspects of the human perception and subconscious, which remain unused. With emerging interfaces such as Virtual, Mixed and Augmented Reality, the HCI gap needs to be bridged.
3 .2 Building Interfaces The developer has many standards and guidelines to follow when designing 2D interfaces. The WIMP interface, controlled with a mouse and keyboard has been dominating for the last 10 years, at least. Similar conventions are not as evident in the design of interactive 3D environments and the evolvement of post-WIMP interfaces. Although attempts have been made to standardize interaction techniques for 3D environments many other factors have to be accounted for. For instance, the lack of universal 3D interaction devices often makes the interaction techniques dependent on the available hardware.
Logical Input Devices make the application independent of the specific hardware by letting the application specify the input data type, instead of specifying which physical input device to be used [GSPC 1977, Foley et. al 1984, Watt 2000]. Many different physical input devices can be used to input the same type of data; the use of logical input devices makes it possible to change hardware without modifying the application. But, the logical input devices do not describe the behavior or the functionality of the input devices. Since there is no reason to make a distinction between logical and physical input devices in the discussion of interaction techniques, “input devices” is used to refer to both of them.
Building a system, different types of interaction techniques may be suitable, depending on the application, input devices and task. It might make more sense to first identify the desired input and then identify suitable interaction techniques and input devices. Compromises are inevitable in such a methodology, but might still be better than the obedient adaptation of the interface to the available input devices. A common approach is simply to identify the input device (physical or logical) and then hardcode its behavior in the application. This tight bound of input device to the application makes it hard to reuse the interaction technique as well as change the type of input device.
For every input device-application combination the interaction technique may need to be modified and customized. Without a mechanism of doing this easily the developer usually attacks the problem with heuristic trial-and-error in modify-compile-run iterations.
22
The development of interaction techniques and their mapping to input devices and applications becomes a repetitive, tedious and ineffective process instead of the creative interaction design work it should be.
3 .3 Objective I propose a platform that facilitates the creation of interaction techniques to address these problems. The goal is to stimulate the creation of nontraditional interfaces that go beyond traditional keyboard/mouse-systems. The interfaces of interest are especially interfaces that:
• Are easily, naturally and intuitively manipulated
• Use multisensory interaction and encourages two-handed input
• Activate the user and stimulates creativity
• Blend the physical and virtual environments
• Make use of physical interaction (e.g. physical/graspable/tangible interfaces)
• Make use of proprioception
• Take advantage of 3D
• Exploit the relations between the involved interaction elements—user, devices and interface.
The desired result is a framework that supports flexible implementation of interaction techniques. The separation of application, interaction technique and input device is considered crucial for the generality. An abstraction layer between application and input devices would simplify this process (see Figure 6). With such a layer, different variants of interaction techniques could be implemented for an application or input device and tuned and switched amongst at runtime. This would greatly help the appreciation of the usability of a specific interface, since it normally has to be tested and compared with other candidates.
Application
Interaction Techniques
InputDevice A
InputDevice B
InputDevice C
Input device layer(physical and/or logical input devices)
Application layer
Interaction technique abstraction layerApplication
InputDevice A
InputDevice B
InputDevice C
a) b)
Figure 6. a) The traditional structure of an interactive application. The different input devices are mapped to functionality
on the application level, which makes it hard to change both devices and their behavior. b) The suggested solution is an abstraction layer between the input devices and the application. This layer can be changed for different combinations of interaction devices and behavior.
Such an abstraction layer should consist of replaceable modules that could simulate different functionality and process/filter the input data before passing it on to the application. The characteristics of the implementation process should be component-based, and the creation of the interface should remind of an assembly process, based on the reuse and customization of objects.
The platform should make it easy to experiment with interaction and it should be trivial to exchange an interaction technique or input device for another, especially at runtime. Interaction investigations, such as the ones made by Hinckley et al. [1998a] and Kurtenbach et al. [1996],
23
show examples of the experimental use of different interaction devices as input to novel interfaces. An abstraction layer would simplify similar work, putting the interaction related problems in focus. It would also help the design of new interaction devices composed of other devices, such as the Virtual Tricorder [Wloka and Greenfield 1995] and the I3stick [Brederson 1999].
The principles listed by Nielsen and Hinckley et al. are still uncommon and absent in most commercial interfaces (see Section 2.4 Future Interfaces and Section 2.5 Spatial Input, above). I consider it important, if not essential, to support them in a framework for implementation of interaction techniques.
24
25
4 I N T E R A C T I O N E X P L O R A T I O N
In order to better understand the issues in 3D interaction, I considered it important to get my hands dirty by practically experiencing the problems in the design and implementation of 3D interfaces. My work on a 3D motion capture application, the Tracker Utility, incorporated some evident problems in the implemented interface, primarily because of a traditional problem solving approach. I continued my work by designing interaction techniques, many of which I later found represented in the literature during my concurrent interaction technique study. I applied the acquired interaction technique knowledge in the implementation of a proposed redesign of an AR interface. Evolving ideas during this exploration resulted in the conceptual design of the Cube Interface, a physical interface with the intention of not only blending the virtual and real worlds, but also materialize abstract entities.
4 .1 Equipment Besides the workstation, a high-end PC with 3D graphics accelerator, the equipment used could be placed into the following four categories:
4 .1 .1 Tracking Systems The InterSense IS-600 Mark 2 Plus and the Ascension Flock of Birds were the primarily available tracking equipments during the time I worked in the Computer Graphics and User Interfaces Laboratory at Columbia University (CGUI lab).
The IS-600 uses a combination of ultrasound, gyros and accelerometers in a hybrid configuration to track 6DOF. Ultrasound is sufficient for the tracking of small 3DOF position sensors. The ultrasound requires line-of-sight between the receiver (microphone) and the transmitter (speaker). The setup had two 6DOF hybrid trackers and two 3DOF ultrasonic, position trackers.
The Flock of Birds uses electromagnetic technology to track four 6DOF sensors. The use of electromagnetism makes the system sensitive to metal in the surroundings; on the other hand no line-of-sight is required between the transmitter and the receiver.
The IS-600 was used together with the Flock of Birds in my AR experiments since it was part of the CGUI lab AR setup.
4 .1 .2 Input Devices The input devices consisted of different mouse variants primarily due to the advantage of their availability and low cost. The input devices were wireless mice, trackballs and gyro mice.
26
4 .1 .3 Sof tware The AR software and the Java drivers for the tracking equipment were provided by the CGUI lab, and a Java driver for supporting multiple USB mice was provided by Hourcade and Bederson [1999]. Java and Java3D were chosen because they were the standard programming language and 3D API used in the lab.
4 .1 .4 Display The CGUI lab’s see-through head-worn stereo display with a resolution of 800×600 pixels was used for AR. I experienced it hard to interact at close range due to the narrow field of view, which is approximately 30 degrees horizontally. It was hard to use the peripheral view to, for instance, see the hands.
The equipment is summarized in Table 1, below.
Table 1. Equipment. Type Specification
Tracking system Ascension Flock of Birds
InterSense IS-600 Mark 2 Plus
Display See-through head-worn display (Sony Glasstron LDI-D100B Stereo SVGA)
Input devices Wireless Optical Mice (Logitech Cordless Optical Mouse)
Wireless Trackball (Logitech TrackMan Live!)
Wireless GyroMice (Gyration GyroMouse)
Software Java & Java3D software
Augmented Reality Software (CGUI lab)
Device drivers
Other hardware High end PC with 3D Graphics Accelerator
4 .2 Tracker Util ity Application Some of my work was done as part of the collaboration between Columbia University and the Oregon Graduate Institute as an “attempt to integrate spoken and gestural interaction with indoor and outdoor augmented reality systems”, a project on Multimodal Input for Wearable Augmented Reality [Feiner 1999b].
I developed the Tracker Utility, a 3D motion capture application. By attaching a number of 6DOF trackers to different parts of the body and map these to a virtual model, one can record the motion and gestures of a person. The Tracker Utility provides the recording and playback of such data, visualized with a virtual model of a human. The Tracker Utility was developed to help the understanding of how information about multiple tracked body parts could be used to extract 3D gestures for use in a multimodal AR interface.
The Tracker Utility allows the user to study the recorded motion data by replaying it with a virtual model. The advantage over recording the motion with a video camera is the 6DOF that
27
the trackers directly provide, thus the motion data can be studied from any angle. This could also have been achieved with multiple video cameras and image analysis, but would have required a far more complex software solution. It also makes more sense to use the same equipment (6DOF trackers) in the Tracker Utility as in the target interface—the multimodal AR interface.
4 .2 .1 Choosing the Tracker Locat ions The determination of the optimal placement of the limited number of trackers on the body imposes a difficult problem. As only four trackers are available, it is important to have an intelligent setup that obtains as much information as possible from their configuration. The Tracker Utility helps solving this problem with a dynamic mapping of the tracker to a location on a body part (see Figure 7). The intention of the dynamic mapping is to make it easy to try out different tracker configurations. The current mapping is however not interactive, which makes it cumbersome to radically change the position of a sensor.
Figure 7. Virtual model of a tracked human. The small coordinate axes indicate the positions and orientations of the 6DOF
sensors. This sequence demonstrates some of the problems with the mapping of the trackers to body parts. For instance, all images have the hand separated from the arm.
4 .2 .2 Control l ing the View Another problem is the control of the model view. The advantage of the 6DOF data is that we can watch the motion replay from any angle, move and zoom in/out on the model. But controlling the view with a keyboard and mouse requires some method for choosing which of the 6DOF that the 2DOF, provided by the mouse, should control.
There is no obvious convention for the control of the involved 3D transformations and many 3D applications use their own mapping of the 2D input devices, which can be confusing to the user. The mouse and keyboard do not usually provide a straightforward way of rotating, zooming and moving the model on the display. The 2D input forces the user to decompose the 3D task into a series of 1D and 2D tasks (see Section 2.2 2D Interfaces for 3D Interaction and [Liang and Green 1994]).
28
Figure 8. Tracker Utility Application, Main Control
Window. The “Play” and “Record” menus activate the replay and recording of tracker data. “Realtime” shows the current mapping of trackers to the virtual model interactively. “Trackers” provide functionality for enabling and disabling trackers. “Settings” takes the user to the advanced setup of trackers. The main window also allows the user to change the data file and the VRML model for each tracker.
Figure 9. Tracker Utility Application, Advanced
Settings. Mapping of sensors to VRML models of body parts. It is currently a bit cumbersome to change the location of a sensor since the position is specified numerically. A direct manipulative approach would be more appropriate.
4 .2 .3 Conclus ions While providing the necessary functionality, some of the problems with the design and implementation of interactive 3D applications are evident with the Tracker Utility. A common mistake is the design of a 3D application as if it was a traditional 2D desktop application. The considered issues are “Which WIMP elements are needed?” and “How should the mouse and keyboard be used to control the application?” instead of the focus on the problem of straightforward interaction. Software Development Kits (SDK) have responsibility in this too; since they rarely support any UI types other than the WIMP based ones.
In this case, the naïve approach resulted in a typical WIMP interface that is basically just a GUI to the configuration file (Figure 8). Many parts of the WIMP based interface are appropriate, but the mapping of trackers to body parts makes little sense. Moving a tracker requires an iterative procedure to numerically identify the corresponding location on the virtual model (see Figure 9). The interactive placing of the trackers directly on a virtual model would be far better.
The view control uses a combination of button presses and mouse input to control the 6DOF. The code for the control behavior was borrowed from an example application and proved to not be particularly good. The user rotates the object by holding down a mouse button while moving the mouse. There are a few serious problems with the control. The mouse input maps directly to the object’s local coordinate system, removing consistency between input and visual feedback. The
29
way the mouse moves relates to the object, not to the view of the object. Thus the meaning of, for example, a vertical mouse movement constantly changes. I experienced it a bit confusing to control the view of the object after it had undergone a few rotations and was forced to mentally envision the transformations in order to predict the correct mouse movement. The user translates the view with the keyboard arrows, left and right mapped to the x-axis, up and down mapped to the z-axis. The translation is however done with respect to the camera and hence conflicts with the mouse control. This also makes it impossible to move the object/view on the y-axis.
The Tracker Utility could benefit from a more user-centered interface, which on the other hand, would most likely increase the complexity of the software. It fulfills its goals, while being somewhat of an expert interface.
4 .2 .4 Future Work It would be desirable to implement a more advanced playback control with features such as rewind, stop, pause, fast-forward, slow motion, etc. Interactive mapping of sensors to body parts with direct manipulation is eventually crucial for the usefulness of the application. Better view control can be achieved in numerous ways. Advanced techniques like passive props would be great for controlling the view. By embedding a 6DOF sensor in a physical human model, it would be intuitive and straightforward to manipulate the view of the virtual model by rotating and moving the physical model [Hinckley et al. 1994a]. Passive props could also be used to map the trackers to the model, by indicating their location with a tracked physical pointer on the physical model.
4 .3 Interaction Techniques— sketches, ideas and inspiration
This section briefly discusses some examples of selection and manipulation techniques. Many are ideas that I got during my work and later or concurrently identified in the literature.
4 .3 .1 Point ing For selecting objects in our environment, there is no doubt that pointing is the most intuitive technique and I see no reason to think that this would not apply to the virtual environments as well. There are many variants of the point-to-select method, where the ray casting technique is the simplest form.
4.3.1.1 Ray Casting The ray casting technique projects a ray into the direction that the user points in (similar to a physical laser pointer). The system reports every intersected object and usually selects the closest. The main problem with a ray is the difficulty to select small objects at a large distance, which led to the development of the flashlight technique (see Section 4.3.1.2 Spotlight/Flashlight, below).
4.3.1.2 Spotlight/Flashlight The flashlight technique is an extension of ray casting and uses a cone that emanates from the user’s hand [Liang and Green 1994]. As the distance grows, so does the radius of the selectable area. Liang shows that the flashlight method is better than ray casting in many cases.
30
An immediate comment is that two-handed pointing should be able to yield better and more intuitive interaction. Two-handed pointing could use the 3D position of each hand and the resulting vector between these two positions would determine the direction in which the flashlight is pointing. The length of the vector could be used to change the size or the length of the cone. It is easier to point by using the position of two hands than using position and orientation of one hand. The pointing breaks down to the relation of the two hands, which basically makes it equally easy to point in all directions. Mine et al. used a similar method for Two-handed flying, where the vector determined the direction and its length the speed [Mine et al. 1997b].
4.3.1.3 Indirect Pointing—The Curved Pointer Selecting by pointing is not always direct, i.e. we do not always point straight at the object of interest. This is especially evident when we want to point out an object to another person. When pointing to something not currently visible, e.g. behind a corner, a common gesture is to curve the pointing gesture to make it easier to understand. The curved gesture is often combined with movement to further clarify the intention. The indirect pointing is also common when there are many objects close to each other. We tend to point sideways, from the direction with most free space. This also occurs when it is hard to point straight at an object because its visible surface is relatively small.
Indirect pointing with a curved pointer could primarily facilitate pointing at large distances and the selection of objects with a small visible surface (from the current point of view). Pointing at a large distance is hard because a small angular error at the origin results in an error that rapidly increases with distance.
I suggest a two-handed pointing with a pointer that can be curved. This could facilitate the selection of objects that have a big selectable surface, but where the visible surface is practically too small to select with direct pointing. It would most likely be useful to limit the amount of allowable curvature, i.e., it makes little sense to use a curve that is bent more than 90 degrees. See Figure 10.
The curved pointer was implemented within the Unit framework (see Chapter 5 The Unit Concept) and tested in the CGUI lab’s AR environment with two Flock of Bird sensors as control points. It appeared to be hard to use, primarily because of the small field of view of the Head Mounted Display (see Section 4.1.4 Display). Further testing and revision of the implementation is necessary.
Figure 10. The Curved Pointer. Top view of a user selecting an obscured object. The circle represents the obscuring object and
the rectangle the object of interest. The distance between the hands map to the length of the pointer (might be a linear or non-linear mapping). In the left image, a sharp turn with a fixed angle determines the direction of the pointer; the right image uses a smooth bézier curve with the shape mapped to a predesigned curve. The user still controls two of the control points with his left and right hand.
31
4 .3 .2 Vir tual Hand The selection techniques above may be good for selection, but it is hard to use them for manipulation (rotation, translation and scaling) of objects. Using a virtual hand at the position of the user’s hand to select and manipulate items within the user’s reach is a popular solution to the problem of intuitive and straightforward interaction. Restricting the interaction to within an arm’s reach is however a problem since the user must use another technique, e.g. navigation, in order to select objects at a distance.
4 .3 .3 Go-go The Go-go interaction technique extends the virtual-hand metaphor by introducing non-linear mapping of the real arm to virtual arm [Poupyrev et al. 1996]. This allows the virtual arm to be extended and thus objects can be selected and manipulated at a distance, as if they were within the real arm’s reach. When the real arm is within a predefined distance to the body, the selection and manipulation works as for a regular virtual hand. The extension of the virtual arm occurs when the user stretches his real arm beyond this predefined distance, and a nonlinear mapping extends the virtual arm to allow the reach of objects far away (see also Section 5.7.3 Implementing the Go-go Interaction Technique).
4 .3 .4 Coarse-grain/F ine-gra in Selec t ion Using ray casting techniques for selection is a problem when the object to be selected is obstructed. An alternative to moving to another position or using other selection techniques is to provide a mechanism for the discrimination of objects intersected by the ray. The problem is that the selection becomes a two-step process, where a fine-grain selection follows a coarse-grain selection.
The “shish-kebab”-technique places all intersected objects on a “stick” and use a second method for selecting the object of interest—the second hand or button presses, for instance.
4 .3 .5 Body Related Se lect ion Selection based on the relations between multiple body parts and the involved objects relate in many ways to how we naturally interact. Two examples are listed below.
4.3.5.1 Line-of-sight Selection An early thought was to use the relation between the hands and eyes to select the object of interest. I sketched a bit on how these techniques could be used, which later seemed to resemble Pierce et al’s Framing Hands technique (see below).
Pierce et al. discuss Image Plane Interaction Techniques in 3D Immersive Environments, where a set of interaction techniques are presented that allow the user to interact with the 2D projections of objects [Pierce et al. 1997]. The techniques all use the user’s hands to specify 2D regions within which the 2D projection of the object of interest lies. The Head Crusher uses the space between the forefinger and thumb, Sticky Finger uses a single finger to point at objects, Lifting Palm uses the space above the palm and Framing Hands uses the space between two hands that “frame” the object. See Figure 11.
32
Figure 11. The Head Crusher, Sticky Finger, Lifting Palm and Framing Hands Image Plane Interaction Techniques [Pierce et al. 1997].
4.3.5.2 Multiple Volume Selection One of the low-level selection ideas in the Multimodal Input for Wearable Augmented Reality project was to make use of different portions of the body [Feiner 1999b]. By attaching different “volumes of interest” to body parts, the goal was to more correctly interpret the user’s intention with the help of gestures and speech (see Figure 12). For instance, two cones emanating from the eyes, a sphere around the hand, and a third cone coming out from the pointing hand define different regions of interest. If the user would say something as “Select this object”, it might make sense to assume that the user refers to an object that lies within the sphere around the hand. “Select that object” could on the other hand refer to an object pointed at with the cone from the hand—perhaps combined with the volume defined by the two cones representing the field of view. These ideas were developed at the CGUI lab during 2001.
Figure 12. Multiple Volume Selection on a model of Columbia University’s campus. Three body-related volumes select objects
in the scene. The sphere around the user’s hand selects the closest building (red). The cone representing the user’s view selects the second building (yellow). The cone emanating from the user’s hand selects both the second and third buildings (yellow).
33
4 .3 .6 Speech Speech might be an efficient interface in combination with direct manipulation. Even with a limited vocabulary, the flexibility over e.g. button presses is evident. Pointing at an object while saying: “move”, “rotate” or “scale” is a straightforward method for manipulating an object. Bolt and Herranz discuss an interface that makes use of speech, two-handed interaction and gaze where the multimodal input is used to help the system infer the user’s intentions, see Figure 13 [Bolt and Herranz 1992].
Figure 13. “The fishing’s great”. The example illustrates how speech, two-handed interaction and gaze
can be combined to better determine the meaning of the user’s action. The left and right images use the same speech and gesture, but gaze information reveals that the user in the right image refers to the size of the fish [Bolt and Herranz 1992].
4 .3 .7 Manipulat ion Factors
4.3.7.1 Specified Selection Factor There are many problems in 3D interaction, and selection ambiguity is one. Support for the facilitation of interaction can be incorporated into the objects themselves. For instance, a scalar value can be introduced, indicating the “selectability” of an object. Thus pointing at several objects, the most “important” get selection priority. To select low priority objects, one would have to make more effort in indicating the intention, e.g. pointing straight at the object.
4 .3.7.2 Physics It might be trivial to bring up, but the importance of physics is easy to neglect when designing interactive 3D environments. Our expectations are that objects behave as in the real world. Acknowledging physical properties of objects and making objects behave as we expect them to could help reducing distracting unnatural behavior. For instance, incorporating gravity and mass could be used to make big objects harder to move than small ones. Physical laws may be used although there is no underlying physical representation, simply to facilitate the interaction. At the same time physics may in many cases be unnecessary and just complicate the interaction. But one should be conscious of the potential effect that they can introduce.
34
4 .3 .8 Low-level Gestura l Manipulat ion For low-level gesture control, a hand-orientation technique was considered to potentially allow easy manipulation of objects. Determining the action by the way the user orients his/her hand seemed interesting (see Figure 14). Sturman thoroughly discusses whole-hand input in his Ph.D. thesis [Sturman 1992].
Figure 14. Low-level gestural manipulation. The two top images illustrate translation controlled by the orientation and
movement of the hand. The two bottom images illustrate scaling (lower left) and rotation (lower right) using two-handed gesture that makes use of the hand relations.
4 .3 .9 Conclus ion The choice of interaction technique is obviously dependent on the characteristics of the application. The Two-handed flashlight is, for instance, probably better than Go-go for selection, since I believe it to provide higher precision. It also reduces fatigue because there is no need of stretching out the arm. Go-go, on the other hand, provides a far better mechanism for manipulating objects.
The process of choosing a suitable set of interaction techniques is not an easy task, especially if the target application has a 3D interface. MR/AR/VR interfaces have many other issues that make the user even more sensitive to cumbersome interaction. The support for choosing and tweaking the techniques is uncommon and is currently usually done on the source code level.
4 .4 Selection in Mobile Augmented Reality The problems with the development of effective and intuitive 3D interfaces became more evident with the work on an interface for Columbia University’s Mobile Augmented Reality System (MARS) [Feiner et al. 1999a, Höllerer et al. 2001, Höllerer et al. 1999]. One MARS application, demonstrated at the Emerging Technologies (ET) floor at SIGGRAPH 2001, consists of a see-through head-worn display which overlays information onto the booths on the ET floor. A wireframe overlay the booth and a label with its name is attached to it. Information about each booth is available and can interactively be shown in the interface. The interface is controlled with a handheld wireless input device.
35
4 .4 .1 Orig inal Inter face The original interface uses a Logitech TrackMan Live!, a handheld trackball with three buttons (Figure 15), to navigate a mouse pointer in the view plane and select the label of the objects that were located beneath it [TrackMan Live 2002].
A window pops up with information about the object when the mouse button is pressed. At the bottom of the popup window further options are available, such as closing the window, viewing more information etc. The label and window management are taken care of by the view management to prevent labels and windows from overlapping [Bell et al. 2001, Bell and Feiner 2000].
Based on an informal analysis I identified several problems in this interface:
• The interaction with the trackball introduces non-intuitive cursor control.
The trackball is held in the hand in a horizontal position or in a vertical position downwards most of the time. Thus, rolling the ball respectively forward or downward moves the cursor up on the screen. Reporting a forward roll as a vertical position increment is most likely a heritage from the functionality of the desktop mouse.
• The way a 2D cursor in the image plane is used in this application conflicts with the AR experience. It creates a layer between the user and the environment and makes the mixing of realities harder to grasp. Although a 2D cursor seems inappropriate, Pierce et al. demonstrates a set of intuitive image plane interaction techniques (see Section 4.3.5.1 Line-of-sight Selection and [Pierce et al. 1997]).
• View management makes it hard to select objects
The way the view management is used in conjunction with the interactive parts of this interface is unfortunate. The problem lies in how the selection/picking is designed. The user uses the 2D cursor to select a label on an object and presses a button to view more information about it. This label is moving most of the time as the view changes, due to the settings in the view management. This makes it very hard to select an object unless the user stands absolutely still (the label can for instance suddenly jump to the other side of the screen if the user’s view changes). The same problem occurs when options are to be selected from a popup window. In this case, the mouse pointer is restricted to lie within the window, which is also moving around. This makes it hard to, for example, navigate to the bottom of the window and close the window.
• Selecting options/modes from the menu
By right clicking, a circular 2D menu is brought up. The pointer is restricted to the area of the menu—to close the menu; the user has to choose a “close” option. The options are selected by the translation of the pointer over the desired option and the pressing of a button. It takes several scrolls on the trackball to move the mouse over the option labels. This is a frustrating process that takes unnecessarily long time.
The lack of indication of the current location in the menu tree made menu navigation somewhat confusing.
• An important advantage of AR was not fully made use of—the ability to directly interact with objects, currently obscured by other objects. The WIM (World In Miniature, a miniature version of the user’s environment that allows interaction with the environment
36
from a birds-eye perspective [Stoakley et al. 1995]) that was available in the interface could be used for this purpose, but would still lack the natural aspects of direct pointing.
4 .4 .2 Proposed Inter face The intention of the proposed interface was the design and implementation of a very simple and intuitive interface, since it was intended to be used by novice users that had very little time for instructions (less than one minute).
Besides controlling the pointer, there is no advantage of using the continuous input data in this interface. Button presses should be sufficient to determine the desired action. It might sound complicated to handle all functionality with just three buttons but the relatively few available options make me believe that consistent assignment of standard operations to the buttons solves the problem. For each of the three buttons the following functionality is assigned:
• Left mouse button: Activate/Action/OK, etc. (compare to [RETURN])
• Middle mouse button: Next option (compare to [TAB])
• Right mouse button: Cancel and Show/hide menu (compare to [ESC])
I consider the flashlight selection technique the best candidate for this interface since there is no need for manipulation in the application (see Section 4.3.1.2 Spotlight/Flashlight and [Liang and Green 1994]). I believe the GyroMouse [2002], a device which senses change in orientation (Figure 16), to be fairly intuitive in this interface, since it is easy to see that, as one changes orientation of the GyroMouse, the corresponding change occurs with the flashlight.
Two other devices for controlling the flashlight were tried as well. With the abovementioned trackball, the principle is to “attach” the cone to the ball. For instance, by rolling the trackball forward, the cone rotates on the horizontal axis. The same principle is used with a TrackPoint (a common pointing device on laptops, that resembles a joystick, Figure 17)—pushing the stick forward results in a similar movement of the cone [TrackPoint 2002].
Figure 15. Logitech TrackMan Live! [Trackman Live 2002]
Figure 16. Gyration GyroMouse Pro
[GyroMouse 2002].
Figure 17. IBM TrackPoint [TrackPoint 2002].
However, the result from moving the track ball and track point horizontally is not obvious, which is the reason why the more intuitive GyroMouse is recommended. It greatly simplifies the interaction when the physics of the interaction devices map to the physics of the virtual objects. Manipulating virtual objects with real tools (input devices) is more intuitive, straightforward and predictable if it resembles the manipulation of real objects. The Virtual Tricorder exploits this in a one-to-one mapping of a physical tool to a virtual multipurpose tool [Wloka and Greenfield 1995].
37
The use of a 3D pointer makes the AR interface more convincing and it would probably be more natural than the current 2D pointing (an alternative is to modify the current 2D pointing using one of the image plane interaction techniques proposed by Pierce et al. [1997], which on the other hand demand more advanced tracking). Due to the unavailable hand tracking for the mobile user, the suggestion is to attach the cone to a fixed body-relative location, which still allows the user to control its orientation. The selection is made with respect to the geometry of each object, not the dynamically positioned labels. As one or multiple objects are intersecting with the flashlight, a list of these objects and the miniature versions of their solid geometric representations appears on the left hand side.
The selected object has its wire frame drawn in a different color and its miniature representation is rotating and drawn in the same color. The closest object gets selected by default.
One cycles through this list of selected objects by pressing the middle mouse button. Further information about the object is obtained by pressing the first mouse button, which results in a screen stabilized information pop-up window on the right-hand side.
The flashlight would by default be active for interaction. The menu appears as the right button is pressed. The middle button changes active choice and left button selects it, which in turn performs an action or opens a submenu.
The proposed interface was informally tested by some lab members and appeared to indeed have most of the abovementioned advantages over the current interface, see Figure 18.
38
Figure 18. The proposed selection technique. The origin of the selection cone is attached to a fixed location indicated by the
Columbia University crown. As the orientation of the GyroMouse is changed, so are the cone and the crown. The crown’s intrinsic 3D structure aids the understanding of the operation, since it sometimes can be hard to perceive the 3D movement of the cone. The example demonstrates the selection of obscured objects. The four objects on the left-hand side are miniature versions of the objects in the 3D scene that are intersected by the cone. The list of miniature objects is sorted according to the distance to the crown and colored with the brightness mapped to proximity. The currently selected object is indicated by a distinctly darker color in the scene and in the list. The user can change the current selection and thus select any object that is intersected by the cone.
4 .4 .3 Implementat ion Problems The greatest problem with the proposed interface was to convince the system main developer of the advantages with the interface and have it integrated into the system. The monolithic character of the application makes it hard to exchange parts, which obviously results in a problematic situation. It was practically impossible for anyone else than the main developer to make any major functionality changes. Switching between different interface functionality would result in a painful process of changing hard-coded parts in various places in the application. Even if the proposed interface would not provide any significant improvements, it would be a lot of work just to test it and if necessary switch back to the original interface. The use of a revision control system would have been helpful in some sense, although a more modular software structure would have been the key.
4 .4 .4 Conclus ions I believe that my proposed interface can be enhanced in many ways and I am not sure that all aspects of it are better than the original interface with respect to efficiency and simplicity. This cannot however be ultimately determined unless the two interfaces are compared side by side.
39
The tweaking of my proposed interface would introduce some low-level modifications of the code. The implementation of, for instance, a mix of the two interfaces would be a complicated task.
This reveals some important issues in interface development. An interface has to be designed with regard to some principles—in this case they are informal and the whole process is based on experience and intuition. As the interface is implemented, the designer usually tweaks and tests the interface before letting other people try it. More than one interaction technique may be suitable for the application and this process of figuring out what actually works best in a situation can quickly become tedious and frustrating.
The ability to modify interaction techniques and their parameters in runtime is desirable for the flexibility of iterative development. Interactive implementation of an interface makes more sense than an iterative modify-recompile-run process. Due to the nature of the problem, the last steps of interaction technique design should be interactive.
4 .5 The Cube Interface—Description and Sketches 4 .5 .1 Introduct ion I recognize the mergence of physical and virtual worlds one of the most interesting issues in HCI. How can a seamless integration of physical and virtual objects in both worlds be made? To partly answer this question, I began to sketch on an interface that revolved around this concept.
The Cube Interface is built by virtual and physical components in the virtual and physical world, where the core components are cubes. Small cubes are chosen because they can be precisely held and manipulated by the fingertips, which are our most delicate means of manipulation.
The cubes represent anything in the world. The cube is the basic element and used as the default interaction object, but could of course be exchanged for any shape. The cubes act as placeholders for not only concrete objects, but also abstract entities such as properties, relations, modes, references etc. The cubes allow direct access to their assigned entities, by rotation and exposure of the appropriate side (compare to a 3D radio button with 1–6 options).
Entities can be transferred between the sides of different cubes; combinations give rise to new entities, the combination of abstract entities is of special interest.
The Cube Interface could be realized with Augmented Reality, which could make use of 6DOF trackers to represent the physical cubes and overlayed graphics to project virtual cubes onto the user’s surroundings.
4 .5 .2 Inter face Examples The work with the Cube Interface resulted in a series of sketches and ideas, a few of which I describe here.
4.5.2.1 Sandbox The sandbox is a proposed interface where the user works with virtual and physical cubes. The user customizes the virtual cubes and when needed, they can be given a physical representation by being combined with a physical cube. The sandbox provides a library of cubes that provide different functionality. The cubes can for instance be used for direct manipulation, as mode selectors and as reference points.
40
Examples of potential cube types:
• Operation cube—Transform, Delete, Create, Select, “Freeze”.
• Transform cube—Rotation, Translation, Scale.
• Primitives—Cube, Sphere, Cylinder, etc.
• Mode cube—Holds properties such as snap, max/min virtual distance, etc.
• Cursor cube—A clone of the current operation, e.g. Scale.
• Navigation cube—Uses the transform on the transform cube for navigation in the environment.
• Avatar cube—Representation of a user.
• Magic lens cube (Camera, Mirror)—Shows an attached image of the view from the viewpoint.
• Physical Attributes cube—Mass, Gravity, etc.
The cubes could be ordered in hierarchical sets, which are a combination of the “classes” of cubes, modes, settings, etc. For example:
• Types
• Custom cubes
• Predefined objects
• Work modes
• Behaviors
One of the key challenges of the Sandbox is the design of an intuitive and extendable arrangement.
4.5.2.2 Relative Transforms Bolt and Herranz discuss two-handed interaction techniques where relative gesture is combined with speech to provide multi-modal natural dialogue [Bolt and Herranz 1992]. The relative gesture seems appropriate to use for the three basic transformations (rotation, translation and scale). The relation between two cubes determines the operation.
Rotation is achieved by rotating the cubes around each other—translation and scale, by moving the cubes relative to each other. In the latter case, some kind of discrimination is needed to indicate the operation of interest, e.g. button presses or speech (compare to Section 4.3.8 Low-level Gestural Manipulation).
41
4.5.2.3 Flashlight As stated earlier (see Section 4.3.1.2 Spotlight/Flashlight, above), the flashlight technique proposed by Liang and Green [1994] could benefit from the use of two-hands interaction. Two cubes serve as reference points, where their relative positions determine the direction of the flashlight, and the relative distance determines the selection distance (Figure 19).
Figure 19. Flashlight. The two black cubes control the flashlight. The vector between their positions determines the orientation of the
flashlight. A linear mapping to the length of the vector determines the selection area of interest (indicated by the dark band in the middle).
4 .5 .3 A Cube Inter face Scenar io—The Creat ion of a F lashl ight
It might be easier to understand the interface with a concrete example. The example involves three operations that could be triggered by any suitable mechanism of choice (e.g. speech). I leave the specification to a future design decision.
• Instantiate—Create a new cube of a certain type
• Assign—A cube can be assigned to be of a specific type
• Combine—The combination of two or more cubes
4.5.3.1 Creating the Geometry We need a control point (ref cube) to attach the cone to and then use a modifier cube to customize the radius and length.
1. Instantiate ref cube 1.
2. Assign to cone type (ref cube 1 will by default be locked in position and orientation to the tip of the cone).
3. Pick up modifier cube.
4. Combine with ref cube 1.
42
5. Select modify length by rotating the modifier cube, so that the length side is visible. The action on the side that is facing the user is made active. (the AR interface makes the labeling possible).
6. Activate & drag to scale. Deactivate.
7. Repeat 6 if necessary (to make very long flashlight).
8. Select modify radius by rotating the modifier cube (the AR interface makes the labeling possible).
9. Activate & drag to scale. Deactivate.
10. Repeat 8 if necessary (to make very wide flashlight).
11. Release modifier from flashlight & put it away.
4.5.3.2 Defining a Second Control Point Two control points enable two-handed interaction.
1. Instantiate ref cube 2.
2. Combine with ref cube 1.
3. Cone snaps axis to ref cube 2.
The orientation constraint on ref cube 1 is now released. The orientation of the cone is now constrained by the positions of ref cube 1 & 2, i.e. by the orientation of the vector their positions define. Note that the length of the vector is not used
4.5.3.3 Creating the Picker Setting the shape of a picker to the cone enables the cone to be used as a flashlight picker.
1. Instantiate ref cube 3.
2. Assign to picker.
3. Pick up modifier cube.
4. Choose Set shape by rotating the modifier cube (the AR interface makes the labeling possible).
5. Combine ref cube 1 or 2 (representing the cone) with ref cube 3 to set Picker’s shape.
43
4 .5 .4 Conclus ions The cube interface offers a hands-on, toolbench-like interface that we have drifted away from with the widespread use of the traditional HCI. The cubes are “magic” instruments that can be whatever the user wants them to—the tool, the object to be manipulated, a relation between the tool and the object, etc. They are used as 1D, 2D and 3D devices as suitable for every application.
Later on, I discovered that the Cube Interface had many similarities with Bricks [Fitzmaurice 1995]. The main differences is that the Cube Interface is designated for AR and uses the whole 3D environment while Bricks utilizes 2D surfaces, such as the ActiveDesk (a large horizontal desktop surface with a rear projected computer screen underneath the writing surface). The Cube Interface does not only relate to concrete objects and properties (the mapping of physical objects to virtual objects) but also to abstract and immaterialized properties.
The “materialization” of abstract properties is highly interesting, along with the investigation of how they can be integrated with a physical world and interacted with in Augmented Reality.
Consider the mapping of a physical object to, for instance, the distance between two objects. If this property can be picked up, moved and interacted with—how can that feature be used, and what meaning should we assign to it?
The Cube Interface is still a rough sketch and has many unresolved problems. Some major design decisions still need to be made about the interaction. It is to be considered a starting point for the design and implementation of a potentially interesting physical 3D interface.
4 .6 Three Significantly Important Interaction Concepts
Proprioception, two-handed interaction and physical interfaces are considered especially relevant and essential in the work of bringing the HCI closer to natural interaction. Proprioception allows us to “interact without looking”. Physical interfaces (also referred to as graspable and tangible interfaces) allow us to interact intuitively with the computer, as if it was part of the real world. Two-handed interaction is our natural way of interacting with the real world, our natural physical interface. Let us look closer at each of these three concepts.
4 .6 .1 Propr iocept ion Proprioception is our innate ability of knowing the relative positions and orientations of our body parts—e.g. where our hands are located relative to each other. Proprioception allows the body to be used as the base in a relative reference frame. An excellent example is the “Over-the-shoulder deletion” that Mine discusses in his dissertation [Mine 1997]. The interaction technique shows an intuitive gesture that makes use of body sense. To delete an object, the user simply throws it over the shoulder. The user retrieves the last deleted object by reaching behind the shoulder, treating the space as a clipboard.
4 .6 .2 Physica l Inter faces The human habitat is an environment full of physical objects, which we have learnt to interface effectively with. Still, concepts from interaction with real world objects are rarely taken advantage of in HCI. Building the interface of real world objects, instead of restricting ourselves to the keyboard, mouse and screen, introduces a new possibility of even more intuitive
44
interaction techniques. Embedding the interface in the environment’s physical object allows the user to focus on the task and not the interface, as Nielsen predicted [Nielsen 1993] (see Section 2.4 Future Interfaces, above).
Ishii and Ullmer discuss the Tangible User Interface (TUI) as an alternative to the traditional desktop GUI. “TUIs will augment the real physical world by coupling digital information to everyday physical objects and environments” [Ishii and Ullmer 1997]. The physical instantiation of GUI elements is suggested with the introduction of lenses, phicons, trays, phandles and instruments as replacements for windows, icons, menus, handles and widgets (Figure 20). The ultimate goal of a TUI is to seamlessly integrate the real and virtual worlds.
Figure 20. Comparison of Tangible UI (TUI) and Graphical UI (GUI) elements [Ishii and Ullmer 1997].
4 .6 .3 Two-handed Interact ion Most of our real world actions involve the simultaneous use of both hands, while the traditional HCI restricts the continuous input to one hand. Two-handed interaction is essential in natural interaction with a strong connection to proprioception and physical interfaces. Research on two-handed interaction states its importance as a vital component of future natural interfaces [Bolt and Herranz 1992, Buxton and Myers 1986, Kabbash et al. 1994, Kurtenbach et al. 1996, Leganchuk et al. 1998, Hinckley et al. 1994b, Hinckley et al. 1997, Hinckley et al. 1998a, Hinckley et al. 1998b].
45
5 T H E U N I T C O N C E P T
5.1 Introduction Abstracting functionality from application is commonly done through the use of a standardized set of operations that are shared among the applications. The WIMP interface standardized the interaction principles for desktop 2D applications, and attempts have been made to find a corresponding standard for 3D. The work revolves around interfaces, interaction techniques, input devices, and software platforms for their implementation. I chose to focus on the latter although a study of the interfaces, interaction techniques and input devices has been necessary to understand the whole.
Unit is an API (Application Programmer’s Interface) that provides uniform mechanisms for controlling and customizing the data flow in an application. This thesis focuses on the use of Unit as a layer between input devices and applications where interaction technique and behavior can be specified.
The intention of the Unit Concept is to introduce a framework that facilitates the design, experimentation and implementation of interactive applications. The Unit Concept uses simplicity to enforce a modular approach. Functionality is specified in self-contained units, which give rise to powerful behavior when they are connected into Unit Networks. These networks abstract the input behavior from the application and can act as replaceable black-box modules for the application control.
5 .2 Related Work The Unit Networks’ flowchart-like structures are in many ways similar to electric circuits. This approach is not new and I will briefly mention three systems based on similar principles.
The Self programming language’s UI acts as a browser and inspector on the object’s relations [Chang and Ungar 1990]. Self’s object-oriented structure is made up of connections between the objects that can be inspected with the 2D GUI.
Resnick describes the design of Behavior Construction Kits with LEGO/Logo [Resnick 1993]. Electronic Bricks, modified LEGO bricks such as actuators and sensors, are connected to one another in the compositional design of functionality and behavior (Figure 21 and 22). Resnick also discusses programmable bricks that go beyond the dedicated functions of the Electronic Bricks. The programmable brick is a universal brick with a limited number of inputs and outputs that are controlled by a downloaded program.
46
Figure 21. Electronic Bricks, modified LEGO bricks, such
as actuators and sensors, can be connected to create behavior [Resnick 1993].
Figure 22. The connection of two electronic bricks, a light sensor and a motor. The motor speed will increase with light and stop in darkness [Resnick 1993].
AlgoBlock [Suzuki and Kato 1993] lets the user program by connecting physical blocks, each representing a command (Figure 23). One of the intentions of this educational programming language was to stimulate the collaboration in programming. Manipulating physical blocks makes it natural to collaborate, since all users can be involved in the process of creating the program.
Figure 23. AlgoBlock users [Suzuki and Kato 1993].
5.3 The Unit Each unit represents an abstract information entity that contains methods, properties and connections. The units can be used independently, but the true power lies in the use of a network of connected units.
5 .3 .1 Propert ies and Connect ions Properties and connections are the nuts and bolts of the unit. A unit uses its properties to mirror the information that needs to be visible externally. A property can be either a value of any kind, or the output from a connection to another unit. A connection is a one-way information channel, a mapping from a source unit to a target unit. One can reference the information in a property not only by the property itself, but also recursively by the components, the methods and variables, of the property’s data type.
5 .3 .2 Methods The methods in a unit represent the information processing that take place when properties and connections are manipulated. The information processing methods are not visible externally; they are accessed through the properties and connections.
47
5.4 The Unit Network
5 .4 .1 Informat ion F low The property connections give rise to a flowchart-like information flow. It is a dynamic structure where changes are instantly propagated through the network of connected units. The individual functionality of the units determines how the information is passed. Whereas a connection is unidirectional, the information flow is not necessarily. The application can for instance pass information back to the Unit Network (e.g. to provide a parameter).
5 .4 .2 Interact ion Abstract ion Layer The units can be used as an interaction abstraction layer between the input devices and the application. Breaking it down, essentially what an input device does is to map its DOF to some functionality in an application. Instead of reading the input device and processing the data in the application, a unit network can be responsible for the information processing and act as a black box for the interaction technique. The interaction techniques can thus easily be replaced, compared and switched amongst—the nature of Unit allows each network to be reconfigured, tweaked and modified with the application running.
5 .5 Making the Real World Interface Natural Not only HCI needs to evolve, but also the way we are interfacing with the increasing number of physical equipment (besides computers) in our surroundings.
It is currently popular to use drag-and-drop tools to facilitate the creation of 2D GUIs. The developer simply determines which basic components he/she wishes to use and customizes them into a suitable combination that works on top of the application. The Unit Concept supports a similar process for creating physical interfaces where the components are the input devices. The input devices are mapped to a Unit network, where both the units and the network can be customized according to the application.
5 .6 Issues One of Unit’s current limitations is the interface, which is entirely code based (although an interactive prototype 3D user interface is available, see Section 5.7.4 An Experimental 3D Unit User Interface). So, for now the assembly of Units is made with Java code. The construction of Unit Networks is assumed to be an initial process that is done once in the beginning of the interaction implementation. The main part of the work then revolves around the iterative control, tweak and change of data flow in the Unit Networks. This, which I consider the most important feature, is of course possible to do interactively.
Unit also has a low level of error control and the result of malicious input is undefined at many places, which imposes a responsibility on the developer/designer of knowing what he/she is doing.
5 .7 Unit Examples This section demonstrates some of Unit’s features and each example is followed by a flowchart of the resulting Unit Network. The first example describes a simple media controller. Simplified
48
code accompanies it to help the understanding of the implementation. The second example describes three interaction techniques for 3D manipulation—it shows how Unit can be used to interactively switch between different interactive techniques and how they can coexist in a running application. The third example discusses the implementation of the Go-go interaction technique (see Section 4.3.3 Go-go and [Poupyrev et al. 1996]). It shows how parts of Unit Networks can be used to experimentally and interactively tweak interaction techniques.
5 .7 .1 A S imple Uni t Appl icat ion—A Media Control ler The media controller application shows how Unit can be used to easily experiment with interaction techniques. Here Unit is used to control the playback of a movie. The playback is controlled with two mice and the desired functionality is to have the distance between the mice determine the playback speed. The lack of absolute position (see Section 7.3.1 The Mouse as an Interaction Component) means that we rely on the mice not being too imprecise in position as well as not lifted up. We assume that the mice are initially close to each other. Figure 24 illustrates the information flow in the media controller application.
The included code has been simplified to make it easier to understand and is not complete.
MultipleMouseListener
unitMouseSensor1
position
unitMouseSensor2
position
unitArithmetic1
value1 value2
difference
0.001
Input device driver layer
Unit layer (Interaction abstraction)quotientproductsum
unitArithmetic2
value1 value2
difference quotientproductsum
setPlaybackSpeed(rate) Application layer
unitOutput
rate
rate
Figure 24. Information flow in the Media Controller Application.
The MultipleMouseListener represents the input device layer and reads the physical devices. The mice are represented by UnitMouseSensors in the Unit layer and are registered with a MultipleMouseListener. UnitMouseSensor mouse1 = new UnitMouseSensor(“Mouse 1”); UnitMouseSensor mouse2 = new UnitMouseSensor(“Mouse 2”); MultipleMouseListener multipleMouseListener = new MultipleMouseListener(); multipleMouseListener.addUnitMouseSensor(mouse1); multipleMouseListener.addUnitMouseSensor(mouse2);
The mouse position information is dynamically available as a property in the UnitMouseSensors and represented by a data type with two integer elements, x and y. The position information from
49
both UnitMouseSensors is connected to a UnitArithmetic’s value properties. There is no restriction on how many value properties a UnitArithmetic has, however in this example we only need two. The operations in a UnitArithmetic are performed componentwise. UnitArithmetic unitArithmetic1 = new UnitArithmetic(“unitArithmetic1”); unitArithmetic1.addConnection(mouse1, Unit.POSITION, unitArithmetic1.getNextValueName()); unitArithmetic1.addConnection(mouse2, Unit.POSITION, unitArithmetic1.getNextValueName());
The UnitArithmetics provide several functions that are applied to their input value properties, among these sum, difference, product and quotient. The results are stored in properties named correspondingly—the difference between the two mouse positions is available in the difference
property. A calculated property is dynamically updated as any of the input value properties changes, if it is connected to another Unit. The difference property is connected to another UnitArithmetic, where we add a scale factor as a value property, since we want to scale our data to suit the range used in the application. UnitArithmetic unitArithmetic2 = new UnitArithmetic(“unitArithmetic2”); unitArithmetic2.addValue(0.001); unitArithmetic2.addConnection(unitArithmetic1, UnitArithmetic.DIFFERENCE,
unitArithmetic2.getNextValueName());
The scaled horizontal difference between the mice can now be obtained by querying for the x element in the product property of the last UnitArithmetic. In order to provide a uniform interface to the application, we let all output pass through a final Unit with a name and property known to the application, we can for instance let the application pass the name as a parameter to the module that creates the network. final String RATE = “rate”; Unit unitOutput = new Unit(“unitOutput”); unitOutput.addConnection(unitArithmetic2, UnitArithmetic.PRODUCT, “x”, RATE);
This makes the playback speed available through the rate property in unitOutput. setPlaybackSpeed(unitOutput.getProperty(RATE).getValue());
To dynamically update the playback speed, we can instead define an anonymous Unit class in the application that bridges to the Unit layer. The playback speed will now be updated as soon as the product property is changed. Unit unitRate = new Unit() { public boolean changeProperty(UnitProperty property) { if (property.getName().equals(RATE)) { setPlaybackSpeed(property.getValue()); return true; } return false;
} }; unitRate.addConnection(unitOutput, RATE, RATE);
Since no distinction is made for the origin of the information, one could map another input device to the scale factor and interactively find a suitable scale factor. One could also implement several different coexisting Unit networks running simultaneously, for instance if one wishes to test different interaction techniques. The Unit framework can then be used to switch among which network’s output should actually affect the application. The units can of course be shared across the networks.
50
5 .7 .2 Di f ferent Solut ions for 3D Manipulat ion 3D Manipulation is essential in every 3D interface and can be described by the three transformations translation, rotation and scaling. How to intuitively provide the interface for these three transforms is a classic problem that has been subject to numerous solutions. A popular method is the use of the mouse’s direction of movement for the transformations, using the mouse buttons to distinguish the operations and axes of operation.
The Tracker Utility interface has several problems with the control of the view—the translation and rotation of the object (see Section 4.2 Tracker Utility Application and Section 4.2.3 Conclusions).
This example will describe three methods for controlling the view and show how Unit can be used to implement and compare them.
Using Unit as an abstraction layer between input devices and application, I decide that the application should take position and Euler angles as input. Three Unit networks are designed for each of the solutions. The main distinction between the three networks are the input devices, but Unit can of course be used to compare different interaction techniques for the same input device configuration. Each network is a standalone component with an output Unit containing the position and angle values.
5.7.2.1 Two Mice Two wireless optical mice with scroll wheels provide four continuous + two semi-continuous (the scroll wheels, actually have discrete states at every roll) DOF. Guiard states the difference between operations of the dominant and non-dominant hand in his principle of “Asymmetric scales of motion”. Guiard concludes that the dominant hand specializes in rapid, fine movement, while the non-dominant in slower, coarse movement [Guiard 1987]. Experience makes me believe that rotation requires higher precision than translation and thus assign this task to the right hand (which is the dominant hand in the case of a right-handed user). The movement of the left mouse is mapped to the vertical and horizontal components of the position (x and y) while the scroll wheel is used for the depth (z). Similarly the right mouse uses the same movement mapped to the rotation around each of the three axes. Horizontal movement maps to rotation around the y-axis, vertical movement to rotation around the x-axis and the scroll wheel to rotation around the z-axis. The not completely straightforward mapping of the scroll wheel is a compromise (physically it resembles a rotation around the x-axis) since it is assumed that the z-axis rotation is not as common as the other two.
The angle and position output data is relative and available in a Unit named unitTwoMice (See Figure 25).
51
MultipleMouseListener
unitMouseSensor1
velocity
unitArithmetic1
value1
unitTwoMice
angles
(0.000π, 0.000π, 0.00π)
product
unitArithmetic2
value1
mod
(2π, 2π, 2π)
unitMouseSensor2
velocity
unitArithmetic3
value1 value2
(0.001, 0.001, 0.01)
product
value2
value1
position
Figure 25. unitTwoMice Unit Network. Illustrates the information flow for a Unit Network where two mice are mapped to
translation and rotation.
5.7.2.2 One Mouse With a traditional setup, only one mouse is available; thus it is desirable to have a one-mouse solution as well. The functionality is similar to the two mice version, but the mouse operation is switched by the click of the left mouse button. The mouse operates translation until the left mouse button is clicked, which makes it operate rotation until the button is clicked again.
The output data is relative for the one mouse solution too, and is available in a Unit named unitOneMouse (See Figure 26).
52
MultipleMouseListener
unitMouseSensor1
velocity
unitArithmetic1
value1
angles
(−0.000π, −0.000π, 0.00π)
product
unitArithmetic2
value1
mod
(2π, 2π, 2π)
value2
value2
unitSelectorRotation
velocity
unitArithmetic3
value1 value2
(0.001, −0.001, 0.01)
product
position
unitSelectorTranslation
(0,0,0)
pause
(0,0,0)
pause
clicks_button1
change
clicks_button1
change
unitOneMouse
angles position
property_value property_value
Figure 26. unitOneMouse Unit Network. Illustrates the information flow for a Unit Network where one mouse controls
rotation and translation. UnitSelectors are used to select mode. (unitMouseSensor1 has duplicated boxes for its velocity and click_button1 properties. The two velocity boxes refer to the same property and the same stands for clicks_button1.)
5.7.2.3 Two 6DOF Sensors The third solution involves two 6DOF Ascension Flock of Birds sensors. One of the sensors represents the frame of reference that the view is changed with respect to. Hinckley et al. showed several advantages of spatial references and relative gesture that this solution tries to make use of (see Section 2.5 Spatial Input and [Hinckley et al. 1994b]). The angle and position output data is absolute and available in a Unit named unitFOB (See Figure 27).
MultipleFOBListener
unitFOB
position
unitArithmetic1
value1
difference
value2
unitArithmetic2
value1
difference
value2
unitFOBSensor1
position angles
unitFOBSensor2
position angles
angles
Figure 27. unitFOB Unit Network. Illustrates the information flow for a Unit Network where 6DOF FOB sensors are
mapped to translation and rotation.
53
5.7.2.4 Connecting the Solutions In order to compare the three solutions, they can easily be connected in a fourth Unit Network and switched amongst with UnitSelectors. The first mouse’s second button controls the switching of solution. Since the two mouse based solutions provide relative values while the FOB provide absolute values, we need a means to pass this information. Besides providing the value of the active property, the UnitSelector also supplies the property’s name. Thus the information on the data type can be made available by naming the properties “relative” or “absolute”. The four output values are hence angles, angles_type, position and position_type and available in a Unit named unitOutput (See Figure 28). It is now up to the application layer to use this information to create the appropriate visual feedback. This network was successfully connected into the Tracker Utility application and replaced the old view control, which had a number of problems (see Section 4.2 Tracker Utility Application).
MultipleMouseListener
unitOutput
relative
unitSelectorRotation
relative absolutechange relative
unitSelectorTranslation
relative absolute change
unitTwoMice
angles position
unitOneMouse
angles position
unitFOB
angles position
unitMouseSensor1
clicks_button2
property_value
angles
property_name
angles_type
property_value
position
property_name
position_type
Figure 28. A Unit Network for 3D manipulation. The mechanisms of Unit are used to switch among three solutions for 3D translation and rotation. A mouse controls the switching of current data provider with a click on the second button.
54
5 .7 .3 Implement ing the Go-go Interact ion Technique The Go-go interaction technique uses a combination of linear and non-linear mapping to allow a VR user to reach and manipulate objects at a distance (see Section 4.3.3 Go-go and [Poupyrev et al. 1996]). The virtual hand matches the movement of the real hand as long as the user operates his/her hand within a specified distance (D) to the body (Figure 29). The user can reach and manipulate objects far away by extending the arm beyond the specified distance, where a non-linear function determines the position of the virtual hand.
Figure 29. The Go-go interaction technique. vR
ris the vector from the body to the virtual hand and rR
ris the vector from the
body to the real hand [Poupyrev et al. 1996].
The position of the virtual hand is determined with the following function:
−+<
==otherwiseDRkR
DRifRRFR
rr
rrrv 2)()( ,
where Rr is the real hand’s distance from the body, Rv the virtual hands distance from the body, k is a constant coefficient in the non-linear function (0 < k < 1), and D the constant that determines where the non-linear mapping begins.
We determine that the input data to the application layer should be the position of the virtual hand, which our Unit Network thus should output. Looking closer at the technique, it is clear that two 3D positions need to be known, a position on the body and the position of the user’s hand. Further two constants are used, one that determines where the non-linear mapping begins (D) and one that determines the characteristics of the non-linear function (k). Not knowing suitable values beforehand, it might be an advantage to determine them experimentally and interactively.
Two Flock of Birds sensors seem suitable to track the two positions. A custom unitGogo is written, representing the Go-go function. It takes Rr, D and k as input and returns Rv. Subtracting the position of the hand sensor from the position of the body sensor returns Rr. D is interactively determined by having the user stretching out the arm and experimentally find a suitable border for linear/non-linear mapping. D is set to the current distance from the real hand to the body as long as mouse button 1 is pressed. One can change the value of D at any time by moving the hand to a desired distance and pressing mouse button 1. The constant k is determined similarly; its value is mapped to the horizontal position of the mouse. The constant is updated when mouse button 2 is pressed. The absolute value of the scaled horizontal position, modulus 1 is taken to make sure that k is in the correct range (0 < k < 1). We then use the output of unitGogo as a scale factor in determining if and how much the virtual hand should be extended. The position of the virtual hand is finally available in outputUnit (See Figure 30). The interactive control of the two constants can of course be replaced with numerical values when such have been determined.
55
MultipleFOBListener
unitGogo
Rr
position
unitFOBSensorChest
position
unitArithmetic1
value1
difference
value2
unitVector
length
endpoint
unitFOBSensorHand
MultipleMouseListener
button_1
unitMouseSensor
button_2
enable
k
unitSwitch
output
enableinput
D
unitSwitch
output
position.x
input
Rv
unitArithmetic5
value1
quotient
value2
unitArithmetic6
value1
product
value2
unitArithmetic4
mod
value2value1
unitArithmetic2
absolute
value1
unitArithmetic3
product
value2value1
1
0.001
position
(0,0,0)
unitOutput
virtual_hand_position
Figure 30. A Unit Network for the Go-go interaction technique. The unitFobSensors determine the distance from the real hand to the chest of the user. A custom-made unitGogo unit represents the Go-go function for determining the distance to the virtual hand. The unitMouseSensor is used for the interactive and experimental determination of appropriate values for the Go-go function constants (D and k). The input to the D and k properties of unitGogo can be replaced with appropriate numerical values, when such are known. The position of the virtual hand is available in unitOutput as the virtual_hand_position property.
56
5 .7 .4 An Exper imenta l 3D Uni t User Inter face I developed a prototype 3D UI with Java3D for the creation and connection of units mapped to graphical elements. The intention was to visually see how Unit could be used for 3D interaction. 3D graphics primitives (such as cubes, cones, cylinders, etc) as well as VRML models can be manipulated in the interface with operations that are interactively available through keyboard shortcuts. The graphics objects have Unit representations that, after instantiation, allow them to be part of a Unit Network. The interface examplifies how units, properties and connections can be created, modified and deleted interactively. Although the UI helps in the illustration of Unit’s features, it does not provide any visual representation of the Unit Networks. Only the 3D graphical elements have visual representations in the interface, there are no such representations for the units, connections or properties. They are instead manipulated with keyboard shortcuts and the feedback is provided as text. A direct manipulative approach would eventually be more appropriate.
The 3D UI proved useful throughout the development. It was in part inspired by the Cube Interface (See Section 4.5 The Cube Interface—Description and Sketches) and supports most of the functionality needed for its implementation. It was among other things used for the implementation of the curved pointer (See Section 4.3.1.3 Indirect Pointing—The Curved Pointer)
Figure 31, 32 and 33 illustrate a VRML model that is manipulated with two control points (represented as cubes, compare to Section 4.5.2.2 Relative Transforms). The model is scaled by the control points’ relative distance, its orientation is mapped to the orientation of the invisible line between the points, and its position is bound to the center on that line. The flowchart is illustrated in Figure 34.
Figure 31. Manipulating the relative distance of the cubes changes the scale of the house.
Figure 32. Rotating the cubes around each other makes the house rotate around the corresponding axis. Here, the right cube is moving
in the direction perpendicular to the image plane, which causes the house to rotate around its y-axis. The house is also enlarged since the relative distance of the cubes is increased.
Figure 33. The house is translated by moving both cubes in parallel, without changing their relative distance. The house is always
positioned exactly between the two cubes.
57
unitVector
position end_point
length vectorpositionorientation
unitModel
scale positionorientation
unitArithmetic1
quotient
value2value1
(2,2,2)
unitArithmetic2
sum
value2value1
unitControlPoint2
position
unitControlPoint1
position
Figure 34. A Unit Network illustrating how two control points can control orientation, scale and position of an object.
58
59
6 C O N C L U S I O N S
It is hard to experimentally design and implement interaction techniques today. One of the main reasons is that the traditional 2D interface standardized the interaction and input devices for desktop computers. 3D interfaces proved to demand more advanced interaction and suddenly tricks had to be made in order for the traditional interface to be useful. The 3D applications are still fairly uncommon for the common user, and do not have corresponding standards of the 2D applications’. 3D interfaces usually have their own solution for the use of the input devices, tailored to the application, whether it is mouse/keyboard, 6DOF trackers or something else. Binding the input devices close to the application makes it cumbersome to tweak, improve and experiment with interaction techniques. It also complicates their reuse. The work on the Tracker Utility, Interaction Techniques, the improved interface for the MARS and the Cube Interface made the potential advantage of an abstraction layer between the application and the input devices evident.
The Unit Concept makes the application focus on its input data and not on the input device. It draws lines between the application’s functionality, how it is controlled and what devices control it. It is now easy to switch the way an input device is used, as well as the configuration of input devices. The Unit Concept currently supports Ascension Flock of Birds and InterSense IS-600 Mark 2 Plus trackers as well as multiple USB Mice.
The Unit Concept is a flexible solution to the development and design of interaction techniques and the modular approach give rise to reusable components. The components that can be created can be considered from two perspectives. The “Input device perspective” considers a set of interaction techniques implemented for a specific configuration of input devices while the “Interaction technique perspective” considers a specific interaction technique implemented for different input device configurations.
I have primarily focused on the Unit Networks as input to the application layer, but Unit does also allow the application layer to output information to a Unit Network. This is essential in the case of the construction of interaction techniques where the input has to cooperate with application specific functionality, such as visual feedback. In fact, Unit makes no distinction at all in where the data comes from nor where it is going—each unit’s responsibility is simply to react on the changes in its properties. This flexibility enables for example, the interaction techniques for a configuration of input devices to be interactively customized with other input devices.
In the striving for an interface, platform, application and hardware independent solution, the visual feedback portions of the interaction techniques are currently not part of the Unit Concept. The visual feedback is application specific and might in the future be a separate Unit package for Java3D. The interface independence lies in Unit’s pure data flow control, the platform
60
independence in the Java implementation, the application and hardware independence in the nature of the abstraction layer.
Unit provides all the mechanisms for dynamic property and connection creation, modification and deletion—its modular structure makes it ideal for extension. From my experience, Unit’s structure makes it very suitable for the development of interaction techniques and can greatly reduce implementation time. Unit makes it possible to interactively explore the properties of an interaction technique and tweak its behavior as well as easily exchange its parts. The implementation problems reduce to the identification of suitable components for the Unit Network and their relations.
61
7 F U T U R E W O R K
7.1 Unit User Interface It would be desirable to have a simple interface for creating, modifying and deleting units, their properties and connections. Instead of creating the Unit Networks with Java code, it would be practical to use a tool that uses drag n’ drop to create Unit flowcharts and automatically generates the code. If implemented as a traditional 2D interface, it might resemble the SELF UI [Chang and Ungar 1990].
The desirable interface is not to be confused with the prototype 3D Unit User Interface described above (see Section 5.7.4 An Experimental 3D Unit User Interface). It provides functionality for interactive Java3D applications, but uses keyboard shortcuts instead of direct manipulation for Unit operations (such as creating a connection).
7 .2 Unit Network Library The Unit Networks need a representation that encapsulates them. For reuse, file formats would be suitable for the Unit and the Unit Networks.
A Unit Network Library could consist of a set of Unit Networks customized from the two perspectives of input devices and interaction techniques (see Chapter 6 Conclusions). Libraries could thus be categorized into input device specific, e.g. interaction techniques for a mouse, and interaction technique specific, e.g. 3D manipulation for different devices.
7 .3 Composite Input Devices I have had several ideas regarding input devices during my work with the Unit Concept. One of the main problems with 3D interfaces is the unavailability of advanced input devices—6DOF trackers are for instance unaffordable to most users. The Unit Concept can take information from several input devices and process them in any desirable fashion. This flexibility for combining input devices, led to the idea of using the information of several input devices as an alternative to expensive and specialized trackers. This solution is hidden from the application thanks to Unit’s role of abstraction layer.
7 .3 .1 The Mouse as an Interact ion Component The mouse, being a fairly inexpensive and extremely common input device, makes it a suitable component in the construction of composite input devices. The mouse reports a relative change of position on the surface, which it is moved on. It provides the state of at least one button,
62
usually three, and occasionally a scrolling wheel. An important feature is the availability of wireless technology.
A mouse merely reports which direction it is moved in, and approximately how fast. It is thus practically impossible to infer absolute position information from a mouse because the approximate accuracy and orientation unawareness. Solving these two issues would result in a relatively inexpensive wireless self-contained tracker that could be used as a phicon [Ishii and Ullmer 1997] for instance.
7.3.1.1 Accuracy The mechanical construction has problems accurately determining the exact traveled distance; this is however becoming a smaller problem with the development of high precision optical mice.
7 .3.1.2 Orientation The other and biggest problem is the sensitivity to orientation changes. The mouse has no way of determining if it is rotated while being translated—thus a small rotation will fool the mouse to believe that it is being translated in another direction.
7 .3 .2 Rela ted Work Balakrishnan et al. developed the Rockin’ Mouse, a mouse that has been modified to sense four DOF [Balakrishnan et al. 1997]. The Rockin’ Mouse provides two additional DOF by having a rounded bottom, which allows it to be tilted left/right and forward/backwards (Figure 35).
Figure 35. The Rockin’ Mouse [Balakrishnan et al. 1997].
Hinckley et al. use a camera-based solution to sense six DOF with their VideoMouse [Hinckley et al. 1999]. The VideoMouse senses rotation around its axis and the distance to the surface (up to three cm). The other four DOF are acquired with a similar construction to the Rockin’ Mouse. The TouchMouse is a modified IntelliMouse that uses capacitance to sense when it is touched by the user [Hinckley et al. 1998a] (See Figure 36).
Figure 36. The VideoMouse (left and middle) uses a camera to obtain six DOF [Hinckley et al. 1999]. The TouchMouse (right)
senses the user’s touch [Hinckley et al. 1998a].
63
7 .3 .3 Mult ip le Vert ica l and Horizonta l Mice An idea is to use multiple mice in an application [Hourcade 1999]. Using several mice allows the exploitation of the advantages of two-handed interaction and provides the application with twice as many continuous DOF. Single Display Groupware (SDG), such as KidPad, where several users concurrently interact using the same display, benefits from the use of multiple mice [Hourcade 1999]. Besides using mice on a horizontal surface, wireless optical technology allows the use of mice on any planar surface. Mounting mice on vertical surfaces, for instance with magnets on a whiteboard, could have several advantages. For instance, in a 3D modeling application, one mouse could be used for translations in the XZ-plane together with another vertically mounted mouse controlling the vertical movement. This setup implies a number of problems. First, we actually have four continuous DOF that are to be mapped to three DOF. The problem has a number of solutions; we could just ignore one of the DOF or combine two DOF by mapping them both to one, etc. Another problem is the above-mentioned orientation problem (Section 7.3.1 The Mouse as an Interaction Component). With a desktop mouse, this is not such a big problem, because the mouse is normally positioned ready to be used and facing the approximately right direction. With a vertically mounted mouse, the same issues have to be regarded to solve the problem. Gravity could for instance be used to assist the mouse in maintaining its orientation.
7 .3 .4 Two Fixed Mice A problem with two mice for two-handed input is that it is hard to exploit the relation between the mice due to their non-absolute nature. However fixing two position sensors with respect to each other’s orientation can help in getting more adequate relation data.
The construction of an input device, consisting of two position sensors (e.g. from two mice) might resolve this problem. Depending on the precision of the sensors, this can create a redundant solution where the orientation information can be extracted from the two reported positions. The number of position sensors could of course be increased for better redundancy.
7 .3 .5 Mouse and Orientat ion Sensor Combining an orientation sensor with a mouse or trackball could be very interesting. Depending on the accuracy, ultimately an absolute positional input device could be created with a mouse and orientation sensor.
A GyroRemote [GyroRemote 2002] (senses changes in horizontal and vertical orientation and maps these to position) could be used as a small and inexpensive orientation sensor.
7 .4 Cube Interface The Cube Interface (see Section 4.5 The Cube Interface—Description and Sketches) needs to be refined and mature before it can be implemented with the Unit Concept. The prototype 3D UI could be used to assist this process (See Section 5.7.4 An Experimental 3D Unit User Interface).
7 .5 Curved Pointer The implementation of the Curved Pointer (see Section 4.3.1.3 Indirect Pointing—The Curved Pointer) needs to be revised and more consideration has to be made to the current CGUI lab setup in order to improve it. It would also be desirable to perform user studies if it informally proves to be useful.
64
65
A B B R E V I A T I O N S
1D—One-dimensional
2D—Two-dimensional
3D—Three-dimensional
API—Applications Programmer’s Interface
AR—Augmented Reality
CGUI lab—Columbia University’s Computer Graphics and User Interfaces Laboratory
CPU—Central Processing Unit
DOF—Degree(s) Of Freedom
ET floor—The Emerging Technologies floor at SIGGRAPH 2001
GUI—Graphical User Interface
HCI—Human-Computer Interaction
HMD—Head Mounted Display
MARS—Columbia University’s Mobile Augmented Reality System
MR—Mixed Reality
SDG—Single Display Groupware
SDK—Software Development Kit
TUI—Tangible User Interface
UI—User Interface
USB—Universal Serial Bus
VE—Virtual Environment
VR—Virtual Reality
VRML—Virtual Reality Modeling Language
WIM—World In Miniature
WIMP—Windows, Icons, Menus and Pointers
66
67
R E F E R E N C E S
3D Studio Max R4. http://www.discreet.com/products/3dsmax/. April 2002.
Azuma, R. A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4. 355–385. August 1997.
Balakrishnan, R., Baudel, T., Kurtenbach, G., Fitzmaurice, G. The RockinʹMouse: Integral 3D Manipulation on a Plane. Proceedings of Human factors in Computing Systems (CHI ʹ97). 311–318. 1997.
Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., Buxton, W. Digital Tape Drawing. Proceedings of User Interface Software and Technology (UIST ’99). 161–169. 1999.
Beaudouin-Lafon, M. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. Proceedings of Human factors in Computing Systems (CHI 2000). 446–453. April 2000.
Bell, B., Feiner, S. Dynamic Space Management for User Interfaces. Proceedings of User interface software and technology (UIST 2000). 239–248. 2000.
Bell, B., Feiner, S., Höllerer, T. View Management for Virtual and Augmented Reality. Proceedings of User Interface Software and Technology (UIST 2001). 101–110. November 2001.
Bier, E., Stone, M., Fishkin, K., Buxton, W., Baudel, T. A Taxonomy of See-Through Tools. Proceedings of Human factors in Computing Systems (CHI ʹ94). 358–364. 1994.
Billinghurst, M., Kato, H. Collaborative Mixed Reality. Proceedings of International Symposium on Mixed Reality (ISMR ʹ99). 261–284. 1999.
Billinghurst, M., Kato, H. Real World Teleconferencing. Paper presented at Proceedings of Human factors in Computing Systems (CHI ʹ99), Conference Companion. 1999.
Billinghurst, M., Kato, H., Weghorst, S., Furness, T. A. A Mixed Reality 3D Conferencing Application. Technical Report R-99-1, Seattle: Human Interface Technology Laboratory, University of Washington. 1999.
Bolt, R. A., Herranz, E. Two-Handed Gesture in Multi-Modal Natural Dialogue. Proceedings of User Interface Software and Technology (UIST ’92). 7–13. 1992.
Bowman, D., Hodges, L.F. An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments. Proceedings of Symposium on Interactive 3D Graphics. 35-38. 1997.
68
Bowman, D.A, Hodges, L.F. Formalizing the Design, Evaluation, and Application of Interaction Techniques for Immersive Virtual Environments. Journal of Visual Languages and Computing, 10. 37–53. 1999.
Brederson, J. The I3Stick: An Inexpensive, Immersive, Interaction Device. Technical Report, UUCS-99-016. University of Utah. November 1999.
Buxton, W., Myers, B. A Study in Two-Handed Input. Proceedings of Human factors in Computing Systems (CHI ʹ86). 321–326. 1986.
Card, S., Mackinlay, J., Robertson, G. The Design Space of Input Devices. Proceedings of Human factors in Computing Systems (CHI ‘89). 117–124. 1989.
Chang, B-W., Ungar, D. Experiencing SELF Objects: An Object-Based Artificial Reality. The Self Papers, Computer Systems Laboratory, Stanford University. 1990.
Chen, M., Mountford, S. J., Sellen, A. A Study in Interactive 3-D Rotation Using 2-D Control Devices. Proceedings of the 15th annual conference on Computer Graphics (SIGGRAPH ’88). 121–129. August 1988.
Conner, D., Snibbe, S., Herndon, K., Robbins, D., Zeleznik, R., van Dam, A. Three-Dimensional Widgets. 1992 Symposium on Interactive 3D Graphics. 183–188. 1992.
Cordless Optical Mouse. Logitech. http://www.logitech.com/. April 2002.
Engelbart, D.C., English, W. A Research Center for Augmenting Human Intellect. AFIPS Conference Proceedings of the 1968 Fall Joint Computer Conference, San Francisco, CA,Vol. 33. 395–410. December 1968.
English, W., Engelbart, D., Berman, M. Display-Selection Techniques for Text Manipulation. IEEE Transactions on Human Factors in Electronics, Vol. HFE-8, No. 1. 5–15. March 1967.
Fastrak. Polhemus. http://www.polhemus.com/. April 2002.
Feiner, S. Multimodal Input for Wearable Augmented Reality. ONR Grant N00014-99-1-0394. 1999.
Feiner, S. STARS: Software Technology for Augmented Reality Systems. ONR Grant N00014-99-1-0249. 1999.
Feiner, S., MacIntyre, B., Hollerer, T. Wearing It Out: First Steps Toward Mobile Augmented Reality Systems. In First International Symposium on Mixed Reality (ISMRʹ99). 1999.
Fitzmaurice, G. W., Balakrishnan, R., Kurtenbach, G., Buxton, B. An Exploration into Supporting Artwork Orientation in the User Interface. Proceedings of Human factors in Computing Systems (CHI ‘99). 167–174. May 1999.
Fitzmaurice, G. W., Buxton, W. An Empirical Evaluation of Graspable User Interfaces: towards specialized, space-multiplexed input. Proceedings on Human factors in Computing Systems (CHI ’97). 43–50. March 1997.
Fitzmaurice, G.W., Ishii, H., Buxton, W. Bricks: Laying the Foundations for Graspable User Interfaces. Proceedings of Human factors in Computing Systems (CHI ʹ95). 442–449. 1995.
Flock of Birds. Ascension Technology Corporation. http://www.ascension-tech.com/. April 2002.
Foley, J.D., Wallace, V.L., Chan, P. The Human Factors of Computer Graphics Interaction Techniques. IEEE Computer Graphics and Applications, 4 (11). 13–48. 1984.
69
Forsberg, A., Herndon, K., Zeleznik, R. Aperture Based Selection for Immersive Virtual Environments Proceedings of User Interface Software and Technology (UISTʹ96). 95-96. 1996.
Fröhlich, B., Plate, J., Wind, J., Wesche, G., Göbel, M. Cubic-Mouse-Based Interaction in Virtual Environments. IEEE Computer Graphics and Applications, 20(4). 12-15. 2000.
Gorbet, M. G., Orth, M., Ishii, H. Triangles: Tangible Interface for Manipulation and Exploration of Digital Information Topography. Conference proceedings on Human factors in Computing Systems. 49–56. April 1998.
GSPC. Status Report of the Graphics Standards Planning Committee. Computer Graphics, 11(3). 1977.
Guiard, Y. Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model. Journal of Motor Behavior, 19, 4. 486–517. 1987.
GyroMouse. Gyration. http://www.gyration.com/. April 2002.
GyroRemote. Gyration. http://www.gyration.com/. April 2002.
Hearn, D., Baker, M. Computer Graphics, Second Edition, C Version. Prentice Hall. ISBN0-13-530924-7. 1997.
Hinckley, K., Czerwinski, M., Sinclair, M. Interaction and Modeling Techniques for Desktop Two-Handed Input. Proceedings of User Interface Software and Technology (UIST ’98). 49–58. 1998.
Hinckley, K., Pausch, R., Goble, J.C., Kassel, N.F. Passive Real-World Interface Props for Neurosurgical Visualization. Proceedings of Human factors in Computing Systems (CHI ʹ94). 452–458. April 1994.
Hinckley, K., Pausch, R., Goble, J.C., Kassell, N.F. A Survey of Design Issues in Spatial Input. Proceedings of User Interface Software and Technology (UIST ’94). 213–222. November 1994.
Hinckley, K., Pausch, R., Proffitt, D., Kassell, N. Two-handed Virtual Maniulation. ACM Transactions on Computer-Human Interaction, 3(5). 260–302. September 1998.
Hinckley, K., Pausch, R., Proffitt, D., Patten, J., Kassell, N.F. Cooperative bimanual action. Proceedings on Human factors in Computing Systems (CHI ’97). 27–34. March 1997.
Hinckley, K., Sinclair, M., Hanson, E., Szeliski, R., Conway, M. The Videomouse: A Camera-Based Multi-degree-of-freedom Input Device. Proceedings of User Interface Software and Technology 1999 (UIST ’99). 103–112. 1999.
Houde, S. Iterative Design of an Interface for Easy 3-D Direct Manipulation. CHI ʹ92: Human Factors in Computing Systems: Striking A Balance. 135—142. May 1992.
Hourcade, J., Bederson, B. Architecture and Implementation of a Java Package for Multiple Input Devices. Technical Report, CS-TR-4018, UMIACS-TR-99-26, Human-Computer Interaction Lab, University of Maryland. May 1999.
Höllerer, T., Feiner, S., Hallway, D., Bell B., Lanzagorta, M., Brown, D., Julier, S., Baillot, Y., Rosenblum, L. User Interface Management Techniques for Collaborative Mobile Augmented Reality. Computers and Graphics, 25(5). 799–810. October 2001.
Höllerer, T., Feiner, S., Terauchi, T., Rashid, G., Hallway, D. Exploring MARS: Developing Indoor and Outdoor User Interfaces to a Mobile Augmented Reality System. Computers and Graphics, 23(6). 779–785. 1999.
70
IS-600 Mark 2 Plus. InterSense. http://www.isense.com/. April 2002.
Ishii, H., Ullmer, B. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proceedings of Human factors in Computing Systems (CHI ʹ97). 1997.
Kabbash, P., Buxton W., Sellen, A. Two-Handed Input in a Compound Task. Proceedings of Human factors in Computing Systems (CHI ʹ94). 417–423. 1994.
Kato, H., Billinghurst, M., Poupyrev, I. ARToolKit version 2.52. http://ftp.hitl.washington.edu/research/shared_space/download. April 2002.
Kitamura, Y., Itoh, Y., Kishino, F. Real-time 3D Interaction with ActiveCube. Proceedings of Human factors in Computing Systems (CHI 2001). 355–356.
Krumm, J., Shafer, S., Wilson, A. How a Smart Environment Can Use Perception. Workshop on Perception for Ubiquitous Computing (UBICOMP 2001). October 2001.
Kurtenbach, G., Fitzmaurice, G., Baudel, T., Buxton, B. The Design of a GUI Paradigm Based on Tablets, Two-Hands, and Transparency. Proceedings on Human factors in Computing Systems (CHI ’97). 35–42. March 1997.
Leganchuk, A., Zhai, S., Buxton, W. Manual and Cognitive Benefits of Two-Handed Input: An Experimental Study. Transactions on Human-Computer Interaction, 5(4). 326–359. 1998.
Leibe, B., Starner, T., Ribarsky, W., Wartell, Z., Krum, D., Singletary, B., Hodges, L. The Perceptive Workbench: Toward Spontaneous and Natural Interaction in Semi-Immersive Virtual Environments. Proceedings of IEEE Virtual Reality Annual Symposium (VRAIS 2000). 13–20. 2000.
Liang, J., Green, M. JDCAD: A Highly Interactive 3D Modeling System. Computers and Graphics, 18(4). 499–506. 1994.
Mine, M. Exploiting Proprioception in Virtual-Environment Interaction. Ph.D. Thesis, Department of Computer Science, University of North Carolina at Chapel Hill. 1997.
Mine, M., Brooks, F., Sequin, C. Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction. Proceedings of SIGGRAPH ʹ97. 19–26. 1997.
Nielsen, J. Noncommand User Interfaces. Communications of the ACM, 36 (4). 83–99. April 1993.
Patten, J., Ishii, H., Hines, J., Pangaro, G. Sensetable: A Wireless Object Tracking Platform For Tangible User Interfaces. Proceedings of Human factors in Computing Systems (CHI 2001). 253–260. 2001.
Pausch, R., Burnette, T., Brockway, D., Weiblen, M. E., Navigation and Locomotion in Virtual Worlds via Flight into Hand-Held Miniatures. Proceedings of the 1995 ACM SIGGRAPH Conference. 399–400. 1995.
Pierce, J.S., Forsberg, A., Conway, M.J., Hong, S., Zeleznik, R. Image Plane Interaction Techniques in 3D Immersive Environments. 1997 Symposium on Interactive 3D Graphics. 39–43. 1997.
Piper, B., Ratti, C., Ishii, H. Illuminating Clay: A 3-D Tangible Interface for Landscape Analysis, Proceedings on Human Factors in Computing Systems (CHI ʹ02). 2002.
Poupyrev, I., Billinghurst, M., Weghorst, S., Ichi-kawa, T. Go-Go Interaction Technique: Non-Linear Mapping for Direct Manipulation in VR. Proceedings of User Interface Software and Technology (UIST ’96). 79–80. 1996.
71
Poupyrev, I., Numada, T., Weghorst, S. Virtual Notepad: Handwriting in Immersive VR. Proceedings of IEEE Virtual Reality Annual Symposium (VRAIS ʹ98). 126–132. 1998.
Poupyrev, I., Weghorst, S., Billinghurst, M., Ichikawa, T. Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques. Computer Graphics Forum: EUROGRAPHICSʹ98 issue, 17(3). 41-52, 1998.
Poupyrev, I., Weghorst, S., Billinghurst, M., Ichikawa, T. A Framework and Testbed for Studying Manipulation Techniques for Immersive VR. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. 21-28. 1997.
Resnick, M. Behavior Construction Kits. Communications of the ACM, 36(7). 64–71. July 1993.
Sachs, E., Roberts, A., Stoops, D. 3-Draw: A Tool for Designing 3D Shapes. IEEE Computer Graphics & Applications. 18-26. November 1991.
Shaw, C.D., Hall, J.A., Ebert, D.S., Roberts, A. Interactive Lens Visualization Techniques. IEEE Visualization ʹ99. 155-160. October 1999.
Snibbe, S., Herndon, K.P., Robbins, D.C., Conner,D.B., van Dam, A. Using Deformations to Explore 3D Widget Design. Computer Graphics, 26, 2. 351–352. July 1992.
Stoakley, R., Conway, M., Pausch, R., Virtual reality on a WIM: Interactive worlds in miniature. Proceedings of Human factors in Computing Systems (CHI ʹ95). 265–272. 1995.
Sturman, D. J. Whole-hand Input. Ph.D. Thesis, Media Arts and Sciences Section, School of Architecture and Planning, Massachusetts Institute of Technology. 1992.
Suzuki, H., Kato, H. AlgoBlock: A Tangible Programming Language, A Tool for Collaborative Learning. Proceedings of 4th European Logo Conference. 297—303. August 1993.
TrackMan Live. Logitech. http://www.logitech.com/. April 2002.
TrackPoint. IBM. http://www.almaden.ibm.com/cs/user/tp/tp.html. April 2002.
Ullmer, B., Ishii, H., Glas, D. mediaBlocks: Physical Containers, Transports, and Controls for Online Media. Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 379–386. July 1998.
Viega, J., Conway, M., Williams, G.H., Pausch, R.F. 3D Magic Lenses. Proceedings of User Interface Software and Technology (UIST ʹ96). 51-58. 1996.
Virtua Cop 2. Sega Enterprises, Ltd. 1995, 1997.
Watt, A. 3D Computer Graphics, Third Edition. Addison-Wesley Publishing Ltd. ISBN 0201398559. 2000.
Wloka, M., Greenfield, E. The Virtual Tricorder: A Uniform Interface for Virtual Reality. Proceedings of User Interface Software and Technology (UIST ’95). 39–40. 1995.
Zeleznik, R. C., Herndon, K. P, Robbins, D. C., Huang, N., Meyer, T., Parker, N., Hughes, J.F. An Interactive 3D Toolkit for Constructing 3D Widgets. Proceedings of SIGGRAPH ʹ93. 81–84. 1993.
Zhai, S., Buxton, W., Milgram, P. The “Silk Cursor”: Investigating Transparency for 3D Target Acquisition. Proceedings of Human factors in Computing Systems (CHI ʹ94). 459-464. April 1994.