Research Collection

Report

Environment-aware freeform displaysAn evaluation in a tabletop setting

Author(s): Cotting, Daniel; Kostka, J.; Adelsberger, Rolf; Gross, Markus H.

Publication Date: 2011

Permanent Link: https://doi.org/10.3929/ethz-a-006787713

Rights / License: In Copyright - Non-Commercial Use Permitted

This page was generated automatically upon download from the ETH Zurich Research Collection. For moreinformation please consult the Terms of use.

ETH Library

1

Environment-Aware Freeform Displays:An Evaluation in a Tabletop Setting

ABSTRACTInteractive environment-aware display bubbles have been introduced in a novel display metaphor [3] which allows screens to be instantiated with a freeform shape anywhere on demand. The bubbles are adapted either using a matching set of interaction schemes in a collaborative setting or as a reac-tion to occluders detected in the environment.In this paper, this novel display metaphor is evaluated in a user study. Based on various test scenarios in a tabletop set-ting, several usability issues are analyzed such as the suit-ability of freeform displays, focus and context areas, environment-aware adaptivity, laser pointer navigation, pie menus and several aspects of collaboration in general. The results show that the freeform displays and the related meta-phor have a wide acceptance and allow the work to be per-formed in an effective way at an efficiency comparable to the one found in the well-established traditional systems.Author KeywordsUser study, usability, tabletop, adaptive displays, focus and context, interaction, imperceptible structured light, projec-tors.ACM Classification KeywordsH.5.2 [User interfaces]; H5.3. [Information interfaces and presentation]: Group and organization interfaces – Com-puter-supported cooperative work, Synchronous interaction, Evaluation/methodology.INTRODUCTIONComputer technology is increasingly migrating from tradi-tional desktops to novel forms of ubiquitous displays on tables and walls of our environments. This process is mainly driven by the desire to lift the inherent limitations of classi-cal computer and home entertainment screens, which are generally restricted in size, position, shape and interaction possibilities. Previous work [3] has introduced so-called interactive environment-aware display bubbles which signif-icantly enhanced the flexibility, interactivity and adaptivity of displays (see figure 1). Both information presentation and interaction issues were presented in a complete framework for smart and space-efficient freeform displays which facili-tate rich interpersonal communication and collaboration. The core contributions included novel bubble-based focus and context display techniques, a matching set of intuitive inter-action schemes, and an environment-aware adaptivity of the bubbles using per-pixel light control. While the cited paper focused on the core idea of the display metaphor, its techni-cal realization and a comprehensive set of matching interac-tions, this paper performs an extensive evaluation of the suggested display metaphor.

In the next sections, a brief overview of the display bubbles metaphor, selected related work and the study setting will be given. Subsequently, the usability test scenarios will be described, followed by an analysis and discussion of the results. We conclude the paper with impacts on future designs derived from the results of this usability study.

DISPLAY BUBBLESThe novel display metaphor [3] allows the users to freely and independently set arbitrary displays anywhere on demand. For that purpose, personalized widgets or ordinary computer applications that have been designed for a conventional, rect-angular layout are mapped into space-efficient bubbles whose warping is performed with an approach based on potential fields. With a set of interaction operators based on a laser pointer tracking, the projected freeform displays can be transformed and elastically deformed using focus and con-text visualization techniques. The metaphor also provides operations for an intuitive instantiation of bubbles, cloning, cut & pasting, deletion and grouping in an interactive way. Additionally, the display bubbles allow for user-drawn anno-tations and a text entry using a projected keyboard. An optional environment-aware adaptivity of the displays is achieved by a continuous, imperceptible, real-time scanning of the projection geometry. Subsequently, collision responses of the bubbles with the non-optimal surface parts are com-puted in a rigid body simulation.

RESEARCH QUESTIONS AND HYPOTHESESThis usability study aims at evaluating the novel display metaphor with respect to the efficiency, effectiveness and satisfaction. For that purpose, we pose the following hypoth-eses which we want to validate:

- H1: Working on projected freeform displays is as efficient as working on projected rectangular displays.

Figure 1: Interactive environment-aware display bubbles.

D. Cotting, J. Kostka, R. Adelsberger, and M. GrossTechnical Report 564

ETH Zurich

2

- H2: Even for inexperienced users, collaborative work using the novel display metaphor is as efficient as the common PC environment the users are familiar with.

Compared to rectangular screens or traditional PC systems, the display metaphor can offer more flexibility and provide additional features. We measure the related satisfaction by observing and questioning subjects in a tabletop setting. We intentionally cover a wide range of topics. The scenarios of the study focus on the following specific aspects of the dis-play metaphor and its realization:

- Benefits of freeform displays.- Advantages of context areas in display representations.- Utility of an environment-aware adaptivity.- Suitability of a navigation based on laser-pointers.- Usability of pie menus.- Impact on collaboration in general.

RELATED WORKFollowing the inspiring tabletop environment presented in Wellner’s DigitalDesk system [20], many novel tabletop sys-tems have been developed recently, and their suitability for different tasks has been evaluated in many publications. Lumisight [9] introduces a view-dependent, space-efficient tabletop setup, which allows personalized contents to be dis-played for up to four users simultaneously. However, the dis-play geometry remains static and the installation is rather cumbersome and fairly expensive. Providing a larger work-ing area, the Escritoire system [2] implements an interactive foveal display for a single user per working site, which con-sists of a rectangular desk. In a comparable setup, DTLens [6] allows multiple users to zoom into certain parts of the display using rectangular stretching regions. Also using restricted rectangular working spaces, Watson et al. [19]explore collaborative group interaction in an educational classroom setting, and the EDC system [1] studies shared understanding and informed participation. In a similar set-ting, the Caretta system [15] integrates personal and shared spaces to support face-to-face collaboration, while BUILD-IT [5] provides a planning tool allowing a group of people to interact with objects in a virtual scene using real bricks. Con-siderably enlarging the available working space, Augmented Surfaces [14] allow users to project displays onto tables or walls as a spatially continuous extension of their portable computers. Abandoning rectangular settings, Vernier et al. present novel visualization techniques and layout schemes on circular tabletop displays [17].

While little research has been performed about the usability of entire systems in the area of Single Display Groupware (SDG), several studies have been conducted concerning spe-cific aspects of such setups. For example, Oh and Stürzlinger [12] have analyzed laser pointers as input devices for collab-orative work. They emphasize the advantages of these point-ers compared to the use of multiple mice. The system tested in our usability study uses laser pointers as well.

As a guideline for setting up user studies, Geis et al. [16]give advice on how to specify the usability requirements and test criteria for interactive systems and expand the ISO 9241 standard [8] with additional usability requirements that meet the increased demands. These were considered for the choice of the questions for our evaluation questionnaires.

METHODThis section describes the method used to perform our usability study, including the apparatus involved, the selec-tion of the participants and the procedure of the study itself.Setting and apparatusBased on the display bubbles metaphor [3], we have imple-mented interactive on-demand freeform displays in a stan-dard meeting room environment. Our user study focuses on tabletop settings, which lend themselves very well to imple-ment natural user interfaces [13] since desks and tables are used extensively in everyday life to work with physical items such as paper, books and pens. We rely on projection tech-nology for display generation since currently no other tech-nology is nearly as competitive and effective to build flexible, large and high-resolution screens. Our approach is entirely based on off-the-shelf, low-cost hardware including DLP-projectors and FireWire cameras. Figure 2 shows a fish-eye view of the setting used in our study.

ParticipantsAccording to Virzi [18], five participants uncover approxi-mately 80% of the usability problems in a product, a state-ment that is supported by Nielsen’s examinations [11]. Faulkner [4] supports this five-user guideline with a statisti-cal sampling method which extends the old empirical evalu-ation. However, Faulkner shows as well that testing more users results in finding noticeably more usability problems. Therefore, we decided to invite 60 participants in total, split into 16 groups of three to four people. We recruited the par-ticipants from all the different departments of our school. Over 270 interested people responded to our request, which allowed us to make a representative selection focused on potential users of interactive on-demand freeform displays. The final candidates consisted of people of both genders, aged 19 to 60, from a wide range of professions including undergraduate students, PhD students, lecturers, managers, designers, architects, researchers, engineers and secretaries. All the participants had some experience with computers, but none of them had a significant expertise in tabletop inter-faces. ProcedureEach experimental session started with a short introduction in which the experimenter briefed the participants about the study and reviewed the main functionality of the display metaphor. After an explanation of the tasks to perform, the

Figure 2: Fish-eye view of the tabletop setting of the study.

3

usage of laser pointers as a navigation tool was demon-strated. During the conduction of the tasks, the users were continuously monitored and assisted by a supervisor to col-lect observations and data. Besides examining the effective-ness, i.e. the quality of the task results, we analyzed the efficiency aspect of usability by taking timing measurements of several tasks. Due to time constraints each group focused on half of the usability study tasks (scenarios 1/3/4 or 2/5/6, see subsequent section). To gain a knowledge about the extent to which the users were satisfied with the display met-aphor, a post-experiment questionnaire was handed to them covering scenario specific issues and general questions related to the display metaphor. The forms allowed the par-ticipants to anonymously rate various usability aspects on a five-point Likert scale [10] and to freely express likes and dislikes of the system, as well as to bring up suggestions for future improvements. No time limit was enforced, but most users completed the questionnaires within five to ten min-utes. At the end of the usability study, participants were given a small gift as a compensation.

SCENARIOS AND TASKSIn each scenario of our study, specific usability issues were tested, i.e. we analyzed the suitability of novel features such as freeform displays, context areas, the environment-aware adaptivity, the laser pointer navigation, pie menus, and sev-eral aspects of collaboration in general. See table 1 for an overview of which aspects were considered in which sce-nario. The color codes in the questionnaire given in the appendix of this paper refer to the labels in table 1. The par-ticipants solved the various tasks collaboratively or on their own, depending on the setting of the scenario. If both the novel display metaphor and a standard PC were used, then the tasks to compare were designed with an equal degree of difficulty. Each participant used a laser pointer in the display metaphor setting, whereas a single mouse was available to the whole group on the PC. In all the comparative studies, the order of the subtasks to compare was randomly defined. The following sections describe the six different scenarios in more detail, the corresponding questions can be found in the appendix.

Scenario 1: Shape of a single display on a cluttered deskProjection surfaces, especially in tabletop settings, are not always guaranteed to provide an adequately large, uniform and continuous display area. A typical situation in a meeting room or office environment consists of cluttered desks which are covered with many objects such as lamps, plants, books, coffee cups, notepads and a variety of electronic devices. In the presence of clutter and obstacles, freeform bubbles can

permit a much tighter geometric packing than rectangular displays and thus allow to use the limited display space more effectively.One target of this experimental task was to determine to what extent these features facilitate working in cluttered environments. To simplify the conduction and the repeatabil-ity of the study, the obstacles on the table were symbolized by gray sheets of paper. The participants had to analyze a large map of a city and to follow a path given precise direc-tions. At any time, only a small part of the whole map could be displayed, so that the participants had to navigate the map to find the desired information. Zooming was intentionally disabled. However, the visible part of the map could be moved using the laser pointers.In order to analyze the benefits of the freeform shapes and of the context areas, the participants were asked to repeat com-parable tasks three times with screens of about the same size, once with a rectangular display, once with a freeform display without a context area, and finally once with a freeform dis-play providing a context area (see figure 3).

Scenario 2: Shape of multiple displays on a cluttered deskSimilar to the first task, a map was shown on a cluttered tabletop, but this time multiple displays could be activated simultaneously, allowing a group to search in parallel for buildings or parks in a city. In addition to the aspects consid-ered in task 1, collaboration and communication between the group members was analyzed by observations of the supervi-sor and by questionnaires filled out by the participants. The task was repeated twice (see figure 4), once with a rectangu-lar display and once with a freeform display providing a con-text area.

Scenario 3: Presentation with individual display copiesWith its support for a content replication, multiple views, a dynamic display reconfiguration and a multi-user remote interaction, the display bubbles metaphor might lend itself very well for collaborative engineering, design and modeling tasks. Additionally, the simple inclusion of any personal lap-top as a display source could make it suitable for brainstorm-ing meetings or presentations, the test scenario of this third task of our usability study. Here, we mainly focused on the aspects of collaboration and information transfer resulting

Aspect Scenario 1 2 3 4 5 6

Freeform shape x x x xContext area x x x xEnvironment-aware adaptivity xLaser pointer navigation x x x x x xPie menu x x xIndividual display copies xCollaboration & communication x x x x x

Table 1: Aspects considered in the various scenarios.

Figure 3: Comparison of the three display types: rectangular, freeform without context area, freeform with context area.

Figure 4: Collaborative task using rectangular displays on a cluttered desk and using freeform displays having a context ar-ea. In the image on the right, a user is instantiating a new dis-

play using his laser pointer and the pie menu.

4

from the powerful new possibilities offered by the display metaphor. For example, every listener can be assigned its own display bubble, enabling him to interact more directly with the presented material. At any time, he can access a his-tory function, allowing him to freeze the current content or to explore any screen content previously shown, even while the presentation is going on (see figure 5). In order to have a reference for comparison, an additional presentation was held in a traditional meeting room with a front projection. For both settings we prepared a corresponding set of ques-tions that referred to the presentation content and helped the participants to assess the degree of information transfer.

Besides exploring the resulting collaborative issues, the opportunities of freeform shapes were explored once again: Even on meeting tables without an excessive clutter, we can maximize the surface usage by tiling and arranging a very large number of freeform displays in a space-efficient man-ner while taking into account a user-defined information focus. Feedback concerning the freeform shapes of displays was gained again using questionnaires.

This presentation setting was also used to evaluate the envi-ronment-aware adaptivity. Given the projection geometry of the setup, we checked the suitability of the surface for dis-play by continuously analyzing its reflection properties and its depth discontinuities, which had possibly been introduced by new objects in the environment. Subsequently, the bub-bles were moved into adequate display areas by computing collision responses with the surface parts, which were classi-fied as not admissible for display. Thus, the environment-aware adaptivity was continuously guaranteeing an optimal visibility of the projected displays. At the beginning of this scenario, we demonstrated the collision detection and the related shadow avoidance, so that the participants could test the functionality and get an impression of it.

Scenario 4: Semantic information of shaped displaysPictures are able to communicate information in different ways. One does not only get information from the shown content itself, but also from the shape of the pictures. This additional cue of information can easily be realized by the freeform shapes of the display metaphor, which offers the possibility to adapt the display shape to the content shown. In this scenario, we aimed at examining the freeform shape feature and its possibility to transfer additional semantic information. The task was split into two parts (see figure 6).

Scenario 4.1: Recognition by shape. In the first subtask, coarse black-and-white maps giving a hint of the overall geographic structure of eighteen European countries were

provided, while physical labels for half of the countries were lying on the table. The participants had to drag the countries to the corresponding tags. The task was performed once with freeform displays whose shapes approximated the borders of the countries, and once with rectangular displays, where parts of the neighboring countries were still shown besides the centered country maps.

Scenario 4.2: Orientation by shape. In the second subtask, the participants had to sequentially locate a collection of cit-ies on a given map of Switzerland. Scrolling was necessary to access the entire map. Again, the task was performed twice, once with the shape of the display adapted to the out-line of Switzerland, and once in a rectangular window of a comparable size.

Scenario 5: Collaborative design

In order to evaluate the aspects of collaboration more exten-sively and to get a feedback on the display transformation operations, we introduced a collaborative design task, where the participants were asked to create a logo by using a subset of fifteen predefined components. For this scenario, the col-lision avoidance of the display bubbles was turned off, so that the components could be arranged to one (overlapping) logo. Transformation features such as translation, rotation, scaling or duplication could be used concurrently on multi-ple components. Furthermore, the users could add annota-tions to perform little changes to the original components. In order to encourage the participants to work together, the supervisor emphasized that all the members of the group had to agree to a common design draft after about five to ten minutes.

Two different logo tasks were prepared (see figure 7), one for the collaborative display metaphor and one for a tradi-tional PC setting with Adobe Illustrator. In order to concen-trate on collaboration and on the work process itself, we kept

Figure 5: Presentation task using the novel display metaphor. The history function can be reached in the pie menu to freeze

the individual display copy and to explore past content.

Figure 6: Top row: Assigning countries to the corresponding la-bels with freeform and rectangular displays. Bottom row: Lo-cating a sequence of cities in a freeform and a rectangular map.

Figure 7: Design tasks. Left: Village logo. Right: Shop logo.

5

the components of the logos very simple. First, a promotion logo for a mountain village had to be assembled. Second, a logo for a corner shop had to be designed.Scenario 6: Collaborative reasoningThe novel display metaphor could provide an environment suited for collaborative activities such as planning, organiz-ing and storyboarding. We evaluated the aspects of collabo-rative reasoning in two different subtasks.

Scenario 6.1: Jigsaw puzzle. In a first part, the participants had to collaboratively solve two jigsaw puzzles in both the tabletop setting and the traditional PC setting (see figure 8).Scenario 6.2: Comic strip. Haller et al. have demonstrated their system called Coeno [7] by collaboratively ordering different shots of an animation. Similarly, we proposed a comic strip task, where the groups were asked to decide about the correct order of pictures provided in a random order (see figure 9). The groups discussed possible arrange-ments and rearranged the pictures until all the members agreed on a final ordering. At the end, the groups had to be able to tell their version of the comic story. The experiment was conducted both in the tabletop and in the PC setting.RESULTSConsidering part 12 of the ISO 9241 standard [8], the usabil-ity should be analyzed in terms of the efficiency, the effec-tiveness and the satisfaction. We evaluated the efficiency by timing the various tasks, the effectiveness by observing the quality of the task completion, and the satisfaction was determined using a number of standardized satisfaction questionnaires, both collected on a task-by-task basis and at the end of an entire session.SummaryThe main findings of our usability study are:- Freeform displays provide a satisfying, efficient and effec-

tive work experience for various applications. Addition-

ally, as ubiquitous displays are gradually being integrated into the surfaces of our everyday environments, the free-form shapes allow a much better usage of the restricted, occluded and arbitrarily structured space. Furthermore, by creating a subconscious association, the freeform shapes also support the users in recognizing their work and remembering the corresponding tasks.

- For the freeform displays, the context areas are useful and important, even with displays of a rather limited resolu-tion. Higher resolutions can further increase the benefits.

- A majority of the users is satisfied with the environment-aware adaptivity, and the navigation based on laser point-ers and pie menus is effective and appropriate. The tested realization in a tabletop setting only requires some minor improvements and implementation enhancements.

- Individual display copies with freeze and history functions are very useful and allow for presentations which are rated to be as convenient as talks in an established front projec-tion setting.

- Compared to the work in a traditional PC setting, the col-laboration using the novel display metaphor leads to a pro-cess which is less error-prone and more dynamic, creative, balanced and fair. However, due to the increased flexibility, an efficient collaboration either requires a strong discipline within the working group or an additional control mecha-nism, which can possibly be enforced by technology.

- Overall, the users are generally very satisfied with the dis-play metaphor, its utility and its handling, and most impor-tantly the display metaphor allows the work to be performed as effectively as in a familiar PC setting, even with unexperienced users having no special training.

The details of the results are given in the subsequent sec-tions, where we analyze our initial hypotheses and then present a subset of our non-hypothesized findings and obser-vations. For both cases, we discuss the resulting open ques-tions and implications, and we indicate necessary future enhancements to the display metaphor and its realization.EfficiencyOur efficiency-related hypotheses were evaluated by mea-suring the completion times of the various tasks. Figure 10shows a graphical representation of the results.

To support our hypotheses, we use the One-Way ANOVA procedure to test the hypotheses that the measured means of two or three related runs of a task are not significantly differ-ent. As can be seen in the diagram, the assumption of equal variances is satisfied rather well, and for small violations ANOVA is known to be robust as long as the samples are of equal or near equal sizes, an assumption which is satisfied.

Figure 8: Left: Collaborative reasoning on the jigsaw puzzle task in the tabletop setting. Right: Two examples of jigsaw puz-

zles in the PC setting.

Figure 9: Left: Collaborative reasoning on the comic strip task. Two users are simultaneously moving distinct images. Right:

Examples of three comic strips used in this task. The images of each comic strip were provided in random order.

Figure 10: Timing measurements for various task scenarios.

6

Hypothesis H1. Our first hypothesis states: Working on pro-jected freeform displays is as efficient as working on pro-jected rectangular displays. By comparing the means of tasks 1, 2, 4.1 or 4.2, we see that the users can efficiently work on freeform displays without any learning phase. ANOVA on task 1 results in F=0.053 and p=0.949, for task 2 we get F=0.030 and p=0.958, for task 4.1 F=0.115 and p=0.740, and for task 4.2 F=0.665 and p=0.428. Therefore, no significant difference in efficiency can be detected. We have shown that working on freeform displays can be as efficient as working on rectangular displays of a comparable size. Note that this hypothesis has not yet considered one of the major advan-tages of freeform displays: Arbitrary shapes allow to use the available space more efficiently and thus can provide larger displays and show more content simultaneously.Hypothesis H2. Our second hypothesis states: Even for inex-perienced users, collaborative work using the novel display metaphor is as efficient as the common PC environment users are familiar with. We support the validity of this hypothesis by performing ANOVA on tasks 6.1 and 6.2. We get F=0.460 and p=0.509 for task 6.1, and F=0.025 and p=0.879 for task 6.2, showing that no significant difference can be found. Even without training, users can perform work in the new setting as efficiently as in a familiar PC setting.EffectivenessThrough a direct observation of the groups’ interactions by the supervisor, and by judging the quality of the task comple-tion, we noted that in all the settings the scenarios were solved properly and without any major problems.SatisfactionNon-hypothesized findings related to the participants’ satis-faction were drawn from the Likert scale questionnaires that the subjects completed (see appendix). The following dis-cussion is structured according to the various aspects of the novel display metaphor. Free-text comments, impressions, criticism and suggestions of the participants are included in their respective context as well. Note that the numbers in braces refer to the pie diagrams of the corresponding figures in the appendix.Freeform shape. Since the users already were familiar with rectangular shapes from traditional printouts, documents or maps, some of them initially preferred the rectangular ver-sions of the displays and found the freeform displays to be somewhat counter-intuitive. But they finally agreed that after a short time they quickly got used to the new opportunities of the display metaphor. In both the scenarios 1 and 2, the vast majority of the participants found the ability to create free-form displays useful, comfortable and important {11.2, 11.4, 12.2, 12.4, 12.5}, and the users thought it was easy to adapt the shapes to their preferences {11.5}.

While the participants still found it beneficial to be able to adapt the shape to individual preferences in the scenario 3 {13.5}, the freeform shape itself was comparatively judged less important {13.6}. This change in the perceived impor-tance of the freeform displays is probably due to the signifi-cant differences in the applications and the displayed content. In their comments, several users noted that the opti-mal display shape strongly depends on the application sce-nario and the current task to perform. The display metaphor

accounts for this fact by allowing the users to freely and dynamically choose their display from a shape continuum using a so-called potential variation operation [3].In the scenario 4, the participants found that freeform dis-plays did not necessarily provide a significant benefit and increase in efficiency {14.2, 14.3, 14.4, 14.5}. This state-ment reflects our measurements in figure 10 and shows that (at least consciously), for the orientation or recognition, the users were not supported by the additional semantic informa-tion conveyed by the shape of the displays. Several users reported that the shape which approximated the borders of Switzerland (scenario 4.2) was judged rather useless in the beginning. However, in the following they found that sub-consciously it helped them to identify themselves with the project, i.e. it continuously reminded them what the task was about. So, in certain cases, there clearly was a benefit result-ing as a side effect of the freeform shapes.Note that in the scenario 4 there were a few factors which might have impaired the participants’ judgement, but which could be optimized for a better acceptance in the future:- In the scenario 4.1, several participants would have pre-

ferred a better approximation of the country borders than the smooth bubble shape we chose.

- In the scenario 4.2, some users found it confusing and counter-intuitive that not the entire map of Switzerland but only a small zoomed portion was displayed in the free-form shape. Additionally, the users agreed that the shown area in the freeform case was smaller, therefore slightly biasing the result in favor of the rectangular setting.

Context area. Ideally, in the future, the design of graphical user interfaces should be adapted to the requirements of non-rectangular displays. However, in the meantime most com-mon applications remain rectangular, and therefore a warp-ing to freeform shapes is required. In this usability study, the context area introduced by the warping approach has pro-vided very satisfactory experiences: A large majority of the participants found it a useful and important feature of the display metaphor {11.3, 11.6, 12.3, 12.6}. Still, in the current implementation, participants noted some potential for future enhancements and improvements:- The resolution of the displays must be increased. For

some participants, the distortion in the context area only appeared attractive at the first glance. In the long run, the distortion was found rather disturbing and not that useful since the limited resolution made the information illegible due to the strong bending of text and straight lines. For the tasks where the information could be found based on color only (e.g. parks are green on a map), this problem was not significant and the users consistently found the context area very useful.

- In order to easily manipulate the focus of the freeform dis-plays, several users desired an overview display showing a thumbnail of the entire screen. This would complement the existing direct scrolling of the warped context area.

Environment-aware adaptivity. A majority of the participants of our study confirmed the importance of the environment-aware adaptivity {13.7}, while a minority noted that in usual settings people would prepare the projection surface and remove any obstacles. We still believe this is not necessarily

7

true for all the use cases, e.g. people tend to bring a lot of objects, such as cell phones, PDAs, coffee cups and notepads to a meeting. As a reaction to the collisions with the obsta-cles, around 20% of the participants desired a displacement of the displays, 20% an automatic reshaping, and 45% a combination of both. Some people judged the deformation to be appealing for small overlaps, whereas a displacement should be considered when the deformation becomes too large. The remaining 15% preferred no reaction at all. As a result, we provide maximum flexibility by allowing the peo-ple to choose on a per application basis whether they want to use the environment-aware adaptivity. The adaptivity is probably also better disabled in the rare cases where the resulting collision detection and shadow avoidance might be too distracting and confusing {13.8}.

Laser pointer navigation. Even though the participants were not familiar with any navigation based on laser pointers, they generally found the proposed interaction device to be com-fortable and practical {18.1}. As noted by the supervisor, for most users, handling the laser pointer with confidence required a training of a few minutes only. However, some users were distracted by the slight delay between the pointer movement and its recognition by the system, which is mainly due to the inherent latency in the image acquisition pipeline and the projection, and to some extent by the Kalman filter-ing of the pointer positions. The participants reported that they were sometimes distracted by the laser pointers of other users and that they lost track of their own pointers. This was reported more often for less structured tasks {15.5, 15.6, 16.5, 17.5}. Selecting items with the laser pointers was a problem for a few persons only {16.2}. These noticed that the selection was sometimes difficult because one does not see the laser dot before activating it. This is due to the lack of dif-ferent interaction modes such as a passive pointing and an active manipulation. Multiple laser modulations in a temporal or spectral space could resolve this problem, but this would require a specialized hardware. Despite these implementation issues, the users found the laser pointers to be comfortable and practical as mentioned in the beginning of this section.

Pie menu. Overall, the usage of our pie menu (depicted in figures 4 and 5) was found neither especially comfortable nor very uncomfortable {18.2}. A few participants would have favored an access of the most important functions with-out the indirection over the pie menu, e.g. using buttons or direct links, possibly displayed in an autohide fashion either besides the user or besides the selected display. Alterna-tively, for long, multiple component tasks the pie menu could remain visible (sticky window). Furthermore, pointer gestures could compensate for the missing keyboard short-cuts. The users also desired to be able to activate the menu inside a bubble and not only on the free working space (as it was the case in the evaluated prototype version), either by holding the pointer steady for a certain time, or by an addi-tional modulation mode triggering an event similar to a right mouse click for context menus. The size of the pie menu was adequate {18.3}, but its readability sometimes insufficient {18.4}. This is mainly due to the limited resolution in some areas of the current setup. As a measure to improve the legi-bility, the participants suggested to replace the rotated menu labels by icons or at least to take the position of the corre-

sponding user into account. The color coding of the seg-ments of the menu entries was found useful, but should take into consideration the various forms of color blindness in a future version. The readability was also impaired by the sub-menus potentially overlapping with the border of the projec-tion area. Thus, menus should better be relocated, similarly to a submenu which opens upwards when reaching the bot-tom of a traditional screen. While most users found the pie menu appropriate in terms of its hierarchical structure {18.5}, several people noted an inherent limit to two hierar-chy levels for a reasonable representation of hierarchical pie menus without an overlap. A rotatable list menu as sug-gested by a participant could potentially solve this issue, but would introduce a view-dependent appearance. As a better alternative, a subdivision of the pie menu segments into dif-ferent wedges would provide more sublevels, while at the same time maintaining the advantages of the pie menus.

Individual display copies. The ability to have an own copy of the screen contents was found very useful {13.2}, also the freeze and history functions {13.3, 13.4}. As a minor improvement over the current scrolling of past content, a visual browsing of the history was desired and a more direct accessibility of the content, possibly in combination with bookmarks. As a side effect of the increased flexibility, the users in the presentation setting found the additional func-tionality to be potentially distracting from work {13.10} since the participants were focusing on past content, rather than the current information discussed by the presenter. Cop-ies of past and current slides next to each other, or alterna-tively a picture-in-picture approach, could prevent the users from losing the thread of the presentation. While a common projection, as found in a traditional meeting room, unites the group mentally, individual copies require a large discipline of the group to discuss the same content. Nevertheless, the new setting was found as convenient as a traditional front projection setting {13.9}. An additional control or supervi-sion mechanism (also see the next subsection on collabora-tion) would allow to exploit the system to its full potential, which was rated very high {18.8}.

Collaboration and communication. As mentioned earlier, our current realization of the display metaphor does not exploit the potential collaborative capabilities to full capacity. The grade of collaboration is rated rather low {12.7}, also in comparison with similar tasks on a traditional PC setting {15.3, 15.4, 16.3, 16.4}, except in the very open comic task of the scenario 6.2 which required an extensive brainstorm-ing {17.3, 17.4}. This grading of the degree of collaboration is inversely correlated with the difficulty to coordinate col-laboration among each other in the various settings {15.7, 15.8, 16.6, 16.7, 17.6, 17.7}. Some participants noted that the lack of collaboration on some tasks was more out of habit than due to the display metaphor. At this stage people are not really used to multiple displays and a multiple control con-cept, so they mostly worked on their task as they would nor-mally do. If the test tasks were bigger and harder, then a collaboration would probably have seemed more necessary to them, as it was the case in the comic strip scenario. How-ever, in some cases, there seems to be a more fundamental problem: Individual displays promote individual work - each person focuses on his own display and the information in the

8

other displays becomes less relevant. More individual actions are performed, resulting in less time and concentration for group discussions. The paradigm “first think, then act” is no longer valid. Therefore, new ways of communication, group organization and rules of cooperation are necessary, option-ally enforced by technology. Note that in the PC setting, the collaboration was automatically controlled by the limitations of the technology involved, especially by the lack of multiple input devices. The person in charge of the mouse and the keyboard naturally took over the lead, and over the entire task this leader almost never changed since the mouse was hardly ever handed over (scenarios 5 and 6). While a person took over the lead only 25% of the time in the display bub-bles setting, in the PC setting this happened 87% of the time.

Nevertheless, the participants thought that the bubble system was very much usable for collaborative work {18.4}. As advantages of the display bubbles in collaborative tasks such as the scenarios 5, 6.1 and 6.2, the participants mentioned:

- All the persons can test, visualize and share their ideas simultaneously, which is faster than discussing the poten-tial result in the group and leads to less misunderstandings and inaccuracies. So, the process is more dynamic, cre-ative and precise than the traditional sequential work on a PC, especially in brainstorming sessions.

- The novel environment leads to a more balanced work among the users: all the persons participate. In a tradi-tional setting, usually a few take over the lead, not giving everybody a chance to express themselves and contribute. With the collaborative display bubbles, ideas do not need an a priori approval by a leader, and can therefore not sim-ply be ignored.

- The display bubbles provide a high gain in efficiency in parallelizable tasks or tasks which can be split into sub-tasks. The sharing of work is possible, similar to the pro-cessing at an assembly line. This is additionally facilitated by the large-scale workspace.

As potential disadvantages the users listed:

- The metaphor does not naturally provide a single leader which filters the ideas and puts the actions of a focused task into a logical, sequential order. Without such a lead-ing instance, the users are not forced to coordinate, and interference might occur between actions with potentially opposite goals (“Too many cooks spoil the broth...”). The increased flexibility makes it much more difficult to agree on a common final goal on-the-fly, especially in a setting with multiple dominant users. Furthermore, without a leader people tend to work on details, forgetting the big picture. As a remedy, an initial planning and discussion phase can become necessary.

- A large-scale workspace facilitates multiple concurrent discussion threads, which can possibly reduce the exchange of information between some members below an acceptable level. This leads to a loss of common knowledge and again encourages the forking of the result into multiple independent versions.

Overall satisfaction. For most scenarios, the participants were very satisfied with the display bubbles {11.1, 12.1, 14.1, 16.1, 17.1} and found the work to be completed easier than in a comparable PC setting {17.2}. As mentioned

before, collaborative issues impaired the general impression in the presentation scenario and the logo design scenario {13.1, 15.1} and made the work harder than in a PC setting {15.2}. Still, overall, the system was found very interesting and useful {18.6}, and easy to handle {18.7}. CONCLUSIONS AND FUTURE WORKMany participants found the possibility to activate displays anywhere on demand very appealing. As shown by this usability study, the freeform displays and the related meta-phor have a wide acceptance and allow the work to be per-formed in an effective way at an efficiency comparable to the one found in the well-established traditional systems. In order to make the display metaphor usable for the widest possible range of applications and to account for the differ-ent working styles of the users, we provide a maximum degree of flexibility by combining the various aspects such as the freeform shapes, the context areas and the environ-ment-aware adaptivity at the user’s discretion. An automatic determination of the settings for various classes of applica-tions could be subject of an additional usability study with more focused questions.Some components of the metaphor or of its current realiza-tion still need further refinement, especially the technology to facilitate and guide an efficient collaboration. Such tech-nology can include:- A global control mechanism for establishing a task leader,

who controls the shared work and can take over the con-trol on all the objects, possibly overriding the operations and settings of the other users. Optionally, the interaction of the other users can be switched off, or access rights on multiple levels be enforced. If required, the control can be handed over in a token passing approach.

- A hierarchical separation of the working space into strictly private and public influence regions, where private areas prevent other users from interacting and interfering with the contained displays and the corresponding work.

- Alternative forms of explorative work on a private and public level, e.g. based on explicit forking and merging operations on versions of the work.

- A visual collaboration support without a strict enforcement of rules, such as a visualization of the connection between the displays and of the work collaborators are performing.

- A few users wished an additional finger-based interaction not requiring any device, a feature we have subsequently integrated into our system. The new form of interaction automatically limits the degree of simultaneous collabora-tion due to the fact that the position of hands and arms more easily in conflict with each other and collide. This can lead to a better synchronization and overview of con-current tasks.

As an important advantage of using display bubbles on a tabletop, people do no longer have to switch between the horizontal surface containing books and documents to a ver-tical screen or front projection, which is rather convenient. As encountered in the presentation scenario, this can admit-tedly lead to a loss of the visual contact with the presenter. In order to prevent a reduced information transfer, an additional real-time video stream bubble could be introduced, or a hybrid approach, where both the private displays and a pub-lic projection on a wall are combined. Additionally, a unified

9

control of the focus of all the bubbles by the presenter would be helpful in order to have a common, continuous visibility of the important and relevant information.

Note that the lack of collaboration in some tasks can also possibly be attributed to the new, unknown and interesting technology: Many users admitted that in an initial phase the productivity was rather low in the new setting since the peo-ple spent time on testing the new display and interaction functionality instead of concentrating on the main task. Therefore, after an accommodation phase possibly even bet-ter results could be expected in a usability study.

APPENDIXIn this section, we summarize the results of the Likert scale questionnaires that the participants of the study completed.QuestionsThe questionnaire of every scenario contained selected ques-tions of the following list. These questions are directly refer-enced in the answers of the subsequent section. The colors of the left border refer to the aspects of table 1.

Answers

Figures 11 to 18 show the distribution and the statistics of the responses of the participants. The answers were either related to the specific scenarios or to generic issues of the entire usability study session.

The corresponding questions, as listed in the previous sec-tion, are referred to with the numbers in brackets next to the pie charts. The labels of the pie slices indicate the number of associated responses. The ranges of the Likert scale are mapped in the following order to the colors shown: red, light red, gray, light green, green.

Q1. Rate the usefulness of the freeform shape feature of the display bubbles (completely useless ... very use-ful).

Q2. Rate the usefulness of the context area feature of the display bubbles (completely useless ... very useful).

Q3. Rate the importance of the freeform shape feature for the display bubbles (not important ... very impor-tant)?

Q4. Rate the importance of the context area feature for the display bubbles (not important ... very impor-tant)?

Q5. Is it a benefit to be able to adapt the shape of the dis-plays to your preferences (not at all ... very much)?

Q6. Rate the usefulness of the following feature of the display bubbles: having an individual display copy (completely useless ... very useful).

Q7. Rate the usefulness of the following feature of the display bubbles: freezing of the content (completely useless ... very useful).

Q8. Rate the usefulness of the history feature of the dis-play bubbles (completely useless ... very useful).

Q9. Were you distracted from the work by the freezing and history features (very much ... not at all)?

Q10. Rate the importance of the environment-aware adaptivity in this scenario (not important ... very important).

Q11. Do you think the environment-aware adaptivity is distracting and confusing (very much ... not at all)?

Q12. Rate the grade of collaboration of your group (low ... high).

Q13. Rate the grade of collaboration of your group in the display bubbles setting (low ... high).

Q14. Rate the grade of collaboration of your group in the PC setting (low ... high).

Q15. Did you have problems to coordinate collaboration among each other (very often ... never)?

Q16. Did you have problems to coordinate collaboration among each other in the display bubbles setting (very often ... never)?

Q17. Did you have problems to coordinate collaboration among each other in the PC setting (very often ... never)?

Q18. Which system is more convenient to follow a presen-tation (front projection ... display bubbles)?

Q19. Was it easier to work together in the display bubbles setting than in the PC setting (not at all ... very much)?

Q20. Was it more comfortable to work with the freeform shaped displays than the rectangular ones (not at all ... very much)?

Q21. Was it easier to orientate yourself with the freeform shaped display bubbles than the rectangular ones (not at all ... very much)?

Q22. Do you think you are conducting this kind of task faster with freeform shaped displays (not at all ... very much)?

Q23. Was it comfortable and practical to use the laser pointer as an input device (not at all ... very much)?

Q24. Rate the handling of the laser pointer for selection (very hard to use ... very easy to use).

Q25. Did you feel distracted by other laser pointers (very often ... never)?

Q26. How often did you loose track of your laser pointer because of others (very often .. never)?

Q27. Was the pie menu comfortable to use (not at all ... very much)?

Q28. Is the design of the pie menu appropriate in terms of its size (not at all ... very much)?

Q29. Is the design of the pie menu appropriate in term of its readability (not at all ... very much)?

Q30. Is the design of the pie menu appropriate in terms of its hierarchical structure (not at all ... very much)?

Q31. Was it easy to adapt the shape of the displays to your preferences (not easy ... very easy)?

Q32. Rate the overall satisfaction with the display bubbles system in this scenario (very dissatisfied ... very sat-isfied).

Q33. Rate the general handling of the display bubbles (very hard to handle ... very easy to handle).

Q34. How is your general impression about the display bubbles system (bad, useless thing ... very interesting and useful)?

Q35. Do you think this system is usable for presentations (not at all ... very much)?

Q36. Do you think this system is usable for collaborative work (not at all ... very much)?

10

REFERENCES1. E. Arias, H. Eden, G. Fischer, A. Gorman, and

E. Scharff. Transcending the individual human mind - creating shared understanding through collaborative design. ACM Trans. Comput.-Hum. Interact., 7(1):84–113, 2000.

2. M. Ashdown and P. Robinson. Experience with the Escri-toire: A personal projected display. IEEE Multimedia, 12(1):34–42, January-March 2005.

3. D. Cotting and M. Gross. Interactive Environment-Aware Display Bubbles. In Proc. of UIST’06, to appear, 2006.

4. L. Faulkner. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Instruments and Computers, 35(3):379–383, 2003.

5. M. Fjeld, F. Voorhorst, M. Bichsel, H. Krueger, and M. Rauterberg. Navigation methods for an augmented reality system. In Proc. of CHI’00, pages 8–9, 2000.

6. C. Forlines and C. Shen. DTLens: multi-user tabletop spatial data exploration. In Proc. of UIST’05, pages 119–122, 2005.

7. M. Haller, D. Leithinger, J. Leitner, and T. Seifried. Coeno storyboard: An augmented surface for storyboard presentations. http://www.coeno.org/cms/, 2005.

8. International Organization for Standardization. ISO 9241. National Standards Organisation, 1996.

9. Y. Kakehi, M. Iida, T. Naemura, Y. Shirai, M. Matsushita, and T. Ohguro. Lumisight Table: An Interactive View-Dependent Tabletop Display. IEEE CG&A, 25(1):48–53, January/February 2005.

10. R. Likert. A Technique for the Measurement of Attitudes. Archives of Psychology, 140:1–55, 1932.

11. J. Nielsen and T. K. Landauer. A mathematical model of the finding of usability problems. In Proc. of CHI’93, pages 206–213, 1993.

12. J. Oh and W. Stuerzlinger. Laser Pointers as Collabora-tive Pointing Devices. In Proc. of GI’02, pages 141–149, 2002.

13. M. Rauterberg, T. Mauch, and R. Stebler. How to improve the quality of human performance with natural user interfaces as a case study for augmented reality. In Advances in Occupational Ergonomics and Safety I, pages 150–153, 1996.

14. J. Rekimoto and M. Saitoh. Augmented Surfaces: A Spa-tially Continuous Work Space for Hybrid Computing Environments. In Proc. of CHI’99, pages 378–385, 1999.

15. M. Sugimoto, K. Hosoi, and H. Hashizume. Caretta: a system for supporting face-to-face collaboration by inte-grating personal and shared spaces. In Proc. of CHI’04, pages 41–48, 2004.

16. W. D. T. Geis and W. Redtenbacher. Specifying usability requirements and test criteria for interactive systems. Dortmund/Berlin : Wirtschaftsverlag NW, 2004.

17. F. Vernier, N. Lesh, and C. Shen. Visualization tech-niques for circular tabletop interfaces. In Proc. of Advanced Visual Interfaces’02, 2002.

18. R. A. Virzi. Refining the test phase of usability evalua-tion: how many subjects is enough? Hum. Factors, 34(4):457–468, 1992.

19. B. Watson, J. Kim, T. McEneany, T. Moher, C. Hindo, L. Gomez, and S. Fransen. StorySpace: technology sup-porting reflection, expression and discourse in classroom narrative. IEEE CG&A, pages 2–4, Jan-Feb 2004.

20. P. Wellner. Interacting with paper on the DigitalDesk. Communications of the ACM, 36(7):86–97, 1993.

Figure 11: Responses to the questions of scenario 1. (Q32) (Q1) (Q2) (Q20) (Q31) (Q4)

Figure 12: Responses to the questions of scenario 2.(Q3) (Q4) (Q12) (Q15)

(Q32) (Q2) (Q20)(Q1)

Figure 13: Responses to the questions of scenario 3.(Q3) (Q10) (Q11) (Q18) (Q9)

(Q32) (Q6) (Q7) (Q8) (Q5)

Figure 14: Responses to the questions of scenarios 4.1 and 4.2.(Q32) (Q1) (Q20) (Q21) (Q22)

Figure 15: Responses to the questions of scenario 5.

(Q32) (Q19) (Q13) (Q14)

(Q25) (Q26) (Q16) (Q17)

Figure 16: Responses to the questions of scenario 6.1.(Q32) (Q24) (Q13) (Q14) (Q25) (Q16) (Q17)

Figure 17: Responses to the questions of scenario 6.2.(Q32) (Q19) (Q13) (Q14) (Q25) (Q16) (Q17)

Figure 18: Responses to the general questions.

(Q23)

(Q30) (Q34) (Q33) (Q35)

(Q29)(Q28)(Q27)

(Q36)