towards enhancing awareness in designing collaborative...
TRANSCRIPT
THESIS FOR THE DEGREE OF LICENTIATE OF ENGINEERING
Towards Enhancing Awareness in
Designing Collaborative Computing Systems
KHANH-DUY LE
Department of Applied Information Technology
Research School of Human-Computer Interaction
CHALMERS UNIVERSITY OF TECHNOLOGY
Gothenburg, Sweden 2017
Towards Enhancing Awareness in Designing Collaborative Computing Systems
KHANH-DUY LE
© KHANH-DUY LE, 2017.
Technical report no xxxx:xx
Department of Applied Information Technology
Research School of Human-Computer Interaction
Chalmers University of Technology
SE-412 96 Gothenburg
Sweden
Telephone + 46 (0)31-772 1000
Cover:
Illustration conceptually showing collocated and distributed collaborative works
facilitated by technology platforms investigated by the works of the author.
Chalmers Reproservice
Gothenburg, Sweden 2017
Abstract
Awareness is critical to the success of collaborative activities. User awareness (from here
on: awareness) is a system’s capacity to provide comprehensible and appropriate
communication cues from one user to the other users of the same system. In collaborative
computing systems, awareness does not only address the needs of individual users; it can
also improve mutual understanding and collaboration (resp. coordination). While
advances in computing technology have transformed the way we communicate, work,
and collaborate; off-the-shelf solutions and form-factors can also imply unexpected
benefits and challenges related to awareness. This licentiate thesis [1] builds upon four
papers; each investigating such benefits and challenges. Besides addressing off-the-shelf
solutions, the thesis also proposes novel interaction techniques aiming to improve the
usability of collaborative systems.
The thesis starts out with an overview of awareness in collaborative systems, then briefly
examines how awareness has been supported within four specific areas: tabletop
computing, social network, creative problem-solving, and haptic feedback. These are
also the areas of the projects reported in the central parts of the thesis. Based on literature,
we briefly describe the contribution of our respective works to the area. This should both
motivate each of the papers and demonstrate how each paper adds to the state-of-the-art.
The first paper (P1) investigates a concept making tabletop systems aware of users’
leaning postures to better support them in their tasks. In the second paper (P2), we
designed and developed a social-network based photo-recommendation tool that is aware
of user context to support reminiscing activities, which are beneficial to reinforcing
relationships between people. In the third paper (P3), we examine different aspects that
influence group awareness in remote collaboration regarding the abilities and challenges
that standard mobile devices have in supporting creative problem solving in a remote
setting. The fourth paper (P4) reports on a new interaction technique allowing users to
employ their finger movements to intuitively define tactile feedback for visual contents
in tabletop applications, serving as a contribution towards making tactile-feedback
awareness more integral to computing systems. The thesis rounds off by presenting
research questions derived from the papers and offers an outlook to future works I am
planning for the next steps towards my doctoral thesis.
1 The licentiate thesis corresponds to a qualification examination, carried out after two years of PhD-
studies.
To my loved ones
Acknowledgements
The first people I want to express my gratitude to are of course my parents who tried
their best to provide me the best education background they could and also support me
in this journey. Thanks all my family members for always caring for me despite
geographical distances. I’m also sorry for not having time to go home to visit more
frequently since I started studying abroad five years ago. I promise I will do that after
my licentiate.
My deep gratitude also goes to Professor Morten Fjeld, my supervisor who decided to
recruit me regardless of the mistakes I had made during the recruitment process for the
position. Without him, I could not be here. Also, I would like to thank him for all of his
support for me over the last two years. I also thank Professors Andreas Kunz and Michel
Chaudron - my two co-advisors.
Thanks to my t2i Lab-mates such as Paweł W. Woźniak, Tomasz Kosinski, Alexandru
Dancu, Jesper Molin, Zlatko Franjcic, for your help and time spent on discussing ideas
with me. I always wish we will be able to collaborate more in the future. I also appreciate
the collaborations with colleagues from partner universities such as ETH Zurich,
National University of Singapore (NUS), and University of Science – Ho Chi Minh City.
I definitely also cannot forget to thank ‘My Phuong’ - my girlfriend for all the events and
‘flavors’ she added into my journey. Those made me more motivated, focused and better
at planning and time management. I treasure all the moments I had with you, honey.
Thanks to my Vietnamese friends in Gothenburg, who have accompanied me from my
early time in Gothenburg and shared with me many joyful moments. Likewise, thanks to
my Vietnamese friends in France who always warmly welcomed me whenever I went
there.
Lastly, I thank all the people who inspired, supported and gave me the chance to study
abroad and pursue my passion of HCI. I hope that readers will see this thesis as a
summary of the early period of my research career. During this journey, I have grown
from a software-engineering student to a Human-Computer Interaction (HCI) researcher
and designer. I have explored various research topics, collaborated and taken
responsibility within numerous projects with partners from different sectors, and have
acquired a wide range of skills and knowledge.
List of papers (P1-P4)
P1: Khanh-Duy Le, Mahsa Paknezhad, Paweł W. Woźniak, Maryam Azh, Gabrielė
Kasparavičiūtė, Morten Fjeld, Shengdong Zhao, and Michael S. Brown. 2016. Towards
Leaning Aware Interaction with Multitouch Tabletops. In Proceedings of the 9th Nordic
Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY,
USA, Article 4, 4 pages.
DOI: https://doi.org/10.1145/2971485.2971553
P2: Vinh-Tiep Nguyen, Khanh-Duy Le, Minh-Triet Tran, Morten Fjeld.
2016. NowAndThen: a Social Network-Based Photo Recommendation Tool Supporting
Reminiscence. In Proceedings of the 15th International Conference on Mobile and
Ubiquitous Multimedia. ACM, New York, NY, USA, 10 pages.
DOI: https://doi.org/10.1145/3012709.3012738
P3: Khanh-Duy Le, Ali Alavi, Andreas Kunz, Morten Fjeld (in review). DigiMetaplan:
Exploring an Interactive System for Facilitated Brainstorming. 2017. Submitted to the
11th ACM SIGCHI Conference on Creativity and Cognition (C&C '17). 10 pages.
P4: Khanh-Duy Le, Kening Zhu, Tomasz Kosinski, Morten Fjeld, Maryam Azh, and
Shengdong Zhao. 2016. Ubitile: A Finger-Worn I/O Device for Tabletop Vibrotactile
Pattern Authoring. In Proceedings of the 9th Nordic Conference on Human-Computer
Interaction (NordiCHI '16). ACM, New York, NY, USA, Article 87, 4 pages.
DOI: https://doi.org/10.1145/2971485.2996721
List of patents (PT1-PT2)
PT1: Fabien Danieau, Julien Fleureau, Khanh-Duy Le. 2016. System and method for
automatically localizing haptic effects on a body.
URL:
https://patentscope.wipo.int/search/en/detail.jsf?docId=WO2016102391&redirectedID
=true
PT2: Julien Fleureau, Fabien Danieau, Khanh-Duy Le. 2016. Method, apparatus and
system for synchronizing audiovisual content with inertial measurements.
URL:
https://patentscope.wipo.int/search/en/detail.jsf?docId=WO2016124580&redirectedID
=true
Contents
1 INTRODUCTION .................................................................................................. 1
2 AWARENESS......................................................................................................... 5
2.1 Workspace Awareness ..................................................................................... 6 2.2 Contextual Awareness ...................................................................................... 7 2.3 Social Awareness ............................................................................................. 8
3 AWARENESS SUPPORT IN TABLETOP SYSTEMS ................................... 11
3.1 Workspace Awareness Supports in Tabletop Systems .................................. 11 3.2 Contextual Awareness Support in Tabletop Systems .................................... 14
4 AWARENESS SUPPORT IN SOCIAL NETWORK SITES .......................... 17
5 AWARENESS SUPPORT IN DISTRIBUTED CREATIVE GROUP WORK
21
6 AWARENESS SUPPORT FOR VIBRO-TACTILE FEEDBACK
INTERACTION IN COLLABORATIVE ENVIRONMENTS ............................... 27
7 SUMMARY OF PAPERS IN THE THESIS ..................................................... 31
7.1 Paper 1 - Towards Leaning Aware Interaction with Multitouch Tabletops. .. 31 7.2 Paper 2 - NowAndThen: a Social Network-Based Photo Recommendation
Tool Supporting Reminiscence. ................................................................................. 31
7.3 Paper 3 - DigiMetaplan: Exploring an Interactive System for Facilitated
Brainstorming. ............................................................................................................ 32
7.4 Paper 4 - Ubitile: A Finger-Worn I/O Device for Tabletop Vibrotactile
Pattern Authoring. ...................................................................................................... 33
8 TOWARDS LEANING AWARE INTERACTION WITH MULTITOUCH
TABLETOPS ................................................................................................................ 39
8.1 Introduction .................................................................................................... 40
8.2 Related Work .................................................................................................. 41
8.3 Study 1: Developing a Lean Posture Set ........................................................ 41 8.4 Proof-of-Concept Algorithm .......................................................................... 42 8.5 Study 2: Evaluating the Model ....................................................................... 44 8.6 Discussion and Outlook ................................................................................. 45 8.7 Acknowledgements ........................................................................................ 46
8.8 References ...................................................................................................... 47
9 NOWANDTHEN: A SOCIAL NETWORK-BASED PHOTO
RECOMMENDATION TOOL SUPPORTING REMINISCENCE ....................... 53
9.1 Introduction .................................................................................................... 54 9.2 Related Work .................................................................................................. 56 9.3 Pre-Study Survey ............................................................................................ 59 9.4 Photo Recommendation Tool ......................................................................... 61
9.5 Preliminary User Study .................................................................................. 68 9.6 Results: Design Insights and Implications ..................................................... 70 9.7 Conclusion and Outlook ................................................................................. 74 9.8 Acknowledgements ........................................................................................ 75
9.9 References ...................................................................................................... 75
10 DIGIMETAPLAN: EXPLORING A DISTRIBUTED SYSTEM FOR
FACILITATED BRAINSTORMING ........................................................................ 83
10.1 Introduction .................................................................................................... 84 10.2 Background .................................................................................................... 85 10.3 Related Work .................................................................................................. 86
10.4 Design of DigiMetaplan ................................................................................. 87 10.5 Exploratory User Study .................................................................................. 91 10.6 Results and Lessons Learned ......................................................................... 96 10.7 Discussion .................................................................................................... 103 10.8 Conclusion .................................................................................................... 105
10.9 References .................................................................................................... 106
11 UBITILE: A FINGER-WORN I/O DEVICE FOR TABLETOP
VIBROTACTILE PATTERN AUTHORING......................................................... 115
11.1 Introduction .................................................................................................. 116 11.2 Related Works .............................................................................................. 116 11.3 System Design .............................................................................................. 117 11.4 Application/Demonstration Scenario ........................................................... 120
11.5 Future Work ................................................................................................. 121 11.6 References .................................................................................................... 121
12 CONCLUSION AND OUTLOOK ................................................................... 123
13 BIBLIOGRAPHY .............................................................................................. 125
1
1 INTRODUCTION
Since more than three decades, the field of Computer Supported Collaborative Systems
(CSCW) has seen major research efforts of theoretical, technological, and empirical
nature. Besides theoretical studies on mechanisms and influences of awareness in
CSCW, technologies have been developed and turned into systems improving the
awareness not only among users but also between users and systems. Those collaborative
systems can be found in a wide range of domains such as in shared text editors, in
interactive surfaces solutions, and in social media. Within this thesis, I present my
contributions ways to improve awareness within four application domains: tabletop
interaction, distributed brainstorming, social media, and vibro-tactile feedback authoring.
The common denominator of my contributions is that they are initial explorations into
technical concepts and interaction techniques designed to facilitate awareness, partly
among users, partly between systems and users. The value proposition of this work is
improved communication practices and work processes.
Context of research
The works included in this thesis were conducted in collaborations with both academic
and industrial partners. Papers P1, P2, and P4 were carried out with our academic partners
from Chalmers, National University of Singapore (NUS), City University of Hong Kong,
and University of Science, Vietnam National University. Paper P3 was the result of a
funded collaborative project called MERCO, which involved partners from both
academia and industry. While collaborative works with academic partners gave us the
freedom to explore various workstyles, the funded collaboration with industrial partners
was framed to meet expectations of both partners and funding agents. Next follows a
brief description of the MERCO project that kept me busy for the first two years.
MERCO project
General information
MERCO, stands for ‘Mediated Effective Remote Collaboration’; it was a collaborative
research project with the involvement of several academic and industrial partners such
2
as Chalmers University of Technology (Sweden), ETH Zurich (Switzerland), SEMCON
AB, Touchtech AB, Ericsson AB (all Sweden), AVS Systeme AG, and Intelliconcept
AG (both Switzerland). The project ran from June 2014 until February 2017. The aim of
the project was to produce close-to-market collaboration solutions for teamwork between
people in multiple locations.
Project motivation
Following a worldwide expansion of international companies and corporations,
workforce is increasingly distributed. Despite this distribution of workforce, people in
different locations still need to work together in tasks such as planning, early design,
problem solving, and decision-making. However, off-the-shelf systems are still unable
to fully support distributed teams to work together as effectively as when they are
physically in the same place [5,45]. For crucial decision-making or for critical tasks,
companies still prefer to let their employees travel to and meet at the same place. As
shown in Figure 1, business travel expenses sum up to a significant amount. In trying to
save these costs, work-related travel is likely to be scrutinized and perhaps reduced. As
a result, information exchange degrades, slowing down innovation and time-to-market
of new products.
Figure 1: Travel expenses in Europe (2008).
In order to partially address this problem, the main expected deliverable of the MERCO
project is a suite of software solutions supporting ideation and decision-making activities
in distributed teams. Despite the availability of many commercial applications, such
3
activities still cannot be performed effectively in distributed settings. The expected result
of the project should be help industrial firms to foster collaboration in distributed teams.
Noteworthy, the industrial partners in our projects also have different expectations from
the project. Ericsson AB expects to use the system in their product development process
to reduce costs. The company also expects to have a convincing application that can be
shown to its customers to demonstrate the benefits of broadband communication.
SEMCON AB, as an international industrial-design consulting firm, expects to have
cutting-edge remote collaboration to raise its appeal to customers and clients worldwide
as well as using it for internal processes. Likewise, Touchtech AB hopes the extended
software capabilities realized within the MERCO project will help towards increasing
sales in the company’s main applications in the field such as exhibitions, showrooms,
fairs, workshops and meetings. On the other hand, for AVS Systeme AG and
Intelliconcept AG, most of the commercial expectations from the project are based on an
extension of existing systems with low technical modification only. In short, the MERCO
project aims to provide software solutions facilitating remote collaboration using off-the-
shelf hardware technologies with a low number of modifications. Since the deliverables
promised were fixed and agreed upon at the beginning of the project, the execution of
the project was clearly defined.
4
5
2 AWARENESS
Despite having been investigated from early on in the history of CSCW, there is no single
definition of ‘awareness’ that works well for the larger community of HCI/CSCW. In a
CSCW context, ‘awareness’ usually does not refer to some special category of mental
state existing independently of action but to an actor being or becoming aware of
something or someone [53]. One of most widely-cited definitions of ‘awareness’ was
suggested by Dourish and Bellotti [11] in which it is referred to as an actor’s
understanding of the activities of others, which provides a context for the activity of that
actor. In addition, awareness also means that actors will adjust, align, or integrate their
activities tacitly and unobtrusively to the thing or the person they are aware of [11]. In
CSCW, this can be understood to appear either between human and human or between
human and computer. An example of awareness between human and human in
collaborative systems is when multiple users are working with a shared text editor; each
user’s presence in the system being represented by a cursor. When a user sees the cursor
of another participant at a place on the document, he/she knows that the other user is
working on at that place and will try to avoid manipulating there in order to avoid
conflicts. As for awareness between human and computer, our focus is on making
computer systems aware of their users. For example, a computer system can also be
aware of its users. For example, depending on the content that a user is interacting with,
an application can suggest similar contents or a suitable tool to assist. Or in a
collaborative application running on large interactive display supporting multiple
concurrent users, the system can locate each user and position necessary tools around to
help them work more efficiently. In short, there are several ways to facilitate awareness
in collaborative systems. A system can provide cues that trigger its users to be aware of
each other in collaborative task or be actively aware of its users in order to provide
suitable support.
When we talk about awareness, we refer to one actor being aware of some particular
occurrence. Since there are different types of occurrences that an actor could be aware
of, the first question should be: awareness of what? [53]. Depending on what information
an actor is aware of, researchers have proposed several categories of ‘awareness’, and
the terminologies used to refer to these varies. This thesis briefly introduces some types
of awareness most commonly found in the literature. We should also keep in mind that
these types of awareness are not mutually exclusive; in reality many systems and
6
interaction techniques can support many types of awareness simultaneously. Next, we
revisit workspace, contextual, and social awareness.
2.1 Workspace Awareness
Gutwin and Greenberg defined workspace awareness as up-to-the-moment
understanding of another person’s interaction within a shared workspace [22]. A shared
workspace may refer to a physical one such as a whiteboard, a control panel, a tabletop,
or a virtual one such as a shared text editor. In this definition of workspace awareness, it
is awareness of people and how they interact with the shared workspace, rather than just
awareness of the workspace itself. Secondly, workspace awareness is limited to events
happening in the workspace - inside the time-space bounds of the task that the group is
carrying on [23]. Therefore, workspace awareness involves knowledge about where
others are working, what they are doing, and what they are going to do next. This
information is also useful for many activities of collaboration - for coordinating actions,
managing coupling (the degree to which people can work together), task referencing,
anticipating others’ actions, and finding opportunities to assist others [23].
Workspace awareness comes naturally in face-to-face collaborations because people in
the same place can easily perceive the actions and expressions of others. However, while
working together through groupware systems - computer systems that support groups of
people engaged in a common task (or goal) and that provide an interface to a shared
environment [13] - it is more difficult to maintain workspace awareness. The reason is
that in groupware systems, members may not fully see the whole workspace or may not
see the same workspace as the others [22]. The richness of communication when working
in the same physical place is usually not fully captured and presented by groupware
systems. For example, when two people are physically located and working on a physical
document, it is easy for them to recognize on which page of the document the other is
working. However, if they are working with a digital document through a shared text
editor, it may require them to scroll a long distance or explicitly ask each other (e.g. by
chatting) to know where they are working.
Due to the importance of workspace awareness in the collaborative environment,
researchers have since long proposed several mechanisms and interaction techniques to
support this type of groupware awareness. For example, GROVE, a real-time shared text
editor designed by Ellis et al. [13], provides a shared telepointer which can be seen and
7
controlled from all users from their computer so that everyone knows exactly where a
person is referring to. However, this shared telepointer can cause a blocking effect when
the telepointer is controlled by one person and others have to wait. To avoid this,
applications such as Google Docs provide each user with a personal cursor differentiating
from others by its color and associated name label. If users want to quickly know the
current position of the cursor of another one, they can click on the representing icon of
that user and the system will automatically scroll to that position. “Radar views” [24] are
another design approach to support workspace awareness in shared environments. When
working on a shared workspace, a user may focus on a specific region and lose track of
what others are doing outside that region. To address this, “Radar views” provide a global
overview of the whole workspace so that users can maintain workspace awareness when
working on their own region of interest. Similarly, Baudisch and Rosenholtz proposed
the design of “Halo” [4] to maintain user workspace awareness on an off-screen location
when using map applications. Gutwin and Greenberg [23] even proposed a descriptive
framework for designing to support workspace awareness in collaborative systems based
on their observations on users interactions on a shared text editor. However, supporting
workspace awareness is not only limited to shared text editor systems. As advanced
technologies, such as tabletops, interactive displays, motion-capture sensors, etc. have
started to be employed in collaborative systems, researchers have developed various
interaction techniques taking advantage of these technologies to facilitate workspace
awareness in new collaborative environments.
2.2 Contextual Awareness
Before defining contextual awareness, we need to clarify the term ‘context’. Salber et al.
[49] define ‘context’ of an actor as information that is part of the actor’s operating
environment and can be sensed by the actor. This information typically includes the
location, identity, activity and state of people, groups and objects [49]. Contextual
awareness refers to the idea of adapting behavioural patterns depending on the contextual
information sensed. In fact, contextual awareness is not limited only to collaborative
systems but increasingly plays an important role in ubiquitous computing systems
[49,51]. In collaborative systems, depending on who the actor is, the systems will present
corresponding information appropriately. Therefore, in designing contextual-awareness-
support features in collaborative systems, the interesting information that needs to be
determined includes i) what information users should be made aware of, and ii) how they
8
should be made aware of it [39]. Carefully considering contextual awareness is also
helpful in addressing the problem of privacy invasion and user disturbance in awareness-
support systems [29].
Examples of employing contextual awareness into computing systems can be found in
various domains. Tour-guide mobile applications sense the current location of users and
provide them information of surrounding potentially interesting places [1,57]. Social
network sites such as Facebook, Youtube, and Flickr suggest news, photos, and videos
having similar locations, publisher, tags, and names with the one users are viewing.
Timestamp is also a contextual awareness cue considered by Facebook to suggest
relevant information to users. In the office context, sensing systems can detect locations
of people to trigger communication and interaction among them or forward phone calls
to work more effectively [62,63]. Similarly, in office contexts which offer large displays
such as interactive whiteboard, the sensing systems can detect a handheld device placed
nearby a display and allow it to use the display or interact with nearby handheld devices
[50,62]. To better support contextual awareness in applications, Salber et al. [49]
proposed a toolkit to design context-aware graphical user interfaces.
2.3 Social Awareness
Although there are various forms of social awareness [64], in CSCW, social awareness
has been defined as awareness of a person about the situation of other people, i.e.,
awareness about what they are doing, whether they are talking to someone, whether they
can be disturbed etc. [6]. Social awareness is important in our everyday work, allowing
us to act appropriately based on the information gathered continuously from our
colleagues. If they listen, we will talk; if they are not at a place, we might phone them or
leave a message; if they are in the right mood, we can start a discussion; if not, we
postpone it [59]. While in collaborative systems, workspace awareness and contextual
awareness can be understood to happen either between users or between users and the
systems, but social awareness always refers to actions between users. Therefore, to
support social awareness, collaborative systems mainly employ technologies to capture,
transfer and express user communication cues to emphasize their presence and activities
in the system, making others well-aware of it.
MASSIVE [20] combines audio, graphics and text to express user’s different levels of
presence as well as their location and orientation so that others have good knowledge
9
about their situation. Portholes support social awareness in distributed teams by
permanently sending images from one side to the other so that people on the other side
are informed about the presence, activities and availabilities of the remote partners [12].
Other systems incorporate emoticons - graphical icons with facial expressions - as a way
for users to express their emotions [16,60], providing users more communication cues
about their partners.
Besides these three types of awareness, there are still other types of awareness
categorized by researchers including temporal awareness, spatial awareness, peripheral
awareness, and situation awareness. However, there are certain overlaps in the
descriptions of these types of awareness with the three described in this thesis.
Furthermore, these categorizations are also less widely used than workspace, contextual
and social awareness. Therefore, this thesis focuses on describing how the works
included can support these three types of awareness, although not every single work can
support all of them. Here I briefly describe which work supports which types of
awareness, and detailed descriptions are given in later sections.
Paper 1: The proposed lean-aware interaction technique on tabletop can facilitate
workspace, contextual, and social awareness when individuals and groups work with this
kind of hardware. We note that the system is aware of different unintentional touch
contacts of the user on tabletop to facilitate their activities in various scenarios.
Paper 2: The concept and design of NowAndThen can help social network sites improve
contextual-awareness in supporting user activities with social network.
Paper 3: Through the design of DigiMetaplan as well as lessons learned from an
exploratory study with the system, we devised a set of design considerations that can
better support workspace awareness and social awareness in distributed facilitated
brainstorming systems.
Paper 4: Ubitile demonstrates a novel interaction technique that allows users to
intuitively author vibro-tactile feedback; a widely-used medium to improve workspace
and contextual awareness in various computing systems such as virtual reality,
augmented reality, mobile and interactive surface applications.
10
The following sections provide the contexts of contributions of the works included in this
thesis. More specifically, these sections offer a survey of how awareness can be
supported by existing systems in four domains: tabletop interaction, social media,
creativity-support collaborative systems and vibro-tactile feedback authoring. Based on
this survey, I describe how the projects presented contributes to their corresponding
domain in better supporting awareness. Then, I summarize the contents of each work so
that readers can quickly have an overall understanding about them, followed by four
papers presenting details of the work included. Finally, the thesis concludes with
discussions and outlook on the works.
11
3 AWARENESS SUPPORT IN TABLETOP SYSTEMS
Tabletops have emerged as a widely-used platform to support collaborative work. They
have many affordances that make it suitable to support groups in many tasks. First, a
tabletop offers a large-size horizontal interactive surface allowing several users to work
simultaneously. Tabletop systems can also provide the users with a wide range of input
interaction techniques such as using keyboard and mouse, touch interaction using fingers
and stylus pen, manipulating digital information using tangible controls or even via
mobile devices (smartphone, tablet) wirelessly connected with the tabletos. Their
implementation cover a wide variety of domains such as drawing, games, exhibitions,
etc. With these affordances, tabletops can easily offer a versatile shared workspace for a
collocated group. Still, interaction issues among users or between users and the system
persist, reducing the performance of users and effectiveness of their work. To help
address such issues, a body of research has devoted itself to improving workspace and
contextual awareness of tabletop systems to provide suitable information and perform
appropriate actions for the users.
3.1 Workspace Awareness Supports in Tabletop Systems
Since a tabletop system is typically shared by multiple concurrent users, this platform
poses several interesting workspace awareness issues to investigate. Scott et al. [56]
studied the territoriality in collaborative tabletop workspaces. After observing users
facing several types of tasks and tabletop form factors, the authors suggested to partition
a tabletop’s screen into three territories: personal, group and storage spaces (see Figure
2). A personal territory of a tabletop user was typically located directly in front of the
user, and was almost exclusively used by that person to do his personal tasks. Meanwhile,
a group territory was usually located in the center of a tabletop system and shared by
everyone at the table. This separation of territories was formed naturally; the space
directly in front of each user being most easily reached by that person, and the center
being equally reachable by everyone. Sometimes, depending on the groups and tasks, the
areas between adjacent people were also used as group territories. In addition, some users
also used spare areas outside the personal and group spaces for storing items. These are
referred to as storage territories and can be present or absent in tabletop collaboration
depending on the task. The authors observed that user interactions and actions differed
depending on the types of territory. For example, if an artefact was in a personal space
12
of a person, it was considered to be owned by that person and others needed to ask for
permission to use it, whereas in the group space they could freely access the artefact
without permission. Scott et al. also found that the size of personal territories was not
fixed, but tended to expand or contract depending on the number of people at the table
and whether or not a person was working independently. The varying size of personal
territories also affected the size of group territories since these were formed from the
remaining areas.
Figure 2: Conceptual diagram of three types of tabletop territories [55]
To support different territories on tabletop, researchers have proposed many interaction
techniques to help users in defining those spaces. Scott et al. suggested that a personal
space can be roughly estimated as a 15x30cm area directly in front of its corresponding
user. Adopting this, Apted et al. [3] defined a personal territory of a tabletop user as a
fixed-size triangle directly in front of that user in their photo sharing application (see
Figure 3a). Likewise, these personal areas were also positioned at fixed locations on the
tabletop. However, as mentioned earlier, the personal territories varied in size and
location. To help address this, Schmidt et al. [52] proposed an interaction technique
called IdLenses, which allows users to perform bimanual hand gestures to explicitly
13
define their personal areas (see Figure 3b). However, although this technique allowed a
user to define a personal area of dynamic location and size, it required explicit
interactions to do so. To automatically determine the location of each user to position
his/her corresponding personal territory, DiamondTouch [10], integrated antenna arrays
under a tabletop surface to help address this. Similarly, Klinkhammer et al. [34] used
arrays of proximity sensors attached on four edges of a tabletop to determine user
locations to display personal spaces appropriately (see Figure 3c). Nevertheless, the
personal spaces of this system still had the weakness of having fixed size and shape,
making them sometimes overlap each other.
Figure 3: (a) Fixed personal territories in photo sharing application of Apted et al. [3],
(b) IdLenses proposed by Schmidt et al [52], (c) Dynamic-location personal territories
in system of Klinkhammer et al. [34].
Besides supporting collocated groups, researchers also extended tabletop workspace
awareness to remote collaboration settings. For example, Tuddenham and Robinson [47]
segmented the arms of users from images captured by a top-mounted camera on a
tabletop and projected the arms’ shadows on the tabletop on the remote side (seeFigure
4). Their observations on the system showed this design had some effect in helping users
on two sides coordinate and partition the workspace. However this interaction technique
still could not afford the territorial partitioning in a collocated group observed by Scott
et al. [56].
Figure 4: Distributed collaboration using tabletops: workspace awareness supported by
the projection of arm images of the remote participants.
14
3.2 Contextual Awareness Support in Tabletop Systems
Contextual awareness support in tabletop systems can be found in various forms but
mostly relies on touch interactions between users’ hands/devices and the tabletop surface.
Although the large multi-touch surface is an advantage of tabletop systems because of its
supported intuitive touch interactions, it can also be annoying to users due to
unintentional touches. A common case of this is that when a user uses a stylus pen to
write on a tabletop surface, his/her hand and arm usually also leans on the surface,
creating undesired touches with the tabletop system that can potentially ruin the user’s
work. This effect is usually caused by user fatigue or reaching distal objects. Because
these unintentional touches happen naturally, tabletop systems should accommodate this.
To address the problem of unintended touch interactions, several researchers have
focused on distinguishing intended and unintended touch. At the firmware level, some
hardware manufacturers such as Synaptics and Samsung increased signal to noise radios
to make it easier to distinguish between narrow activation signals generated by thin,
pointed objects such as styli, and larger, bulbous objects such as fingers, palms, wrists,
or forearms [2]. At the software level, Hinckley et al. [28] distinguished between touch
contacts of stylus pen and of human body parts such as hand or palm based on touch
contact sizes. Vogel et al. [61] abstracted the user’s hand and wrist into a circle and
intersecting rectangle located at a specified distance and angle from the stylus. At the
application level, applications such as Moleskine had a simple approach to distinguish
stylus touches and hand touches in sketching. They reserved a static rectangular area for
users to rest their hand while sketching. Touches in that reserved area were considered
as unintended in the sketching task. To better distinguish intended and unintended
touches in sketching, Schwarz et al. [54] used spatiotemporal touch features and iterative
classification to differentiate not only styli but also finger touches from palm touches.
Designers usually simply discard unintended touches in tabletop interaction to avoid
undesired effects on users. Besides that, other efforts also consider the attributes of
unintended and intended touches (e.g. size or angle) to adjust user interfaces accordingly.
For example, Brandl et al. [7] proposed a menu that is aware of the occlusion of arms
and hands on a tabletop and adjusts its location and shape to be convenient for users to
interact with (see Figure 5a). Leithinger and Haller [37] proposed a design that allows
users to draw paths for displaying dropdown menus to avoid physical occlusions on the
15
tabletop (see Figure 5b). By contrast, other works take into account the various shapes
of touch contacts of arms, hands and fingers to provide users a richer set of hand gestures
when interacting with tabletop systems. Marquardt et al. [40] suggested a set of hand
gesture designs facilitated by various shapes of user hands and hand-parts (see Figure
5c). Likewise, Cao et al. [65] developed a tabletop system that can recognize several
hand postures depending on the touch contacts of user hand and hand parts and adjusted
the behaviour of digital objects accordingly. The common point of these approaches is
that they require users to intentionally perform such gestural interactions; therefore hand-
part interactions in these cases are not unintended touches.
Figure 5: (a) Occlusion-aware menu [7], (b) User-drawn path menus [37], (c) Hand and
handpart-aware tabletop interaction with TouchID toolkit [40].
Contextual awareness can also be facilitated through user position at the table. For
example, the adaptive personal territories designed by Klinkhammer et al.[34] can merge
together to become a composite space if the system detects corresponding users standing
close together for a long time.
Author’s contributions
Although unintended touch interactions on tabletops have been investigated in previous
studies, there has been no attempt to facilitate unintended touch contacts to improve user
convenience and effectiveness in working with tabletop. In other words, there has been
no attempt to categorize and make use of those unintended touches to better support
users. We believe that those unintentional touches (here referred to as lean contacts) may
convey meaningful information related to user’s posture, intention or even their mental
state. For example, based on the lean contacts of user forearms, palms and hands with
the table, we can recognize user postures and actions. Leaning elbows on the table means
the user is thinking, or lean contacts of forearms and hands open and orienting towards
the centre of the table mean the user is actively working. Regarding tabletop territoriality,
if lean contacts show forearms and hands forming an arch closed by an edge of the table,
16
it could mean that the enclosed area has been unintentionally claimed as the personal
space of the user.
In order to recognize these gestures and actions, it is important to recognize different
types of lean contacts generated by different human body parts. Therefore, my
contributions to tabletop interaction include a categorization of lean contacts into four
classes of gesture (seeFigure 6) and a proof-of-concept algorithm to demonstrate the
feasibility to recognize those gestures. Based on these, my contributions also include
envisioned scenarios taking advantage of the recognition of lean gestures such as
dynamically determining user personal territories and designing adaptive graphical
widgets based on the location, size, shape and type of lean contacts. These will help
improve workspace and contextual awareness of the tabletop system. Furthermore, by
recognizing user state through lean gestures, we can also improve social awareness in
collaboration. For example, if through lean gestures the system recognizes that a
collaboration participant is too active while another is too passive, it can give some hints
to trigger the activeness of the passive one. This would help everyone participate
relatively equally to the collaborative task and establish balanced communication among
all the participants. Figure 7 illustrates these use cases.
Figure 6: Tabletop lean posture classes (1-4): elbow, forearm, partial hand, and full
palm (standing and seated). The black shapes are lean contacts between a user’s body
part(s) and a table’s surface.
Figure 7: Example use cases of lean gestures recognition: (a) dynamically determining
personal territories, (b) designing adaptive graphical widgets, (c) recognizing user state
to support social awareness in collaboration.
17
4 AWARENESS SUPPORT IN SOCIAL NETWORK
SITES
Social networking has become enormously popular due to the rapid development of the
Internet. Social network sites (SNS) are major platforms for connecting people and
sharing information. On SNS, users can share information (text, image, and video),
interact with each other (vote a share, comment on a share, chat or even talk via video
conferencing) or even cooperate (e.g. co-organizing events, making SNS collaborative
environments). These features are the reasons why SNS still keep attracting more users.
While the number of people joining SNS still keeps increasing, the amount of data shared
by SNS users has consequentially exploded. Due to this massive amount of data, it is
becoming more and more difficult for a social network user to quickly find information
of interest. It is also challenging for SNS to present information to users since they have
to make sure that the users can see their information of interest as early as possible. To
better support users, it is necessary for SNS to know about them. Therefore, awareness,
more particularly contextual awareness, is important to be supported by SNS. SNS need
to be aware of information about the users such as their location, their relationships, and
even their habits, interests, and beliefs. Based on that, SNS act and present information
accordingly to ensure that the presented information is highly interesting to the users.
In fact, contextual awareness has been widely implemented in SNS. Facebook, for
example, has many representative examples of contextual awareness in its features. For
example, Facebook only shows a user the updates from the people that the user followed
to ensure not showing undesired information to the user. Similarly, when a user clicks
on a link of an article shared on Facebook, he/she will see suggestions for other articles
published by the same publisher of the link. These suggestions rely on the assumptions
that other material by the same publisher could also be of interest to that user. Likewise,
when suggesting users for friending, Facebook considers candidates based on things in
common with the user such as friends, hometown, schools, companies, etc. Similar
features like this can be found on other SNS. Youtube has several video suggestion
strategies. A video can be suggested to a user because it has things in common (e.g
publisher, tags, similar texts in title) with the videos that the user has watched recently.
Or the user can also be suggested a video that he/she had stopped watching earlier, in
case want to continue watching it. Contextual awareness is also present when a user
performs a certain action on Facebook, for instance when a user shares photos Facebook
18
can extract the location information from the photos’ EXIF and automatically suggest
location tags for the user, helping save tagging time. Similarly, for users who usually tag
people in their photos but are tired of searching people’s names in their friend-list,
Facebook speeds up this process by face recognition and automatically suggests names
of the recognized people.
In addition to easing the tasks of users, contextual awareness is also integrated in SNS to
support user emotion, mentality and social relationship. Since a user may have several
events in a day on SNS such as becoming friend with someone, sharing a photo or a
status, checking in a location etc., when time passes by these events can be buried by a
huge amount of data and event and the user may forget them. To remind the user about
these events, Facebook has a feature called “On This Day” which shows the user his/her
events that happened on the same day in past years (see Figure 8a). Carousel, a social
application for photo management, also has a similar approach in reminding users about
photos they shared on different SNS on the same day in the past (see Figure 8b). While
the relationship between the current day and the day when the events happened is
arbitrary, it does not really aim to aid the user any specific task. Rather, its purpose is to
help users reminisce about things and people that can be meaningful to them, which can
help strengthen their narrative identity, self-esteem and maintain relationships between
them and other people [38,41].
(a) (b)
Figure 8: (a) Facebook’s On This Day, (b) Carousel.
Social awareness is also supported by some SNS. For example, Facebook Messenger -
Facebook’s chat application - has various icons showing different statuses of a user such
as offline, online or busy. When looking at the icon, other users can determine their action
accordingly such as simply jumping to chat with that person or conversely avoiding
19
disturbing them if they are busy. Emoticons are also employed by chat features of SNS
to facilitate social awareness by helping people express their emotions better.
Author’s contributions
One of my contributions in this thesis is the design of a photo recommendation tool called
NowAndThen facilitating reminiscence in SNS users. Although time-based reminiscence
features like Facebook’s On This Day do have positive effects on users, they can also
encounter the problem of suggesting undesired memories due to solely relying on time.
To partially address this, our designed photo recommendation tool relies on the visual
similarity between photos. We based our design on the assumption that suggesting photos
that look similar to the one a user is viewing can also be interesting to the user. Besides
that, our design of the tool also includes a side-by-side view that allows users to compare
two photos taken at different times and see how things have changed (see Figure 9). This
helps improve the effectiveness of the tool in triggering the reminiscence effect in users.
The preliminary user study with the tool revealed interesting behaviours and feedback of
users regarding the features of the tool as well as photos suggested by the tool. These
could be helpful in designing context-aware features on SNS.
Figure 9: The design of NowAndThen - social-network based photo recommendation
tool based on photos’ visual similarity; the screenshot shows relevant photos in side-
by-side view.
In addition, while augmented reality (AR) headsets are emerging as a promising
technology for interacting with the real world, combining the concept of NowAndThen
20
and augmented reality can open a fascinating approach for exploring historical places.
Once we can retrieve photos that look similar to what a user is viewing, we can blend
them together. We envision that this blend of photos will show the users how the present
and the past of the same things can be compared in front of the eyes of the user. This
design can be beneficial in the contexts of museum, history learning and tourism. Figure
10 demonstrates an example where the algorithm used in the design of NowAndThen
can blend two photos of the same place taken at different times together.
Figure 10: (a) photo of the place at the present, (b) photo of the place in the past, (c)
The past and present photos are blended in one.
21
5 AWARENESS SUPPORT IN DISTRIBUTED
CREATIVE GROUP WORK
In office settings, there are many creative activities that need to be carried out by
collaborative groups. These range from project planning, early sketching and designing
to brainstorming solutions for problems. In these activities, it is important that the topic
discussed is viewed from different perspectives; therefore, it is necessary to involve many
people to work together in collaborative settings. In such sessions, the interactions
between the participants and artefacts involved are all crucial to make sure that people
can easily express their opinions and contributions. In conventional settings of such
activities, artefacts involved are mainly physical tools such as pen, paper, sticky notes,
flipchart/whiteboard or even room walls for writing, sketching and arranging ideas. Some
settings such as for product designs also may involve physical prototypes.
Due to the rich communication between participants and the interactions between
participants and artefacts, social and workspace awareness are two important factors in
such activities. For example, for social awareness during a collaboration session, a
participant listens to another’s opinions and observes behaviours and expressions of that
person to determine his/her own reactions such as agree, disagree or elaborate on that.
Workspace awareness also happens frequently in such sessions. For example, when
discussing some idea on a flipchart or whiteboard, a participant can simply perform
deictic gestures to refer to it (i.e. using hands, fingers or even a pen to point to the idea)
and just simply vocally mention it with the words “this” or “that”, and the other
participants can immediately understand where to orient their attention.
Nevertheless, such conventional collaborations suffer many limitations. For example,
using normal pen, paper and flipchart, it is difficult to archive and version the result of a
collaboration session (e.g a brainstorming outcome with ideas arranged on a
flipchart/whiteboard) to participants to retrieve later if needed. The most popular
approach is to take a photograph of the latest state of the artefacts involved in the session.
However, this can be annoying since participants may need to continuously take pictures
if they want to save several stages of the session. Worse, until recently such conventional
creative collaborative activities were also impossible to be conducted in distributed teams
where members were located in different locations, because remote members could not
see the collocated members or access the artefacts.
22
Thanks to the advent of internet and hardware technologies, computing systems were
developed to address these problems. The issue with archiving and versioning can be
easily resolved thanks to new input devices such as keyboards, touchpads and even
smartpens which can digitize the user input. However, remote collaboration still remains
challenging despite considerable research effort. The main challenge is to achieve a
similar social and workspace awareness among dispersed participants as when they are
physically collocated. To overcome this challenge, research efforts have mainly looked
into capturing and presenting remote participants as well as the workspace of such
activities.
Since the 1980s, researchers have developed computing systems for distributed teams to
collaboratively perform brainstorming. An example of these systems is GROVE, a real-
time shared text editor with telepointers that can be seen and controlled by all participants
to facilitate workspace awareness. However, social awareness is almost unsupported by
this system since participants cannot see each other. After video conferencing was
introduced, researchers started to integrate video streams into collaborative systems
supporting creative tasks. TeamWorkStation of Hiroshi Ishii [31] is one of the earliest
systems of such (see Figure 11a). This system consists of windows showing video
streams of remote participants placed aside the workspace window (typically a shared
window for sketching or a shared document viewer), and allows all participants to
interact with the workspace as well as see the facial expressions of their remote partners,
consequentially partially maintaining workspace and social awareness. However,
separating video stream windows from the workspace led to confusions in eye contact of
participants. Furthermore, the system also could capture and present hand gestures of
users, leading to reduced social and workspace awareness in the system. To overcome
the problem of presenting hand gestures, Tang and Minneman proposed VideoDraw [58];
a shared drawing tool which integrates the shadow of the hands of a remote partner onto
the drawing space of a user (see Figure 11b). However, facial expressions and eye contact
are missing in this system. After that, Ishii et al. designed ClearBoard [30] based on the
“talking through and drawing on a transparent glass window” (see Figure 11c). The
system overlays the shared drawing space on the top of the video stream of the remote
participant. The video stream of each participant is captured by a camera placed behind
a liquid crystal screen, therefore able to capture both facial expressions and hand gestures
of the participant on the workspace. With this design, the ClearBoard can maintain both
23
workspace and social awareness. However, the limitation of this system is the bulky
hardware setup and that it is only able to support remote collaboration between two
endpoints.
Figure 11: (a) TeamWorkStation [31], (b) VideoDraw [58], (c) ClearBoard [30].
The approaches of VideoDraw and ClearBoard were then adopted by several systems to
support other types of creative groupwork in distributed settings. Everitt et al. [15]
extended The Designers’ Outpost system [33] to become a distributed brainstorming
system using sticky notes and interactive whiteboard. To facilitate workspace awareness,
the system shows the shadows of the whole body of the remote participant on the
brainstorming space (the interactive whiteboard) so that the collocated participants know
where on the whiteboard the remote user is referring to (see Figure 12a). However,
similar to VideoDraw, this system also suffers the problem with transferring facial
expressions. Holoport [35] has the content-on-video approach similar to ClearBoard to
facilitate workspace and social awareness in various collaborative tasks in distributed
teams (see Figure 12b). CollaBoard [36] extracted the image remote participant from the
captured video and put it on the top of the workspace in order to create the impression of
working side by side between disperse people (see Figure 12c). Teleboard [21] adopted
the content-on-video metaphor of ClearBoard to build a note-based brainstorming system
for distributed teams in other to deliver both users gestures and facial expressions. In
addition to the above, together with the development of depth cameras, other systems
such as 3DBoard [67] (see Figure 12d), ImmerseBoard [27] utilized this technology to
capture the full body of a participant and visualize it on the shared workspace in different
ways such as work-through-glass or side-by-side. However, the 3D reconstruction
quality of the human body using current off-the-shelf 3D cameras is still limited, hence
affecting the quality of social awareness in the communication.
24
Figure 12: (a) Distributed Designers’ Outpost [15], (b) Holoport [35], (c) CollaBoard
[36], (d) 3DBoard [67].
Author’s contributions
While previous research can effectively address the problems of workspace and social
awareness in distributed teams performing creative group-works, it still assumes that all
the sides involved in such activities have equal roles and similar hardware setups.
Further, hardware setups in those works until now are still only suitable for collaboration
between two sides. In fact, distributed teams in creative groupwork can have different
settings. Creative groupworks, especially brainstorming, are usually carried out in
facilitated settings in which one person will coordinate and facilitate the sessions and
have greater accessibility to the artefacts involved in the sessions. Although studies have
shown the importance of facilitation in the success of a creative collaboration session,
digital systems supporting remote creative groupworks still overlook this factor, and
consequently result in reduced effectiveness of such activities in distributed settings.
Second, distributed settings can be in form of one or many individuals working remotely
with a collocated group of members. They do not have similar hardware infrastructure,
for example, the remote individuals may just have personal devices such as tablets while
the collocated team is usually equipped with interactive screens and personal devices.
Having different hardware settings like this may also affect the awareness in their
communication in the collaboration.
One of my contributions in this thesis is an effort aiming to improve awareness in
distributed facilitated brainstorming. “DigiMetaplan” is a facilitated brainstorming
system that allows remote individuals to participate in a facilitated brainstorming session
with their collocated team using off-the-shelf tablets. The design equips the remote
individuals with necessary tools for them to have equivalent accessibility to the
brainstorming whiteboard to the collocated participants. More particularly, remote
individuals are equipped with a note editor to input ideas and a pointing highlighter to
aid them in expressing their opinions using deictic gestures. These features help maintain
25
workspace and social awareness between the remote and collocated participants without
breaking the structures and rules of facilitated brainstorming. Besides that, through an
exploratory user study with DigiMetaplan, we derived several lessons learned about the
design of DigiMetaplan as well as accompanying video conferencing applications to
better improve workspace, social and even contextual awareness in future systems
supporting facilitated brainstorming.
Figure 13: DigiMetaplan system supporting distributed facilitated brainstorming. Left:
facilitator working with brainstorming whiteboard in collocated group. Right: remote
participant joining the brainstorming session using a tablet.
26
27
6 AWARENESS SUPPORT FOR VIBRO-TACTILE
FEEDBACK INTERACTION IN COLLABORATIVE
ENVIRONMENTS
One of the major limitations of digital content is that it cannot parallel the sense of touch
that physical objects can. This reduces the realistic experience in the interaction between
users and digital content and makes computing systems less favoured in scenarios where
having the sense of touch is crucial. To help users ‘feel’ physical attributes of digital
content via the sense of touch, HCI researchers employ several approaches to apply
physical feedback on users. Among those, vibro-tactile feedback is one most the most
common approaches. Vibro-tactile feedback is supported by using applying vibrations
generated by actuators on users to express the sensation of vibrating. This feedback is
favoured due to its safety, low hardware cost and wide range of applications. Vibro-
tactile feedback can be used to help a user realistically experience the motions of an
object, the strength of a force, the stiffness of a textured surface or an alert signal in
digital content, for example. [19,26,43,44].
In collaborative systems, vibrotactile feedback has also been widely integrated to
improve awareness between users. Oakley et al. [Oakley et al., 2001] employed vibro-
tactile feedback into a shared text editor to enhance the interaction between multiple
cursors so that users could be more easily aware of other user’s location on a document.
Henrysson et al. [25] used vibro-tactile feedback in an AR multi-player game to inform
players about collisions or rule violations, maintaining their workspace and contextual
awareness in the game. Yatani et al. [66] investigated the effects of vibro-tactile feedback
of mobile devices in spatial coordination in a shared workspace. They realized that spatial
tactile feedback can reduce the overload of information in visual space and gently guide
the user’s attention to an area of interest. McAdam and Brewster [42] used smartphones
to provide tactile feedback in performing text entry with touch-screen keyboards on a
tabletop. By providing different vibro-tactile feedback for different tactile events such as
fingertip-over, fingertip-click, and fingertip-slip, the system helps users have better
awareness on where their fingers are on the tabletop (workspace awareness) and the event
they are going to perform (contextual awareness); improving their performance in the
text-input task.
28
In distributed teams that need to collaborate over a virtual environment, vibro-tactile
feedback also plays a primary role in improving the realistic and immersive experience.
Galambos [18] used a glove that can produce vibro-tactile feedback in order to fulfil the
sense of touch for an operator in manipulating an object in a virtual reality (VR)
environment. This will be useful for the scenarios in which multiple operators need to
collaboratively manipulate a virtual object together in a virtual setting, because each
operator can have better awareness on which part on the virtual object they are interacting
with. Pfeiffer and Stuerzlinger [46] also showed that vibrotactile feedback was beneficial
for 3d virtual hand pointing; potentially improving user workspace awareness in a virtual
3D environment.
In HCI research related to vibro-tactile feedback, besides rendering vibro-tactile
feedback on users, authoring vibro-tactile feedback is also an important phase in the
production of this feedback in a computing system. Vibro-tactile feedback needs to be
correlated with the visual information that a user is viewing. Because uncorrelated vibro-
tactile feedback can reduce the realistic experience and satisfaction of users [Danieau et
al., 2014]. The most popular approach for users to author vibro-tactile feedback is to use
a graphical user interface such as posVibEditor [48], Hapticicon [14] or even just using
a slider to determine vibration intensities like in many mobile applications. Obviously,
this approach is not intuitive to non-expert users because it is hard for them to imagine
the sense they will feel on their skin just by looking at graphical items. Chi et al. [8]
proposed a more intuitive approach that can automatically generate vibro-tactile patterns
based on sounds of a media item. However, this can be troublesome if the sound is not
related to the visual content. In my Master’s thesis [9,17,32], I investigated a method that
records the inertial data from built-in sensors of a mobile device and aligns it with the
video content captured by the device’s built-in camera. The resulting inertial data (more
specifically acceleration) is quantized on its energy and converted into vibration patterns.
With this, vibro-tactile feedback can be unobtrusively synthesized with high correlation
with the visual content. Vibro-tactile feedback can also be rendered on-the-fly with a
mobile device using a built-in vibrator. However, this approach can only work with
videos captured by the device, not an arbitrary video. Further, this approach also does
not work with large displays like tabletop systems due to the size and weight of this type
of device.
29
Author’s contribution
In this paper, I contribute an interaction technique that allows a user to intuitively author
vibro-tactile feedback for digital content on tabletop. I introduce Ubitile - a finger-worn
(ring) I/O device that can allow users to create a vibration pattern using the movements
of their fingers and also feel the created vibration pattern directly using the ring. A user
just needs to wear the ring, select digital content on a tabletop with which s/he wants to
associate a vibration pattern, and wave the finger in the way s/he feels best describes the
vibrating effect of the digital content (seeFigure 14). The system will automatically
generate the vibration pattern based on the recorded inertial data from the sensor of the
ring. After that, still wearing the ring, the user can feel the vibration pattern s/he created
when touching on the corresponding digital content on the tabletop. With this interaction
technique, each user can create personalized vibration patterns for specific content,
meaning that different users may feel different feedback from the same digital object.
Therefore, the intuitive nature of creating vibration patterns is not the only advantage of
this interaction technique. Thanks to the use of a wearable device for authoring and
rendering vibrotactile feedback, this interaction technique can facilitate personalization
and privacy in tabletop interaction- a typical collaborative platform. Hence, it opens new
design dimensions to improve workspace and contextual awareness in interactions with
tabletop as well as other collaborative platforms.
Figure 14 Ubitile: (a) Illustration of finger wearing the ring and waving to create a
vibration pattern, (b) Demonstration of a user selecting a digital content on tabletop and
authoring the associated vibrotactile feedback with the ring.
30
31
7 SUMMARY OF PAPERS IN THE THESIS
7.1 Paper 1 - Towards Leaning Aware Interaction with Multitouch
Tabletops.
In interacting with multi-touch tabletops, users tend to lean different parts of their body
such as forearm, palm, or hand on the table due to fatigue or selections of distal objects.
These intended interactions (here referred to as ‘lean interactions’) can lead to undesired
events on the tabletop systems. In handling these unintended interactions, a common
approach is to disregard them; however, we believe lean interactions can contain
meaningful information about postures and states of users, hence useful in improving
user interactions with tabletop.
In investigating this kind of interaction, we first conducted a user study with four
participants to categorize lean gestures used with tabletop systems. We proposed four
classes of lean gestures including elbow, forearm, partial hand and full palm based on
the difference in their shape, size and the body parts generating them. Then we built a
proof-of-concept algorithm using ASM models to recognize lean gestures. To train and
evaluate the algorithm, we ran a second user study with twenty participants. The
evaluation showed the highest recognition accuracy of 94% for full palm and the lowest
one of 72% for forearm. Based on this, we envisioned several scenarios where lean
interactions can be exploited to support users such as in dynamically determining
tabletop territories, designing adaptive widgets or combining with theories in
conversational gestures to better support group collaboration.
7.2 Paper 2 - NowAndThen: a Social Network-Based Photo
Recommendation Tool Supporting Reminiscence.
Thanks to the development of social network sites (SNS), it has become easy to post
photos to share memorable moments, emotions or locations visited. However, due to the
massive number of photos shared, people barely have time to revisit their past photos.
While revisiting past photos that have the same subject as the one a user is currently
interested in could lead to reminiscence and beneficial to the user, existing tools cannot
support this effectively. Therefore, we propose the concept and the design of
NowAndThen - a social network-based photo recommendation tool that relies on the
visual similarities between photos.
32
To derive design features for NowAndThen, we ran an online survey on user interest and
habits in sharing and revisiting photos. Then we designed the photo recommendation
features and the user interfaces of NowAndThen based on the results collected from 521
answers. When a user is viewing or sharing a photo, the NowAndThen tool can suggest
relevant photos that look similar to the one the user is viewing. The tool also offers
different view modes and user interactions with the relevant photos. We then built a
prototype of NowAndThen as a mobile application using an in-house image retrieval
algorithm, and conducted a preliminary study with 12 participants. Generally, we
received positive feedback on the benefits that NowAndThen brought to them. Most
reported that NowAndThen helped them revive memorable moments in the past and
reconnect with their old friends. Besides that, our observations on how participants
interacted with relevant photos also revealed interesting design considerations for future
systems.
7.3 Paper 3 - DigiMetaplan: Exploring an Interactive System for
Facilitated Brainstorming.
Facilitated brainstorming is brainstorming coordinated by one or more facilitators. This
technique is effective in helping teams produce outcomes from clarified thinking and
global participation thanks to its structured interaction. Facilitated brainstorming is
widely used in industries; however, despite several digital solutions helping distributed
teams perform brainstorming, none of these are designed with clear role separations
among participants to support this brainstorming technique. This is one of the reasons
why brainstorming in distributed teams may not be optimally effective.
To address this, we designed DigiMetaplan; a system that specifically supports facilitated
brainstorming both in collocated and distributed settings. DigiMetaplan consists of two
main components: DigiMetaplan Board; an application running on interactive
whiteboard used by the facilitator and DigiMetaplan Pad; a mobile application used by
other participants in the brainstorming session. Remote individuals can also use the
DigiMetaplan Pad, because our design makes sure that they have equivalent accessibility
to the brainstorming board.
Next, we ran an exploratory user study with five groups of four using DigiMetaplan and
Skype to brainstorm in a distributed setting (3 collocated + 1 remote). We investigated
the communications and interactions of all participants using both quantitative and
33
qualitative measures. From these investigations, we derived lessons learned from the
design of DigiMetaplan and the accompanying videoconferencing solution. These
lessons may be useful for designing future system supporting facilitating brainstorming.
7.4 Paper 4 - Ubitile: A Finger-Worn I/O Device for Tabletop
Vibrotactile Pattern Authoring.
Vibrotactile feedback has been utilized to improve user experience in interaction with
digital content on many platforms from virtual reality to mobile devices and even
tabletops. While the mobile platform offers motion sensing as an intuitive input for
creating vibrotactile feedback, this technique is hard to be applied to tabletops where the
screen becomes larger. To address this, we propose a concept of Ubitile - an interaction
technique using a finger-worn device (in the form of a ring) offering both motion sensing
and vibration rendering for authoring vibrotactile feedback on tabletops. Using Ubitile,
a user can intuitively create a vibration pattern using a freehand mid-air motion.
Additionally, the user can also experience the created vibration pattern using the device.
In short, Ubitile, integrates both input and output spaces within a single wearable
interface, jointly affording spatial authoring and active tactile feedback on and above
tabletops.
34