collaborative image- based tagging and interactive mapping on tabletops presented by: glen whitaker...

Collaborative Image-based Tagging and Interactive Mapping on TabletopsPresented By: Glen Whitaker

Supervisor: Judy Kay

Research Question

How to Support Tabletop interaction for organisation of Content?

Why is it important? Novelty of User Interface Organising is fundamental

Especially file systems

Constraints? Limited methods of input

Challenges? No established toolkits, primitives, UI design No guidelines Research - Cruiser Tabletop as exploratory framework

Refined Research Question

How to Support Tabletop interaction for organisation of Content using Tagging and Maps?

Why? Tagging text is similar to associating file name and

location to files in a file system Maps similar concept to the folder structure in a file

system

Sub-Problem: How to Support Tagging at Tabletop Adding Tags to Images

Linking Images to maps

Single text/image tags

Multiple Tags

Refinement: Elements of Tagging

Way to organise and find digital artefacts

Key interface tasks for tagging Defining tags Adding tags to objects Viewing tags Searching tags Deleting tags

How to achieve this on Tabletop Interfaces?

Approach 1: MyTags Inspired by TeamTag

Morris, M. R., A. Paepcke, T. Winograd, and J. Stamberger. "TeamTag: Exploring Centralized Versus Replicated Controls for Co-Located Tabletop Groupware." Proceedings of the SIGCHI conference on Human Factors in computing systems CHI: ACM Press, 2006. 1273 - 82.

Text-Based Tagging

Adding Text to Images

Adding Multiple Text to Images

Approach 2: MyMaps

Image-based Tagging in context of maps

Adding Images to Map Locations Placing image on a single map location/region

Adding Images to Multiple Map Locations Placing an image to multiple regions on map

Prototype

Defining Tags

Tags are predefined (externally in a file/database)

Tags represented by special objects

Adding Tags

MyTags MyMaps

Single Tag (1)

Move text to back of image

(1 gesture)

Dwell image on Map within region(s)

(1 gesture)

Multiple Tags (n)

Repeat single

(n gestures)

Same as single

(1 gesture)

Viewing Tags

MyTags Flip image over to view list of text

MyMaps Dwell on tag to view image

Viewing is same for single and multiple tags in both MyTags and MyMaps

Deleting Tags

Move tag to blackhole

Same for tags in both MyMaps and MyTags

Same for single and multiple tags

The Framework

Cruiser Images Blackhole (garbage collection object)

Tagging Primitives Adding Text Tags Adding Image based tags Adding Regions View Tags Delete Tags

Previous Elements

Previous Elements used by MyMaps: Blackhole (garbage collection object) Images (used to represent exhibits)

Audio can be associated with image Images can have other images placed on back

Images Blackhole

New Elements “New” elements of the Cruiser Tabletop

Map Resources Map Region Creation Tool “Pin Images” similar to pins on a notice board Image Region Identification

Region Tool

Pin Images

Map

Region

Back of Image

Text Object

Text Attachment

Architecture

Image

Original Sharepic Image

MyMaps Image Extensions

Pin

Text

Region Tag Component

Layout

Map Layout Map

Region Tool

Resource

Original Sharepic Resource

MyMaps Resource Extensions

Environment

Original Sharepic Environment

MyMaps Environment Extensions

Original Sharepic Blackhole

= Trent’s Cruiser Elements = New Elements

Evaluation Performance

Successful Tagging with MyTag Successful Tagging with MyMaps

Other usability elements Comparison of affordances Learnability Error Rate Error Recovery

Collocated Collaboration UI aspects Duplicated Maps UI Facilitates Collaboration

Evaluate Effects of Short term memory load Images Maps

Evaluation Design Qualitative Evaluations

6 Pairs (User A, User B)

Asked to design their own museum

Double cross over condition (TeamTag and MyMaps + MyTags)

User A and B given different sets of text attached to 10 images

Questionnaire for effective issues: Preference, attitudes, perspective

Evaluation Steps Tutorial

Task 1 Design virtual Museum

Task 2 Free-form virtual Museum

Tutorial of cross-over system

Repeat Tasks 1 and 2 for cross over system

Questionnaire

Double cross-over to control first system effect

Evaluate Collaboration and usability without design

limitations

Encourage Collaboration in Tagging Context

Participant Pair

TeamTag: Time (minutes)

MyMaps + MyTags: Time (minutes)

Notes Relating to Participant Background

1 (A, B) 27 33 Minimal Computing background

2 (A, B) 14 19 No prior contact with each other

3 (A, B) 13 19 Experienced Tabletop Users

4 (A, B) 16 12 Minimal Computing background

5 (A, B) 45 14 No prior contact with each user affected early communication. Mixture of participant with minimal computing background and experienced computer user

6 (A, B) 41 21 Mixture of participant with minimal computing background and experienced computer user

= Represents System used Second (always faster)

Average of System being used First (minutes) Average of System being used Second (minutes)

TeamTag MyMaps + MyTags TeamTag MyMaps + MyTags

34 23 18 16

= Represents System used Second = Preferred MethodX

User TeamTag MyMaps + MyTags

Pile Region Replicated Maps

Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X



Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X


10/12 Preferred Second System

2 preferred MyMaps even as first system



Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X


2 People (1 Pair) liked bothOthers evenly split

Some commented regions suited task better and less

planning needed



Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X= Represents System used Second = Preferred MethodX

5 People (2 Pairs + 1) liked but hardware limitations prevented effective use



Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X


All participants liked Image Tagging and could use

comfortably



Image Tags

Fixed Text

1:A X X X X X

1:B X X X X X

2:A X X X X

2:B X X X X X

3:A X X X X

3:B X X X X

4:A X X X X X

4:B X X X X X

5:A X X X

5:B X X X X

6:A X X X X

6:B X X X X


Manly yes, participants lost tags creating objects

Pair 1 preferred flexibility to organise

Performance Results

Successful Tagging with MyTag Association of Text to objects performed with low

error rate Pairs completed tasks successfully using method

Successful Tagging with MyMaps Association of Images was performed with low error

rate Pairs completed tasks successfully using method

Other usability elements Swipe action for text tagging was liked and not really in the

design

Learnability Easy to learn tagging primitives

Error Rate/Recovery Low error rate for Swipe/movement text association technique

Errors were due to having multiple images flipped within path of movement

Low error rate for image tags on maps Errors were mainly a result of lack planning suitable location

Image Tags Errors fixed by deletion of reference point and further planning

Text Tags Errors fixed by deletion of attached text

Collocated Collaboration

Duplicate Maps Participants did not use replicated maps as expected

Hardware limitations - multiple use - affected this 3 out of 12 participants used second map as a

thumbnail view of the focus map

UI Facilitates Collaboration Participants shared text objects Participants collaboratively planned image tag

locations Participants collaboratively planned text tags for each

image

Effects on Short-Term Memory Load Images

Images are flipped in tagging, causing short-term memory load but avoiding confusion

Images could not be semi-transparent since it would conflict with other functions

Maps Dwelling needed to view images relating to tags image tag design reduced clutter and confusion about map

objects as the cost of the need to check what reference points represented

Participants were observed to have minor problems remembering tags with the small sample of images and text

Caveats

Hardware Mimio capturing technology used for Cruiser only

allows one interaction at a time

Software Objects are 2-Dimensional representations Text predefined in external file loaded at start up Precision of interaction depends on accuracy of

calibration

Future Work Handwriting for tag inputs

Backend database for efficiency (minimise I/O)

Personalised recommendations based on tags

Link tagging to file system – (Anthony Collins thesis)

Contribution

Significant progress in understanding how to support collocated collaborative planning at a tabletop

Insight into new gestures for image tagging and text tagging on tabletops

Swipe gesture discovered

No more slides. Click to continue.

Results

Was 6 pairs enough? Sample showed consistent patterns relating to Image

Tagging

Provided substantial usability, performance, collaboration and affordance results

More detailed and extensive evaluations required for further results

Background: Virtual Museum Implementation

Virtual Museums Digital Technology being used to display exhibits while preserving the

contents of an exhibit in digital format Augmented Reality and 3D Modelling advancements allow

photorealistic exhibits

Huang, C., C. Chen, and P. Chung. "Projects in VR: Tangible Photorealistic Virtual Museum." IEEE Computer Graphics and Applications: IEEE, 2005. pp 15 - 17.

Direct Touch Interfaces

DiamondTouch Tabletop Direct-Touch Based Interaction Unique identification based on circuits through chairs/pads

Other Touch based tables include: SmartSkin, Entertaible and Misto

Ryall, K., M.R. Morris, K. Everitt, C. Forlines, and C. Shen. "Experiences with and Observations of Direct-Touch Tabletops." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006.

Lumisight – Rear Projection Design

Lumisight Tabletop Projectors inside the table project different images in different

directions Number of users limited to number of projectors

Kakehi, Y., T. Hosomi, M. Iida, T. Naemura, and M. Matsushita. "Transparent Tabletop Interface for Multiple Users on Lumisight Table." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006.

TeamSearch TeamSearch

Use of special widgets to represent metadata shared by all users Search images through use of widgets Search through manually tagged images using widgets

Morris, M.R., A. Paepcke, and T. Winograd. "Teamsearch: Comparing Techniques for Co-Present Collaborative Search of Digital Media." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006. pp 97 - 104.

Widget

Metadata inside widgets

Personal Space

Query Objects

TeamSearch Added Tags

Tags in the original implementation were added manually to each image

Searching Users search the photos by placing query object onto the metadata

values Thumbnails of matching photo’s appear in that user’s personal space

Tags are Metadata that matches the contents of the widgets

TeamTag TeamTag

Presentation and association of metadata Centralised-control (TeamSearch approach) and Replicated-control

design

Morris, M. R., A. Paepcke, T. Winograd, and J. Stamberger. "TeamTag: Exploring Centralized Versus Replicated Controls for Co-Located Tabletop Groupware." Proceedings of the SIGCHI conference on Human Factors in computing systems CHI: ACM Press, 2006. 1273 - 82.

Centralised-Controls Design Replicated-Controls Design

TeamTag Centralised-controls design

Implementation of TeamSearch widgets Users attach metadata by tapping image,

then subsequently selecting a piece of metadata in a widget

Replicated-controls design Presents metadata categorised into

columns that are presented in front of each user

Users attach metadata by tapping image, then subsequently selecting a piece of metadata in the collection presented in front of them

TeamTag Added Tags

Tapping the image to be tagged and the subsequent piece of metadata selected by that user is attached

Multiple tags can be associated to images using this method

Tags are Defined sets of metadata

Viewing Tags Popup lists present what metadata is associated with an image Popup lists activated by a user touching an image with two fingers

Centralised Vs Replicated Evaluated in context of zoo, labelling each image with

appropriate metadata in groups of 4

19 out of 24 subjects preferred replicated metadata interface

Collaborative evaluation of replicated interface showed on average 1.24 users contributed to the tagging of an image

Group collaboration and communication levels presented no significant difference

Google Maps Google Maps

Interactively move around map of world Zoom in on satellite and road maps of areas around the world Identify businesses on the maps by providing business info to Google

to tag based on street address WikiMapia provides user defined region tagging

Google Maps Tour. Google. Available: http://www.google.com/help/maps/tour/. 10th of August 2006.

Google Maps Added Tags

WikiMapia – Drawing of a box on the map to indicate a region Google Maps – Entering of data with the location being labelled by the system

based on data inputted about a business

Searching Searches performed by comparing text inputted to the metadata that is

associated with a tag

Tags WikiMapia – regions are user defined boxes with a text tag identifying what the

region is Google Maps – identifies single addresses with business details associated

with that address (limited to USA, Japan, UK, China and Canada)

Point Vs Region WikiMapia allows user identification of regions Google maps allows identification of single addresses on maps

The Cruiser Cruiser Tabletop

Uses Mimio® to Capture Interaction Uses Pen stylus devices for interaction Number of users limited to Pen Numbers

MyMaps and current software on the cruiser can be implemented using DiamondTouch interfaces

Trent Apted’s First Year of PhD presentation on Photo sharing with Sharepic system and the Cruiser Tabletop

Map Resources Contribution

Explore how collaborative planning activities can be performed on tabletop interfaces

Provides a context for image based tagging

Transparency levels represent the interactive state

Maps are replicated for the number of users

Non-Interactive MapInteractive Map

Walls in Museum

Map With some Tags

Regions

Contribution Categorisation through areas on map objects Evaluate the effectiveness of planning through the identification

of regions

Using a region creation tool to identify these regions Image tagged to maps within regions inherit any text for

those regions

Region Creation Tool

Attached Regions

E.g. Greek

E.g. Roman

Image Tags Contribution

Evaluation of the effectiveness of image based tagging on collaborative interfaces

Evaluate how this assists in planning tasks on tabletop interfaces

Assigned by performing a gesture on the image to be attached while placed on top of the map

ImagesMap with Image Tags of 2

different Users

Text Association

Contribution Compare swipe (MyMaps) to tapping (TeamTag) Compare Map-based tagging and TeamTag approach

to tag Images

Attach text by dragging over the back of images and regions

Back of Image

Text Object

Text Attachment

Image Based Tagging Metadata stored in text format within a designated folder

Metadata stored for images and regions: Maps Regions Rotations Locations Text Scale

Each entry of a location in the tag files for each image and region represents a tag on a designated map

##LOCATION<map>: <x coordinate>, <y coordinate>, <z coordinate> <map>: <x coordinate>, <y coordinate>, <z coordinate>…

Image Based Tagging

Tags are represented by special “Pin Images” which are small (at the moment) blue spherical-like images that resemble a pin

Each “Pin Image” relies on a designated rotation entry and location entry

##ROTATION<radian value of rotation><radian value of rotation>…

##LOCATION<map>: <x coordinate>, <y coordinate>, <z coordinate> <map>: <x coordinate>, <y coordinate>, <z coordinate>…

Images that are linked to regions are also indirectly associated to the map that the region is linked to

collaborative image- based tagging and interactive mapping on tabletops presented by: glen whitaker...

Documents

tags tags

text tags

images slide

image images

map slide

mymaps slide

filedatabase tags

images blackhole slide