Designing for Collaborative Cross-modal Interaction

Oussama MetatlaQueen Mary Uni of LondonMile End road, London, UK

oussama@eecs.qmul.ac.uk

Nick Bryan-Kinns,Tony Stockman

Queen Mary Uni of LondonMile End road, London, UK

{nickbk,tonys}@eecs.qmul.ac.uk

Fiore MartinQueen Mary Uni of LondonMile End road, London, UKfiore@eecs.qmul.ac.uk

ABSTRACTWe describe the design of a collaborative cross-modal toolthat supports visually-impaired and sighted coworkers to ac-cess and edit shared diagrams in real time and a case studyof its use in a real world workplace environment. Our find-ings highlight the potential of cross-modal collaboration toimprove workplace inclusion and identify initial challengesand opportunities for cross-modal design.

Categories and Subject DescriptorsH.1.2 [User/Machine Systems]: Human factors; K.4.2[Social Issues]: Assistive technologies for persons with dis-abilities; H.5.3 [Group and Organisation Interfaces]:Collaborative computing, Computer-supported cooperativework.

General TermsDesign, Human Factors.

KeywordsCollaboration, Cross-modal interaction, Accessibility, Inclu-sion, Haptics, Auditory interfaces, Diagrams.

1. INTRODUCTIONCollaboration is a fundamental form of human interaction.

However, software tools which support collaboration assumethat all collaborators have access to the same sets of modal-ities. For example, shared whiteboards assume equal accessto the visual modality [2]. This disadvantages users withdiffering access to sensory channels due to their context andabilities. The lack of access to shared displays can form amajor barrier to workplace collaboration, particularly whenvisual representations are used which typically form a cen-tral part of certain job roles (e.g. System Analysts, ProjectManagers). These situations contribute to the exclusion anddisengagement of visually impaired individuals.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Digital Futures ’11 NewCastle, England, UKCopyright 2011 ACM X-XXXXX-XX-X/XX/XX ...$10.00.

2. CROSS-MODAL INTERACTIONOur brains receive and combine information from different

perceptual modalities to make sense of our environment, forinstance when watching and hearing someone speak. Theprocess of coordinating information received through mul-tiple perceptual modalities is fundamental and is known ascross-modal interaction [3]. Cross-modality is particularlyrelevant to individuals with perceptual impairments whorely on sensory substitution to interpret information usingalternative modalities. In the design of interactive systems,the phrase cross-modal interaction has also been used to re-fer to situations where individuals interact with each otherwhile accessing the same shared space through different setsof modalities (e.g. [6, 1]). Current technological develop-ments mean that it is increasingly feasible to support cross-modal input and output in a range of devices and environ-ments, yet there are no practical examples of such systems,e.g. Apple’s iPhone provides touch, visual, and speech inter-action, but there is no coherent way of collaborating usingdiffering sets of modalities if collaborators cannot see theshared screen. This paper outlines a project that addressesthis problem.

3. WORKSHOPTo address the issues identified above we gathered require-

ments and feedback from project partners and potentialusers to inform an ongoing development process. We rana workshop to engage with representatives from end usergroups in order to encourage discussion and sharing of ex-periences with using diagrams in the workplace. Eight par-ticipants attended the workshop including participants fromBT and the Royal Bank of Scotland and representatives fromthe British Computer Association of the Blind and the RoyalNational Institute for the Blind. Activities ranged fromround table discussions exploring how participants encoun-tered diagrams in their workplaces, to hands-on demonstra-tions of early audio and haptic prototype diagramming sys-tems. The discussions highlighted the diversity of diagramsencountered by the participants; from nodes-and-links de-sign diagrams for databases and networks, to business modeldiagrams, organisation and flow charts and field specific vi-sualisations. Additionally, participants discussed the variousmeans they currently use for accessing diagrams and theirlimitations. Approaches included the use of a human reader,swell paper, transcriptions and stationary-based diagrams,all of which share two main limitations; the inability to cre-ate and edit diagrams autonomously, and inefficiency of usein collaborative interaction.

Figure 1: The collaborative cross-modal tool in use.

4. PROTOTYPE TOOL DEVELOPMENTOur prototype cross-modal tool supports autonomous non-

visual editing of diagrams as well as real-time collaboration.It combines three main views of any given diagram; a vi-sual, auditory and a haptic view. The auditory view uses ahierarchical metaphor to capture and structure diagram el-ements. Users can browse the tree structure, similar to typ-ical file explorers, and receive auditory feedback in responseto their interaction. This approach is thoroughly describedin [4] and [5]. We use the PHANTOM Omni Haptic deviceto support interaction with the haptic view, which capturesthe content of a diagram and its spatial arrangements anddisplays them on a virtual vertical plane to match the vi-sual display on a computer screen. We use a number ofhaptic effects to support the non-visual interaction in thisview. Namely, magnetic wells and lines to represent thenodes and edges of the diagrams respectively. Users manip-ulate the stylus of the device (shown on Figure 1) and areattracted to diagram elements through a simulated mag-netic field force, andcan then trace the edges of the diagramthrough the magnetic lines. Encountered items are also pre-sented in audio using a speech and non-speech sounds.

5. FEEDBACK FROM EXHIBITIONWe engaged with potential users over three days at the

recent QAC Sight Village event which is the premier tech-nology exhibition for blind and partially-sighted people. Theevent allowed us to showcase a functional prototype of thecross-modal collaborative tool, to gather feedback about ourinitial design decisions and to identify avenues for improve-ments. Among the identified issues were the need for addingcontextual features to aid orientation within the haptic viewand the variance in users’ preference of the vertical orienta-tion of the haptic view and the levels of magnetic attractionforces used.

6. CASE STUDIES OF USE IN THE WILDWe are currently running tests to assess how well the tool

supports real-world collaborative scenarios. So far, we havedeployed the tool in the head office of a London-based Chil-dren and Families Department in local government and ob-served its use by sighted and visually-impaired coworkers.Our initial observations shows that collaborations typicallyinvolve three phases of interaction which could portray dif-ferent stages on the learning curve of a collaborative cross-modal system; 1) A first phase consisting of the visually-impaired user dominating the interaction. This includes

exploring the diagram, editing its content and altering itsspatial arrangements. The sighted coworker in this phasetypically engages in discussion about the diagram, in addi-tion to providing general guidance about where things arelocated and how to get to them; 2) In a second phase, thevisually-impaired user continues to dominate the interac-tion, while the sight coworker engages in refining the spa-tial arrangements produced by their coworker; 3) In a thirdphase, both coworkers engage in manipulating the diagram,working independently on different parts of its content whilecontinuing to discuss the task and updating each other abouttheir progress. One major issue that affected how coworkersexchanged information, including guidance and feedback, isthe lack of representation of partner’s actions on their re-spective views of the system. This meant that the usersfound it hard to keep track of their partners moment-to-moment progress and could only provide feedback once part-ner’s actions were completed. An interesting point to high-light is that, even when working independently, coworkerswould focus on the same task when this is a spatial one,i.e. whenever the visually-impaired user turns to the hapticview. It could be that having a somewhat similar represen-tation (2D visual diagram and haptic diagram) provided thepair with a common perspective of the task, more so thanwhen using the audio tree view.We will be exploring suchissues through further workplace studies.

7. CONCLUSIONSOur work has so far highlighted the difficulty in design-

ing for cross-modal collaboration and has raised interestingquestions about issues such as coherence of representationsand support for awareness across modalities. Moreover, ourinitial findings indicate that providing cross-modal mecha-nisms for all team members to collaborate, share, and editdiagrams has the potential to increase productivity and tosignificantly improve the working lives and inclusion of per-ceptually impaired workers.

8. REFERENCES[1] N. Bryan-Kinns, O. Metatla, and T. Stockman.

Collaborative cross-modal interfaces. In Proc. of DigitalFutures’10 RCUK Digital Economy All Hands Meeting.

[2] M. Cherubini, G. Venolia, R. DeLine, and A. J. Ko.Let’s go to the whiteboard: how and why softwaredevelopers use drawings. In Proc. of the SIGCHI conf.on Human factors in computing systems, pages557–566. ACM, 2007.

[3] J. Driver and C. Spence. Attention and the crossmodalconstruction of space. Trends in Cognitive Sciences,2(7):254 – 262, 1998.

[4] O. Metatla, N. Bryan-Kinns, and T. Stockman.Constructing relational diagrams in audio: the multipleperspective hierarchical approach. In Proc. of the 10thACM SIGACCESS conf. on Computers andaccessibility, pages 97–104. ACM, 2008.

[5] O. Metatla, N. Bryan-Kinns, and T. Stockman.Interactive hierarchy-based auditory displays foraccessing and manipulating relational diagrams.Journal on Multimodal User Interfaces, in press.

[6] F. Winberg. Supporting cross-modal collaboration:Adding a social dimension to accessibility. Haptic andAudio Interaction Design, pages 102–110, 2006.