Designing ‘Robo-Buddies': Technologies to Mediate Communication in the Wild

Julie A. Hengst, Laura S. DeThorne, Hillary Valentino &

Maeve McCartin

University of IllinoisChampaign, IL, USAOral Session 25

1430 – 1530 Platinum Lounge

“Robo-Buddies” Project• Participatory Design Project Team

– Engineers, computer scientists, communication specialists, participant-users, disability specialists

• Collaborating to develop communication technologies that function as pseudo-intelligent mediators of interactions – to improve communication between diverse communicators ( e.g.,

students on campus with and without disabilities)– by blending strengths of human mediators with features of AAC devices.

Augmentative & Alternative Communication (AAC)

www.hawking.org.uk/images.html

• Forms of communication other than oral speech • e.g., facial expressions, gestures, symbols,

pictures, or writing;• people with communication disorders may rely

on AAC

• AAC Devices • specialized assistive technologies • used by people to supplement or replace

impaired speech abilities. • e.g., communication boards, dedicated

computer systems

http://www.asha.org/public/speech/disorders/AAC.htm

http://uk.dynavoxtech.com/success/cerebral-palsy/details.aspx?id=92

Activity-based Approach to AAC Prosthetic Approach• Grounded in transmission models

of communication– Design technology to replace or

augment impaired body part or function

– For individual user– To produce linguistic messages

• Proven, but limited, approach for…– speech production in pre-planned,

predictable, controlled activities

Activity Approach• Grounded in CHAT theories

– Design technology to mediate interactions and support activities

– For functional system– To support interactional

alignments in routine activities

• Novel approach aimed at…– supporting interaction in dynamic,

emergent, less-controlled activities

Collecting Interactional Data

Participants• 13 primary participants (7 AT users with disabilities; 6 comparisons)• 7 secondary participants; 51 incidental participants

Data Collection—42 video recorded sessions• 13 semi-structured interviews (one with each primary participant)• 24 On-campus observations (e.g., scavenger hunts; working with PA in

dorm; navigating campus) • 5 lab trials (e.g., working with trial technologies)

Analyzing Interactional Patterns

• 17 on-campus observations– 4 with “David”; 3 w/ “Artemesia” & “Iris”; 2

w/ “Chip”, “Jester” & “Jesse”; 1 w/ “Izzy”.

• 11 hrs 20 mins, video data.

• Analysis:– Transcribing all sessions. – Coding all sessions for interactional patterns.– Completing situated analysis of selected

interactions.

“Artemesia”

“Iris”

“David”

Coding Interactional Patterns• Conversational Features:

– Conversational trouble source (TS) disruptions in conversational flow, may or may not lead to a breakdown.

– Conversational repetition, or reformulation (R) marked repetition, reported speech, and replays, within or across modalities .

• Interactional Discourse Resources: – Humor, or playful episodes (PE) utterances sharing a common playful theme.– Conversational narratives (CN) Verbal or nonverbal telling of event(s) displaced from

the moment of telling, plus evaluation. – Procedural discourse (PD) telling how to do something (e.g., giving directions).

• Object Specific Codes:– Contextual orchestration with objects (CO) objects within activity.– Fidgets (F), fidgeting with objects, unrelated to communicative activity.

Coding Summary: Average #/10mins of observation by participant. (Note: Total counts across participants in grey.)

Participant et al.

Time (mins)

TS R PE CN PD CO F

Artemesia 115 2.5 30 4.6 2.3 1.6 5.3 0.6

Iris 112 1.9 42.8 6.9 5.4 2.0 5.1 0.8David 152 3.7 31.6 1.1 0.5 2.2 9.5 0.1Chip 93 2.7 69.8 6.3 3.5 1.1 10.7 2.4

Jester 34 3.2 49.1 2.4 3.2 4.7 13.5 0Jessie 124 2.7 36.6 0.5 3.7 1.6 3.9 0.4Izzy 50 1.0 25.4 3.0 3.0 1.4 3.8 0.2

                 Total

Counts680 181 2701 235 200 126 475 45

Interactional Profiles

Average of All (60.0*): R>CO>PE>CN>TS>PD>F

Chip (96.5): R, CO, PE, CN, TS, F, PDJester (76.1): R, CO, PD, CN, TS, PE, FIris (64.9): R, PE, CN, CO, PD, TS, FJessie (49.4): R, CO, CN, TS, PD, PE, FDavid (48.7): R, CO, TS, PD, PE, CN, FArtem. (46.9): R, CO, PE, TS, CN, PD, FIzzy (37.8): R, CO, CN, PE, PD, TS, F

• Listed in order from most to least.

• Overall frequency of codes varies (e.g., 96.5-37.8 /10 min).

• Frequency of specific codes varies (e.g., PD varied from 3rd - 7th) *#codes/10 minutes of observation

Interactional Profiles

Average of All (60.0*): R>CO>PE>CN>TS>PD>F

Chip (96.5): R, CO, PE, CN, TS, F, PDJester (76.1): R, CO, PD, CN, TS, PE, FIris (64.9): R, PE, CN, CO, PD, TS, FJessie (49.4): R, CO, CN, TS, PD, PE, FDavid (48.7): R, CO, TS, PD, PE, CN, FArtem. (46.9): R, CO, PE, TS, CN, PD, FIzzy (37.8): R, CO, CN, PE, PD, TS, F

• Data Excerpts: • Artemesia & Jessie• Service Encounters,

with PA’s included

• Examples of: • Repetition• Contextual use of

objects• Trouble source• Mediational roles of

the PAs

“Artemesia”

• 38-year-old history major;– Employs multiple PA’s to assist

24hrs/day with physical needs due to CP

• 2 on-campus observations – Dining hall with a PA; – Outside dining all, with a PA

Profile: R, CO, PE, TS, CN, PD, F

Dining Hall: “selecting meal items with PA”

Dining Hall: “PA interpreting story”

Service encounter in dining hall with familiar partner.

Repetition for confirming.

Artemesia directing PA to selecting items; PA questions some choices.

“Jessie”• 30-yr-old, doctoral student in

Community Health & Disability Studies;– Employs 6 PA’s to assist with

physical needs due to CP

• 2 on-campus observations – Student Union bowling alley; – w/ his PA, Tech Store in Union

Profile: R, CO, CN, TS, PD, PE, F

At SHS Building: “story of first meeting”

At Tech Store: “clickers”

Service encounter, shopping in store with unfamiliar clerk.

Repetition helps to resolve trouble source.

PA is co-shopper.

CHAT Approaches to AAC

• Shift in unit of analysis– from “replacing” individual abilities to mediating activities of functional

systems

• Interactional data and profiles – empirically grounded discussions– distributed, flexible, and persistent nature of functional systems

• Imagining mediational roles for technologies– connecting people to histories and broader contexts– reformulating words/actions in the moment– layering ongoing activities with interpretive frames

Acknowledgements Grant Support:

M. Hasegawa-Johnson (PI), J.A. Hengst, L.S. DeThorne, T. Huang, P. Malik, and T. Gunderson. Pseud-intelligent mediators ("Robo-Buddies") to improve communication between students with and students without disabilities. UIUC IN3 Grant. Funded August 2012 - August 2014; Amount: $199,898.

Speech and Hearing ScienceSara Small, Katie Lester, Maeve McCartin, Hillary Valentino, Monique Kammo, Shanthi Sivasankaran, Carissa Ernat, Caroline Eichelberger, Gena Carpenter, Mariana Aparicio, Suma Devanga, Christina Bronson-Lowe,

Human Computer Interface Karrie Karahalios, Jennifer Kim, Ha Kyung Kong

Disability Resource Educational Services

Electrical & Computer EngineeringXuesong Yang, Yang Zhang, Dennis Lin, Pooya Khorrami

Research Team and Lab Assistants:

Participants: A total of 71 individuals as primary, secondary, or incidental participants.

References