Aphasia and Technology: The GReAT Project

Abi Roper and Jane Marshall

On behalf of the GReAT Project TeamDivision of Language and Communication Science

Department of Human Computer Interaction DesignCity University London

Presentation Outline•The Project

•Designing and Refining a Computer Gesture Therapy - Gest

•Gest Demonstration

•Delivering a Computer Therapy

•Gest Pilot Study

•Preliminary Outcomes

Project Aims

• To develop an affordable, computer-based technology that can be used in therapy at home to help people with severe aphasia to gesture.

• To establish how to design effective/engaging interactions for people with aphasia.

• To evaluate the efficacy of the technology within a pilot therapy study

Project Structure

• Phase 1: Designing a prototype gesture therapy using participatory design methods.

• Phase 2: Testing and piloting the prototype

Project TeamHuman Computer Interaction Design & Language and Communication Science

Stephanie Wilson Sam Muscroft Julia Galliers Jane Marshall

Naomi Cocks Tim Pring Abi Roper

Phase 1

• Designing a prototype gesture therapy using participatory design methods.

Consultants

Justine Everson Gerald Hartup Carol Watson

Philip Pepper Emma Buswell

Consultants

• Role: to test and feedback about relevant technology.

• Person Specifications: – Expressive aphasia language difficulties. – Able to attend university once or twice a month for

participatory design sessions.

• Recruited through in house clinic and through links with the Stroke Association Communication Support

Co-ordinators.

• Employed by City University London as Casual Staff members.

Methods: Participatory Design Sessions

• Participatory design – engaging end users in design process

• Sessions explored offline gesture therapy, computer gesture recognition, interaction within 3D worlds and computer interfaces.

• Consultants took part in 9 sessions each

• Project team involved in each session- 1 HCID Researcher- 1 HCID Developer- 1 Speech and Language Therapist Researcher- 2 or 3 Consultants

Session Structure

1. Introduction to scheduled activities

2. Round table gesture activity

3. Demonstration of Technology

4. Trial use of technology by one consultant - followed by interview at computer

5. Tea break

6. Trial use of technology by remaining consultant(s)

Participatory Design

• Watch a video excerpt of the design process in action on the next slide

What did we learn from the Sessions?

1. Consistency2. Simplicity3. Pace4. Reliability5. Rewards6. Individual Differences7. Potential of ‘gaming’.

The Prototype

OK

← →

Demonstration

•Watch a video Demo of the Gest prototype in action on the next slide

Using the Therapy at home

•How does this work at home?Key differences between lab and home – User practising independently, User intending to practise daily. User practising in non-lab conditions.

Things to consider when setting up

•Lighting conditions

•Safety and permanence (negotiate!)

•User comfort and access

Things to consider when training

• Develop the user’s confidence in the system. (Be confident yourself)

Demonstrate:

1. Allow user to observe entirely2. Allow user to observe and operate

interaction buttons3. Allow user to operate alone but with

support as needed (confidence)

Things to consider when training

•Reinforce how to switch the computer on and off several times.

•Make an appointment to come back in one week to review.

•At review appointment, observe and re-train difficult procedures.

Phase 2

• The Pilot Study

Questions• Will practice with Gest improve participants’ production of

gestures &/or spoken words?• Will improvements be specific to items that feature in the

programme?• Will gains occur when Gest is used without ongoing therapist

support?• Will gains be maintained after Gest is withdrawn?• What are participants’ views about Gest?• What are carers’ views about Gest? (where relevant)• Is Gest easy and enjoyable to use?

Participants

• 10 people with severe aphasia– Consent to take part– Fluent pre-stroke users of English– Naming score <20% – Able to recognise pictures– No known dementia or other cognitive impairment

Consent

Screening

Tests (1)

3 Weeks Practice

Tests (2)

Phase 1 with weekly visits from therapist

3 Weeks Practice

Phase 2 with no weekly visits from therapist

Tests (3)

3 weeks no tool

Tests (4)Total time commitment: about 14 weeks

Practice Phases

• Each last 3 weeks• Each practise 15 gestures with the tool• Phase 1: Weekly visits from therapist• Phase 2: Initial but no weekly visits

Tests

• 60 items– Gesture from picture– Name from picture What is the

name of this?

How would you gesture this?Items:

30 practised with Gest

15 familiarised only

15 controls

Scoring Gestures

• Gesture tests are filmed• 4 Scoring videos created• Each video contains 60 gestures in random

order:– 15 from test 1– 15 from test 2– 15 from test 3– 15 from test 4

• Scores– Recognition Score– Rating Score

• Scorers are ‘blind’ to the time of assessment

Scoring Gestures

Usability Evaluations

• Observe participants using the tool• Interview participants• Interview carers (if relevant)

– When technology is installed– After each practice phase

Usage Logs

• Record– Number of sessions– Length of sessions– Levels of programme accessed– Number of gestures recognised

Results so far

Usage

Mean Usage: 7 Participants

0

10

20

30

40

50

60

Daysavailable

No ofsessions

Time used(hrs)

Time persession(mins)

Individual Usage: 3 participants

0

10

20

30

40

50

60

70

80

Days available No ofsessions

Time used(hrs)

Time persession (mins)

Usage x Recognition

Mean Usage over Phases

Usage: Levels

• Three participants use level 1 more than 2 & 3• Two participants use all 3 levels and rate them

equally highly• Two participants rate levels 2 & 3 more highly than 1

• Possibly contingent on navigation abilities

Usage Observations: Challenges

• Set up– Lighting– Positioning (e.g. wheelchairs)– Security

• Glove– Putting glove on the wrong hand– Using the peg board (although often not

necessary)

Usage Observations: Challenges

• Starting and stopping– Pressing key board buttons before menu has

appeared– Not always pressing ‘off’ at end of session

Usage Observations: Challenges

• Navigation– Variable use of OK, forward, back & menu buttons– Variable navigation between levels– Some unprincipled button pushing

Speed and competence may relate to prior computer usage

Usage Observations: Challenges

• Gesture production– Knowing when to gesture; waiting for 321 ping– Knowing when the gesture has been recognised– Variable use of cues; e.g. some adjust handshape

in response to glove image others do not

Usage Observations: Enjoyment

• All signal high enjoyment levels– Thumbs up sign– Drawn smiley face

• Positive reactions to level 2– Game format– Narrative context – Environments

• Positive reactions to level 3– Humour (spider, dentures)– Stroke survivors as actors– Presence of children

Usage Observations: Enjoyment

Other Observations

• Some target spoken words produced during Gest use

• Spontaneous uses of practised gestures (‘umbrella’ gestured when participant noticed that it was raining outside; ‘child’ gesture when talking about grandchild)

Results so far

‘Carer’ Comments

Independence of Use

• ‘She uses it all on her own, I don’t know how to operate it’

• The first session I stayed with L, after that I’ve helped only if she’s found something particularly frustrating’

• All comment that the participant initiated use of Gest

Enjoyment

• All say that the participant enjoyed Gest

• ‘he likes it when they clapped’

• ‘some of the gestures are particularly fitting and she enjoyed rainbow’

Views about Technology

• ‘I was a technophobe and when they said ‘computer’ I thought it was going to cause problems. I thought I wouldn’t understand and he wouldn’t understand it. But it’s so ‘easy’

Reservations

• Carry over to real life (1 carer):

• ‘while she works on it here (points to computer) it doesn’t necessarily translate’

• She wanted a hankie last night and didn’t make a gesture’

Conclusions

• Gest was created through participative design involving people with aphasia– It offers 6 packages of hierarchical practice on 30 gestures– It is accessible even to people with severe strokes– It can be used successfully in diverse home settings– It allows for flexible, self directed practice and is typically

intensively used– It is enjoyable to use, with no reports of increased ‘carer

burden’

Conclusions

• But we do not know if– Gest improves gesture production– Gest improves spoken naming– Effects generalise to unpractised targets– Effects are maintained

• The results of the pilot study will give us answers to these questions

Acknowledgements

The Research Councils UK Digital Economy Programme

The Stroke Association

Consultants and their families

Participants and their families

Thank YouGReAT@city.ac.uk

www.soi.city.ac.uk/great