the 13th international scientific conference elearning and ...existing solutions proposed solution...

AN ADAPTIVE MULTIMODAL INTERFACE

TO IMPROVE ELDERLY PEOPLE'S REHABILIATION EXERCISES

Drd. Ing. Imad Alex Awada, Drd. Ing. Andra Codreanu,

Prof. Dr. Ing. Irina Mocanu, Prof. Dr. Ing. Adina Magda Florea,

Prof. Dr. Mihaela Apostu

The 13th International Scientific ConferenceeLearning and Software for Education

Bucharest, April 27-28, 2017

An Adaptive Multimodal Interface to Improve Elderly People’s Rehabilitation Exercises

Table of contents

• Introduction

• Existing Solutions

• Proposed Solution

• Evaluation

• Conclusion and Future Work

Context

IntroductionExisting Solutions

Proposed SolutionEvaluation

Conclusion and Future Work

Goals

3

1

• Assist elderly people recovering from physical affections:

• Adapt the multimodal interface according to the needs

• Monitor the performances and health condition

• Adapt the exercises difficulty levels by learning from user

feedback

• Increase:

• the usability

• the accessibility

• the efficiency of the system

Context




System components

426/06/2015

Research Report Presentation

4. Adaptation

The serious game change the

exercise levels depending on users

physical condition and results

The multimodal interface adapts

depending on user habits

2

1. Medical

Contain a set of recommended

exercises prescribed by a therapist

2. Serious Game

Tracks the user rehabilitation

exercises evolution

Inform the user about performances

or mistakes

3. Multimodal Interface

Interact with the user through the

game and give feedback like

sound or screen messages


526/06/2015

Table of contents

• Introduction


• Multimodal Interfaces

• Serious Games


• Evaluation


IntroductionExisting SolutionsProposed Solution

EvaluationConclusion and Future Work

Existing multimodal interfaces

Multimodal InterfacesSerious Games

3

Multimodal interface Interface type Input Output

“AT&T’s MATCHKiosk”[3]

Multimodalinteractive cityguide

Speech, pen andtouch screen

Speechsynchronized withgraphics and texts

“Canesta” [4] Virtual projectedkeyboard

User’s fingermovement on theprojected surface

Projectedkeyboard and thedisplayed text onscreen

“Put That There” [5] Virtual projectedwall sized screen

Speech andpointing gesture

Speech andgraphics on screen

Gaze detectionpointing system forpeople withdisabilities [6]

Computer screen Speech andmovements of thehead

Visual

Inputs and outputs of multimodal interfaces

IntroductionExisting SolutionsProposed Solution

EvaluationConclusion and Future Work

Existing serious games

Multimodal InterfacesSerious Games

4

Serious Game Name Rehabilitation type

Input Output

“Voracy Fish” [8] Functional(upper limb)

gesture detectedby a Kinect device

Avatar displayedon a screen

“Hammer & Planks”[9]

Hemiplegic(musclesaffection)

Kinect,Wii Board,Joystik

Avatar displayedon a Smart TV

“Jewel Mine” [10] Therapeutic(physicalrestoration)

gesture detectedby a Kinect device

Graphics displayedon a screen

“Bombs&Flowers”[11]“Can” [11]

Functional(upper body)

Kinect,LeapMotion,Sifteo Cubes

Visual feedbacks,instructions andscore displayed onthe screens

Inputs and outputs of serious games


826/06/2015

Table of contents

• Introduction



• Framework Overview

• Workflow

• Preview

• eLearning Model

• Evaluation





9

Application communication framework overview

Framework overviewWorkflowPrevieweLearning Model

5

Main Pages of the Interface (Romanian Version)




10


6




11

Multimodal Interface Preview


7




12



8




13



8




14

Model Adaptation


9




15

Game Preview


10

Screenshot of the game with two avatars


1626/06/2015

Table of contents

• Introduction



• Evaluation



Proposed solutionEvaluation


Speech recognition test results

11

English UtteranceAv. Lev. Score

Romanian UtteranceAv. Lev. Score

I have started exercising 81.20 Am început exercițiul 80.15

When did I do my last

exerciseMake the font size biggerPause the exercise

83.8682.3498.13

Când am făcut ultimul exercițiuMăriți dimensiunea scrisului Pauză

85.1278.16

100.00

Resume the exercise 97.25 Continuă exercițiul 95.43


1826/06/2015

Table of contents

• Introduction



• Evaluation


ConclusionFuture Work

IntroductionState Of The Art

Proposed ApproachConclusion and Future Work

Conclusions

The system:

• integrates speech and gesture as input modalities

• generates different visual and phonetic outputs

• user profile and adaptation model

• multilingual (English and Romanian)

• compatible with all the operating system

• fully responsive (adapt to any display size and orientation)

• rehabilitation shoulder exercise “Standing flexion stick”

• feedback eLearning model

• users have used the system for longer periods and achieved

effective results in an easier manner than previously

12

ConclusionFuture Work

IntroductionState Of The Art

Proposed ApproachConclusion and Future Work

Future work

• improve the multimodal interface by integrating new modes of

interactions and improving the existing ones

• improve the presented game approach by the meaning of adding

new exercises to the system

• evaluate the interface and the exercises with elderlies and people

recovering from specific diseases, in a supervised environment

• integrate the interface into an ambient assisted living system for

elderlies (health monitoring, fall detection, telepresence, supervised

physical exercises and activity planners [12])

13

Bibliography

2123/06/2015


[1] Friedemann Mattern, et. al., “From the Internet of Computers to the Internet of Things”, From

Active Data Management to Event-based Systems and more”, pp 242-259, Berlin 2010.

[2] Élvio Rodrigues, et. al., “Developing a Multimodal Interface for the Elderly”, Procedia Computer

Science, Volume 27, pp. 359-368, 2014.

[3] Sanya Harneja, et. al., “Human Computer Interaction: Smarter way of Communication”,

International Journal of Engineering and Technical Research, Volume 2, pp. 201-205, April 2014.

[4] Kuntal Saroha, et. al., “Human Computer Interaction: An intellectual approach”, International

Journal of Computer Science and Management Studies, Volume 11, pp 147-154, 2011.

[5] Chang-Beom Park, et. al., “Real-time 3D pointing gesture recognition for mobile robots with

cascade HMM and particle filter”, Image and Vision Computing, Volume 29, pp. 51-63, January 2011.

[6] Pradipta Biswas, et. al., “A new input system for disabled users involving eye gaze tracker and

scanning interface”, Journal of Assistive Technologies, Volume 5, pp. 58 – 66, 2011.

[7] E.Brox, L.Luque, et. al., “Exergames for elderly: Social exergames to persuade seniors to

increase physical activity”, Pervasive Computing Technologies for Healthcare, pp. 546–549, 2011.

[8] Voracy Fish Game website: www.voracy.com | accessed on: 2017-02-11.

[9] Ines Di Loreto, et. al., “Game Design for All: The Example of Hammer and Planks”, Serious

Games Development and Applications, pp. 70-75, Norway 2013.

[10] Jewel Mine website: http://ict.usc.edu/prototypes/jewel-mine | accessed on: 2017-02-11.

[11] Gabrielli S., et. al., “A Game-based solution for In-Home Rehabilitation”, Proceedings of the 2nd

International Congress on Neurotechnology, Electronics & Informatics, pp. 145-149, 2014.

[12] CAMI Project - Official Website, www.camiproject.eu/info | accessed on: 2017-02-11.

2223/06/2015


Thank you for your

attention!

Questions?

the 13th international scientific conference elearning and ...existing solutions proposed solution...

Documents