multimodal interaction in distributed and ubiquitous computing - iciw 2010

Titol de la presentació

Cognom Nom, CàrrecNom de la jornada, lloc

Multimodal Interaction in Distributed and Ubiquitous Computing

Marc Pous and Luigi Ceccaronibdigital

ICIW 2010 - The Fifth International Conference on Internet and Web Applications and Services

May 9 - 15, 2010 - Barcelona, Spain

• Why multimodality?

• Multimodal distributed services deployment

• User tests and conclusions

SUMMARY

In our cities, urban services practically have not changed in a century

1: Location-based services

MOTIVATIONS

http://www.flickr.com/photos/nnova/4512346900/

http://picasaweb.google.com/marc.pous/Xina2007#5102695479944027890

http://picasaweb.google.com/marcpous.nyc2009/NYC2009#5415374502175450370

http://picasaweb.google.com/marcpous.nyc2009/NYC2009#5415371534810675298

http://www.flickr.com/photos/loquat73/3385335980/

2: Interactions and multimodality

MOTIVATIONS

http://www.flickr.com/photos/nnova/4423756547/in/set-72157623598200316/

http://www.flickr.com/photos/nnova/4424521014/in/set-72157623598200316/

3: Accessibility

MOTIVATIONS

http://www.dynamiclanguageblog.com/2009/12/technology-grows-for-hard-of-hearing.html

MOTIVATIONS

What is an ICD?(Interactive Community Display)

INREDIS

3G = ACCESSIBILITY + UBIQUITY + INTEROPERABILITY

http://www.inredis.es

INREDIS ICD

Management of distributed services that offer the capability of processing and synthesizing multiple modalities of interaction

DISTRIBUTED INTERACTION SERVICES

• Voice services– Voice recognition

• Via microphone, real-time streaming converting the voice signal into text

– Voice synthesis

• Sign language service– Spanish sign language recognition

• Via webcam, real-time streaming converting the image into text

• Emotion service– Emotion recognition (voice + image)

• Via webcam and microphone, real-time streaming

• Avatar service– Emotional avatar– Sign-language speaker avatar

DISTRIBUTED SERVICES

SIGN LANGUAGE SERVICES

DISTRIBUTED SERVICES

VOICE SERVICES

How to consume multimodal distributed services offered by an ICD?

W3C MMI Architecture

• W3C MMI Architecture

http://www.w3.org/TR/mmi-arch/

W3C MMI Framework

http://www.w3.org/TR/mmi-framework/

Proposed multimodal interaction framework

• ICD Architecture

ICD ARCHITECTURE

(1) Interaction identification

(2) Main page- Map or avatar- Command buttons- Content- Scenario options

(3) Contextualized map with services

(4) Avatar with map and Street View service

ICD DESIGN and INTERFACES

USER–CENTERED DESIGN

• User profiles (Personas)– Blind users– Deaf users

• Unable to read– People living or visiting a city

• Scenarios– Context-aware informative scenario– Emergency scenario

IMPLEMENTATION

USABILITY TESTING

• PLI (People Led Innovation) methodology

• Preliminar usability tests with a limited number of users

• But the ICD implementation received good feedback!

CONCLUSIONS and FUTURE WORK

• Implementation based on the W3C MMI Architecture idea• Platform able to integrate multimodal interactive distributed

services and offer interaction in real-time• Web-based applications and device-independent

implementation• Improvement of the user experience and accessibility of

people with special needs

What are we working on now?– Real-time interaction: delay reduction between interactions– Browser-based apps not being able to easily access device hardware– Synchronous interaction vs. Asynchronous technologies– Improvement of the distributed services orchestration– Enhancement of the mobile usability

MOBILE PHONES?

http://www.slideshare.net/nokiaconversations/younghee-jung-and-jan-chipchase-presentation-for-nokia-design-london-studio-event-london-29th-april-2008

Titol de la presentació

Cognom Nom, CàrrecNom de la jornada, lloc

Thank you!

Moltes gràcies!

¡Muchas gracias!

Marc Pous and Luigi Ceccaronimpous@bdigital.org and lceccaroni@bdigital.org