Multimodal InteractionsThe what, the why and the how

Diego Paez, Despegar

About Me

Diego Paez => DEKΔ

G E U Tco-founder

JS developer with a passion for HCI ♥�

Currently working at:

AGENDA

➔ Definitions

➔ Foundations

➔ An alternative approach

➔ Future work

What is a multimodal interaction (MMI)?

#The what

Human ComputerInteraction

Feedback&

Senses

5 major senses: sight, hearing, touch, taste and smell. But there are more; eg: Proprioception.

You Real World

Interaction

Highway 401 by Nayuki

Senses

But, why?

#The why

Why?

➔ Lowering barriers to HCI. Eg: gestures research.

➔ Accessibility [1]➔ Killer feature➔ IoT

Foundations

#The how

Architecture proposed

by Dumas et al [2]

1

2

3

An approach

#The how

Plusultra Gyes

Proposed Architecture

Modalities(HW, distributed)

Pub/sub pattern used to separate

layers

Web Clients (distributed)

1

2

3

Proposed Architecture

Getting started

npm install plusultraDocker-compose up

mkdir myMultimodalDeviceDrivercd myMultimodalDeviceDrivernpm install gyes

Creating a MM Device Drivervar Inherits = require( 'inherits' );var ModalityChannel = require( 'gyes' ).ModalityDriver;

module.exports = MyModality;

function MyModality( opts ){// constructor code...

}

Inherits(MyModality, ModalityChannel);

MyModality.prototype.fission = function( time ){ … };

MyModality.prototype.newEvent = function( data ){ … };

Inside the app// lets create a new plusultra clientplusultraClient = new gyes( app_key, socketURI, options );plusultraClient.authenticate( app_key );

// create a new modality (this is like metadata)var myMod = new gyes.Modality( modName, 'input', {} );

// instance a driver for the modalityvar modDriver = new myMultimodalDeviceDriver( options );myMod.use( modDriver );

// finally add the modality into the systemplusultraClient.addModality( app_key, airMod );

Future Workaka: Help is welcome!

➔ Experimenting with WebRTC to enable P2P interactions.

➔ Define Interpretations as components. W3C MMI. React can

help?

➔ Improve API. Follow a reactive/functional approach. Cycle/rxjs?

Future Work

Project repos➔ Plusultra: https://github.com/dpaez/plusultra

➔ Gyes: https://github.com/dpaez/gyes

➔ Demo app (shapes): https://github.com/dpaez/shapes_app

➔ Gitter: https://gitter.im/gyes_mmi/Lobby

Thanks!Who is the hipster?Follow me @caraxGithub @dpaezGet the GEUT experience: geutstudio.com / Some cool code @geut