15-nov-2002mumin workshop1 multimodal research at uiah kristiina jokinen medialab university of art...

15-Nov-2002 MUMIN Workshop 1

Multimodal research at UIAH

Kristiina Jokinen

MediaLab

University of Art and Design [email protected]


University of Art and Design

Education and research in the fields of design, new media, audiovisual communication, art education, arts

Largest art school in Scandinavia Nearly 1600 students, 15% from abroad Lume, the Finnish centre for media research

and development, was opened in 2000• Departments of film and television, design for theatre,

film and television, and new media


Media Lab

Formed in 1993 Explore, discover and comprehend the new digital

technology • Impact in society

• Possibilities for communication, interaction and expression

• Challenges to new media and information design 2-year masters programme

• MA in New Media (full time)

• New Media program for professionals (flexible study method) 20 full-time, 15 professionals, and approx. 30 minor

subject students annually to the MA programmes


Adaptive Systems for Complex Interaction

Research on natural interaction between humans and computers• Interaction strategies, cooperative response planning

• Various input modalities

• Concepts, models Apply machine-learning techniques to dialogue

processing • Compare and test applicability of the techniques

Support Design-for-all principles in designing intelligent interfaces


Human-computer interaction

Computer as a tool• Passive and transparent

• Supports the human goals, human control

Computer as an agent• Models of beliefs, desires, intentions (BDI)

• Intelligent software mediating between the human and an application

• Cooperation, negotiation

• Complex interaction

• Multimodal communication


Projects

USIX-Interact: Natural Interaction and Adaptive Methods http://www.mlab.uiah.fi/interact/

DUMAS: Dynamic User Modelling for Adaptive Speech Interfaces http://www.sics.se/dumas/

MUMMI: Multi-Modal Museum Interfaces (Study project together with Marjo Mäenpää and Antti Raike, Design for All, Virtual Art Exhibition) http://mlab.uiah.fi/mummi/


Natural interaction Language that suits to computers Language that humans use to

communicate

=> Language that humans and computers use when interacting with each other

- Different ? How?

- How it emerges from interaction?


Interact: key aspects for adaptivity Conversational ability

• Dialogue modelling and natural communication Learning systems

• Various methods and techniques

• Various interface techniques (speech, text, map) Language technology

• Finnish and multilingual Agent-based architecture

• Jaspis development platform http://www.mlab.uiah.fi/interact/


Interact Partners

University of Art and Design, Media Lab

University of Helsinki, Language Technology

University of Tampere, TAUCHI unit

Helsinki University of Technology, Neural Networks Research Centre

Fujitsu Invia oyj Tecnomen oyj Lingsoft oy The Arla Institute Finnish Association

for the Deaf Finnish Technology

Agency


DUMAS - Dynamic Universal Mobility for Adaptive Speech interfaces

EU 5th framework R&D project• Swedish Institute of Computer

Science

• UIAH, Media Lab

• University of Tampere, TAUCHI-unit

• UMIST, Manchester

• ETEX, Frankfurt

• Conexor oy, Helsinki

• Timehouse oy, Helsinki

• KTH, Stockholm http://www.sics.se/dumas


DUMAS Objectives

Interactive email application• Dynamic – various capabilities

• Universal – various situations and formats

• Mobility – various mobile applications for

• Adaptive – learning systems

• Speech – spoken and text input

• Interfaces – intelligent interaction

http://www.sics.se/dumas/

Goals

Development of speech-based applications• multilingual: Finnish, Swedish, English

• adapts to the user’s needs and habits

Main application: AthosMail• Experiments at the end of the project:

• Athos-radiostation

• Athos-text-TV

UIAH responsibility: user modelling components

http://www.sics.se/dumas/

Challenges for User Modelling

Learn from the user-computer interaction those aspects that are important in making the use flexible and enjoyable: • cognitive load

• speaking habits

• dialogue strategies

User Model is involved in almost all decision making from speech recognition to dialogue management to speech synthesis

Classification and learning methods e.g. • neural networks, Bayes-nets, reinforcement learning


Learning via Interaction Situation: language is activity between rational agents (cf.

Allwood)• Contact + perception + understanding + reaction

Task: achieve a communicative goal • maximise mutual comprehensibility• minimise ambiguity

Constraints:• language is possessed by a group of agents => cooperation • limited resources => adaptation to new situations

Reinforcement learning• agent takes an action a, finds itself in a state s, and receives a

reward r• The task is to find a policy that maximizes the agent's reward in an

environment


Adaptive multimodal interfaces

What and when to adapt? User-centred parameters:

• Habits and preferences

• Attitudes and intentions

Environmental parameters:• E.g. speech recognizer accuracy


Action paths for an average user


MUMMI: Multimodality and Museum Interfaces

Study project together with Marjo Mäenpää, Antti Raike Cooperation with the Finnish National Gallery: Marjatta

Levanto, Riikka Haapalainen New ways of relating the arts that are both visually

interesting and accessible in terms of contents:• Virtual art exhibition, interactive guiding of the user

through the exhibition

• Text, speech, signing avatar

• Design for all Accessibility to the virtual visitors on museum web sites


Design for All Places the user in the centre (user-centred design) Cognitive factors (perception, memory, learning,

problem-solving, etc.) as they come into play during interactions with things

Usefulness: what is relevant• do the functions, information, etc., match what the user actually

needs? Usability: ease-of-use

• a simple concept, but not always easy or intuitive to implement New ways to interact with computers?


Other Multimodal Projects at MediaLab

QuiQui’s Giant Bounce (Kukakumma Muumaassa)• Perttu Hämäläinen, Johanna Höysniemi

• http://www.kukakumma.net/

• use your body to play

• interaction with your body

• child-centred design



Cinemasense (Elokuvantaju)• Antti Raike

• http://elokuvantaju.uiah.fi/

• web portal for film production (learning material)

• organise cinematic concepts in the student’s mind

• especially sign language



Experimental, affective interfaces• Jukka Ylitalo, Heidi Tikka

• http://mlab.uiah.fi/eia/

• interactive media, media and art


Burning Issues Conversational interfaces

• Dialogue processing: turn taking, feedback, repairs, non-verbal elements

Architectures• Learning in agent-based architectures• How to plug-&-play?

Processing techniques• Cognitive models of language understanding• Machine learning (supervised vs. unsupervised)

Design for all• Usability: for whom, why, what• Evaluation


References Cinemasense http://elokuvantaju.uiah.fi/ DUMAS http://www.sics.se/dumas Experimental interfaces http://mlab.uiah.fi/eia/ Interact http://www.mlab.uiah.fi/interact/ MUMMI http://mlab.uiah.fi//mummi/ QuiQui http://www.kukakumma.net/ Jokinen et al. (2002). Adaptive Dialogue Systems – Interaction with

Interact. Proceedings of the 3rd SIGDial Workshop, Philadelphia, US. Jokinen, K., J. Rissanen, H. Keränen, and K. Kanto (2002). Learning

interaction patterns for adaptive user interfaces. The 7th ERCIM UI4All Workshop, October, Paris, France.

Jokinen, K. and A. Raike (2002). Multimodality – the latest technology and visions and demands for the future. Multimodality IT-seminar, Castberggård, Denmark.

15-nov-2002mumin workshop1 multimodal research at uiah kristiina jokinen medialab university of art...

Documents

media research

information design

fields of design

techniquessupport design

art education

workshopuniversity of

timenew media program

adaptive methods http