15-nov-2002mumin workshop1 multimodal research at uiah kristiina jokinen medialab university of art...
Post on 11-Jan-2016
218 Views
Preview:
TRANSCRIPT
15-Nov-2002 MUMIN Workshop 1
Multimodal research at UIAH
Kristiina Jokinen
MediaLab
University of Art and Design HelsinkiKristiina.Jokinen@uiah.fi
15-Nov-2002 MUMIN Workshop 2
University of Art and Design
Education and research in the fields of design, new media, audiovisual communication, art education, arts
Largest art school in Scandinavia Nearly 1600 students, 15% from abroad Lume, the Finnish centre for media research
and development, was opened in 2000• Departments of film and television, design for theatre,
film and television, and new media
15-Nov-2002 MUMIN Workshop 3
Media Lab
Formed in 1993 Explore, discover and comprehend the new digital
technology • Impact in society
• Possibilities for communication, interaction and expression
• Challenges to new media and information design 2-year masters programme
• MA in New Media (full time)
• New Media program for professionals (flexible study method) 20 full-time, 15 professionals, and approx. 30 minor
subject students annually to the MA programmes
15-Nov-2002 MUMIN Workshop 4
Adaptive Systems for Complex Interaction
Research on natural interaction between humans and computers• Interaction strategies, cooperative response planning
• Various input modalities
• Concepts, models Apply machine-learning techniques to dialogue
processing • Compare and test applicability of the techniques
Support Design-for-all principles in designing intelligent interfaces
15-Nov-2002 MUMIN Workshop 5
Human-computer interaction
Computer as a tool• Passive and transparent
• Supports the human goals, human control
Computer as an agent• Models of beliefs, desires, intentions (BDI)
• Intelligent software mediating between the human and an application
• Cooperation, negotiation
• Complex interaction
• Multimodal communication
15-Nov-2002 MUMIN Workshop 6
Projects
USIX-Interact: Natural Interaction and Adaptive Methods http://www.mlab.uiah.fi/interact/
DUMAS: Dynamic User Modelling for Adaptive Speech Interfaces http://www.sics.se/dumas/
MUMMI: Multi-Modal Museum Interfaces (Study project together with Marjo Mäenpää and Antti Raike, Design for All, Virtual Art Exhibition) http://mlab.uiah.fi/mummi/
15-Nov-2002 MUMIN Workshop 7
Natural interaction Language that suits to computers Language that humans use to
communicate
=> Language that humans and computers use when interacting with each other
- Different ? How?
- How it emerges from interaction?
15-Nov-2002 MUMIN Workshop 8
Interact: key aspects for adaptivity Conversational ability
• Dialogue modelling and natural communication Learning systems
• Various methods and techniques
• Various interface techniques (speech, text, map) Language technology
• Finnish and multilingual Agent-based architecture
• Jaspis development platform http://www.mlab.uiah.fi/interact/
15-Nov-2002 MUMIN Workshop 9
Interact Partners
University of Art and Design, Media Lab
University of Helsinki, Language Technology
University of Tampere, TAUCHI unit
Helsinki University of Technology, Neural Networks Research Centre
Fujitsu Invia oyj Tecnomen oyj Lingsoft oy The Arla Institute Finnish Association
for the Deaf Finnish Technology
Agency
15-Nov-2002 MUMIN Workshop 10
DUMAS - Dynamic Universal Mobility for Adaptive Speech interfaces
EU 5th framework R&D project• Swedish Institute of Computer
Science
• UIAH, Media Lab
• University of Tampere, TAUCHI-unit
• UMIST, Manchester
• ETEX, Frankfurt
• Conexor oy, Helsinki
• Timehouse oy, Helsinki
• KTH, Stockholm http://www.sics.se/dumas
15-Nov-2002 MUMIN Workshop 11
DUMAS Objectives
Interactive email application• Dynamic – various capabilities
• Universal – various situations and formats
• Mobility – various mobile applications for
• Adaptive – learning systems
• Speech – spoken and text input
• Interfaces – intelligent interaction
http://www.sics.se/dumas/
Goals
Development of speech-based applications• multilingual: Finnish, Swedish, English
• adapts to the user’s needs and habits
Main application: AthosMail• Experiments at the end of the project:
• Athos-radiostation
• Athos-text-TV
UIAH responsibility: user modelling components
http://www.sics.se/dumas/
Challenges for User Modelling
Learn from the user-computer interaction those aspects that are important in making the use flexible and enjoyable: • cognitive load
• speaking habits
• dialogue strategies
User Model is involved in almost all decision making from speech recognition to dialogue management to speech synthesis
Classification and learning methods e.g. • neural networks, Bayes-nets, reinforcement learning
15-Nov-2002 MUMIN Workshop 14
Learning via Interaction Situation: language is activity between rational agents (cf.
Allwood)• Contact + perception + understanding + reaction
Task: achieve a communicative goal • maximise mutual comprehensibility• minimise ambiguity
Constraints:• language is possessed by a group of agents => cooperation • limited resources => adaptation to new situations
Reinforcement learning• agent takes an action a, finds itself in a state s, and receives a
reward r• The task is to find a policy that maximizes the agent's reward in an
environment
15-Nov-2002 MUMIN Workshop 15
Adaptive multimodal interfaces
What and when to adapt? User-centred parameters:
• Habits and preferences
• Attitudes and intentions
Environmental parameters:• E.g. speech recognizer accuracy
15-Nov-2002 MUMIN Workshop 16
Action paths for an average user
15-Nov-2002 MUMIN Workshop 17
MUMMI: Multimodality and Museum Interfaces
Study project together with Marjo Mäenpää, Antti Raike Cooperation with the Finnish National Gallery: Marjatta
Levanto, Riikka Haapalainen New ways of relating the arts that are both visually
interesting and accessible in terms of contents:• Virtual art exhibition, interactive guiding of the user
through the exhibition
• Text, speech, signing avatar
• Design for all Accessibility to the virtual visitors on museum web sites
15-Nov-2002 MUMIN Workshop 18
Design for All Places the user in the centre (user-centred design) Cognitive factors (perception, memory, learning,
problem-solving, etc.) as they come into play during interactions with things
Usefulness: what is relevant• do the functions, information, etc., match what the user actually
needs? Usability: ease-of-use
• a simple concept, but not always easy or intuitive to implement New ways to interact with computers?
15-Nov-2002 MUMIN Workshop 19
Other Multimodal Projects at MediaLab
QuiQui’s Giant Bounce (Kukakumma Muumaassa)• Perttu Hämäläinen, Johanna Höysniemi
• http://www.kukakumma.net/
• use your body to play
• interaction with your body
• child-centred design
15-Nov-2002 MUMIN Workshop 20
Other Multimodal Projects at MediaLab
Cinemasense (Elokuvantaju)• Antti Raike
• http://elokuvantaju.uiah.fi/
• web portal for film production (learning material)
• organise cinematic concepts in the student’s mind
• especially sign language
15-Nov-2002 MUMIN Workshop 21
Other Multimodal Projects at MediaLab
Experimental, affective interfaces• Jukka Ylitalo, Heidi Tikka
• http://mlab.uiah.fi/eia/
• interactive media, media and art
15-Nov-2002 MUMIN Workshop 22
Burning Issues Conversational interfaces
• Dialogue processing: turn taking, feedback, repairs, non-verbal elements
Architectures• Learning in agent-based architectures• How to plug-&-play?
Processing techniques• Cognitive models of language understanding• Machine learning (supervised vs. unsupervised)
Design for all• Usability: for whom, why, what• Evaluation
15-Nov-2002 MUMIN Workshop 23
References Cinemasense http://elokuvantaju.uiah.fi/ DUMAS http://www.sics.se/dumas Experimental interfaces http://mlab.uiah.fi/eia/ Interact http://www.mlab.uiah.fi/interact/ MUMMI http://mlab.uiah.fi//mummi/ QuiQui http://www.kukakumma.net/ Jokinen et al. (2002). Adaptive Dialogue Systems – Interaction with
Interact. Proceedings of the 3rd SIGDial Workshop, Philadelphia, US. Jokinen, K., J. Rissanen, H. Keränen, and K. Kanto (2002). Learning
interaction patterns for adaptive user interfaces. The 7th ERCIM UI4All Workshop, October, Paris, France.
Jokinen, K. and A. Raike (2002). Multimodality – the latest technology and visions and demands for the future. Multimodality IT-seminar, Castberggård, Denmark.
top related