multi-modal human-computer interaction · multi-modal human-computer interaction - 18. ted in...
TRANSCRIPT
![Page 2: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/2.jpg)
Hungary and Debrecen
Multi-modal Human-Computer Interaction - 2
![Page 3: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/3.jpg)
Debrecen – Big Church
Multi-modal Human-Computer Interaction - 3
![Page 4: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/4.jpg)
University of DebrecenMain Building
Multi-modal Human-Computer Interaction - 4
![Page 5: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/5.jpg)
Road Map
➡ Multi-modal interactions and systems (main cate-
gories, examples, benefits)
➡ Turk-2 – Multi-modal chess player
➡ Face detection, facial gestures recognition
➡ Experimental results
➡ Examples
Multi-modal Human-Computer Interaction - 5
![Page 6: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/6.jpg)
Defining Multi-modal Interaction1
➡ There are two views on multi-modal interaction:
![Page 7: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/7.jpg)
Defining Multi-modal Interaction1
➡ There are two views on multi-modal interaction:
➠ The first focuses on the human side: perception
and control. There the word modality refers to
human input and output channels.
1L. Schomaker et all, A Taxonomy of Multimodal Interaction in theHuman Information Processing System. A Report of the Espirit BasicResearch Action 8579 MIAMI. February, 1995
Multi-modal Human-Computer Interaction - 6
![Page 8: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/8.jpg)
➠ The second view focuses on using two or more
computer input or output modalities to build
system that make synergistic use of parallel input
or output of these modalities.
Multi-modal Human-Computer Interaction - 7
![Page 9: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/9.jpg)
Multi-modal Interaction: AHuman-Centered View2
➡ The focus is on multi-modal perception and cont-
rol, that is, human input and output channels.
![Page 10: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/10.jpg)
Multi-modal Interaction: AHuman-Centered View2
➡ The focus is on multi-modal perception and cont-
rol, that is, human input and output channels.
➡ Perception means the process of transforming sen-
sory information to higher-level representation.
2L. Schomaker et all, A Taxonomy of Multimodal Interaction in theHuman Information Processing System. A Report of the Espirit BasicResearch Action 8579 MIAMI. February, 1995
Multi-modal Human-Computer Interaction - 8
![Page 11: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/11.jpg)
The Modalities From a NeurobiologicalPoint of View 3
➡ We can divide the modalities in seven groups
![Page 12: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/12.jpg)
The Modalities From a NeurobiologicalPoint of View 3
➡ We can divide the modalities in seven groups
➠ Internal chemical (blood oxygen, glucose, pH)
![Page 13: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/13.jpg)
The Modalities From a NeurobiologicalPoint of View 3
➡ We can divide the modalities in seven groups
➠ Internal chemical (blood oxygen, glucose, pH)
➠ External chemical (taste, smell)
![Page 14: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/14.jpg)
The Modalities From a NeurobiologicalPoint of View 3
➡ We can divide the modalities in seven groups
➠ Internal chemical (blood oxygen, glucose, pH)
➠ External chemical (taste, smell)
➠ Somatic senses (touch,pressure, temperature,
pain)
![Page 15: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/15.jpg)
The Modalities From a NeurobiologicalPoint of View 3
➡ We can divide the modalities in seven groups
➠ Internal chemical (blood oxygen, glucose, pH)
➠ External chemical (taste, smell)
➠ Somatic senses (touch,pressure, temperature,
pain)
➠ Muscle sense (stretch,tension, join position)3E.R. Kandel and J.R. Schwartz, Principles of Neural Sciencies. Elsevier
Science Publisher, 1981.
Multi-modal Human-Computer Interaction - 9
![Page 16: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/16.jpg)
➠ Sense of balance
![Page 17: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/17.jpg)
➠ Sense of balance
➠ Hearing
![Page 18: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/18.jpg)
➠ Sense of balance
➠ Hearing
➠ Vision
Multi-modal Human-Computer Interaction - 10
![Page 19: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/19.jpg)
Multi-modal Interaction: ASystem-Centered View4
➡ In computer science multi-modal user interfaces
have been defined in many ways. Chatty gives a
summary of definitions for multi-modal interaction
by explaining that most authors defined systems
that4S. Chatty, Extending a graphical toolkit for two-handed interaction,
ACM UIST’94 Symposium on User Interface Software and Technology,ACM Press, 1994, 195–204.
Multi-modal Human-Computer Interaction - 11
![Page 20: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/20.jpg)
➠ multiple input devices (multi-sensor interaction),
![Page 21: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/21.jpg)
➠ multiple input devices (multi-sensor interaction),
➠ multiple interpretations of input issued through
a single device.
![Page 22: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/22.jpg)
➠ multiple input devices (multi-sensor interaction),
➠ multiple interpretations of input issued through
a single device.
➡ Chatty’s explanation of multi-modal interaction is
the one that most computer scientist use. With
the term multi-modal user interface they mean a
system that accepts many different inputs that are
combined in a meaningful way.
Multi-modal Human-Computer Interaction - 12
![Page 23: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/23.jpg)
Definition of the Multimodality5
➡ ”Multi-modality is the capacity of the system to
communicate with a user along different types of
communication channels and to extract and convey
meaning automatically.”
5L. Nigay and J. Coutaz, A design space for multi-modal systems: con-current processing and data fusion. Human Factors in Computer Systems,INTERCHI’93 Conference Proceedings, ACM Press, 1993, 172-178.
Multi-modal Human-Computer Interaction - 13
![Page 24: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/24.jpg)
➡ Both multimedia and multi-modal systems use
multiple communication channels.
![Page 25: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/25.jpg)
➡ Both multimedia and multi-modal systems use
multiple communication channels. But a multi-
modal system strives for meaning.
![Page 26: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/26.jpg)
➡ Both multimedia and multi-modal systems use
multiple communication channels. But a multi-
modal system strives for meaning.
➡ For example, an electronic mail system that sup-
ports voice and video clips is not multi-modal if
it only transfer them and does not interpret the
inputs.
Multi-modal Human-Computer Interaction - 14
![Page 27: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/27.jpg)
Two Main Categories of Multi-modalSystems
➡ The goal is to use the computer as a tool.
![Page 28: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/28.jpg)
Two Main Categories of Multi-modalSystems
➡ The goal is to use the computer as a tool.
➡ The computer as a dialogue partner.
Multi-modal Human-Computer Interaction - 15
![Page 29: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/29.jpg)
The History of Multi-modal UserInterfaces6
➡ Morton Heiling’s Sensorama
![Page 30: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/30.jpg)
The History of Multi-modal UserInterfaces6
➡ Morton Heiling’s Sensorama. Virtual reality sys-
tems are also quite different from multi-modal user
interfaces.
![Page 31: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/31.jpg)
The History of Multi-modal UserInterfaces6
➡ Morton Heiling’s Sensorama. Virtual reality sys-
tems are also quite different from multi-modal user
interfaces.
➡ Bolt’s Put-That-There system
![Page 32: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/32.jpg)
The History of Multi-modal UserInterfaces6
➡ Morton Heiling’s Sensorama. Virtual reality sys-
tems are also quite different from multi-modal user
interfaces.
➡ Bolt’s Put-That-There system. In this system the
user could move objects on screen by pointing and6R. Raisamo, Multimodal Human-Computer Interaction: a construc-
tive and empirical study, Academic Dissertation, University of Tampere,Tampere, 1999.
Multi-modal Human-Computer Interaction - 16
![Page 33: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/33.jpg)
speaking.
![Page 34: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/34.jpg)
speaking.
➡ CUBRICON is a system that uses mouse pointing
and speech.
![Page 35: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/35.jpg)
speaking.
➡ CUBRICON is a system that uses mouse pointing
and speech.
➡ Oviatt presented a multi-modal system for dyna-
mic interactive maps.
![Page 36: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/36.jpg)
speaking.
➡ CUBRICON is a system that uses mouse pointing
and speech.
➡ Oviatt presented a multi-modal system for dyna-
mic interactive maps.
➡ Digital Smart Kiosk.
Multi-modal Human-Computer Interaction - 17
![Page 37: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/37.jpg)
Benefits of Multi-modal Interfaces7
➡ Efficiency follows from using each modality for the
task that it is best suited for.
![Page 38: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/38.jpg)
Benefits of Multi-modal Interfaces7
➡ Efficiency follows from using each modality for the
task that it is best suited for.
➡ Redundancy increases the likelihood that com-
munication proceeds smoothly because there are
many simultaneous references to the same issue.
![Page 39: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/39.jpg)
Benefits of Multi-modal Interfaces7
➡ Efficiency follows from using each modality for the
task that it is best suited for.
➡ Redundancy increases the likelihood that com-
munication proceeds smoothly because there are
many simultaneous references to the same issue.
➡ Perceptability increas when the tasks are facilita-7M.T. Maybury and W. Wahlster (Eds.), Readings in Intelligent User
Interfaces, Morgan Kaufmann Publisher, 1998.
Multi-modal Human-Computer Interaction - 18
![Page 40: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/40.jpg)
ted in spatial context.
![Page 41: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/41.jpg)
ted in spatial context.
➡ Naturalness follows from the free choice of mo-
dalities and may result in a human-computer
communication that is close to human-human
communication.
![Page 42: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/42.jpg)
ted in spatial context.
➡ Naturalness follows from the free choice of mo-
dalities and may result in a human-computer
communication that is close to human-human
communication.
➡ Accuracy increases when another modality can
indicate an object more accurately than the main
modality.
![Page 43: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/43.jpg)
ted in spatial context.
➡ Naturalness follows from the free choice of mo-
dalities and may result in a human-computer
communication that is close to human-human
communication.
➡ Accuracy increases when another modality can
indicate an object more accurately than the main
modality.
➡ Synergy occurs when one channel of communica-
Multi-modal Human-Computer Interaction - 19
![Page 44: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/44.jpg)
tion can help refine imprecision, modify the mea-
ning, or resolve ambihuities in another channel.
Multi-modal Human-Computer Interaction - 20
![Page 45: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/45.jpg)
Applications
➡ Mobile telecommunication
![Page 46: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/46.jpg)
Applications
➡ Mobile telecommunication
➡ Hands-free devices to computers
![Page 47: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/47.jpg)
Applications
➡ Mobile telecommunication
➡ Hands-free devices to computers
➡ Using in a car
![Page 48: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/48.jpg)
Applications
➡ Mobile telecommunication
➡ Hands-free devices to computers
➡ Using in a car
➡ Interactive information panel
Multi-modal Human-Computer Interaction - 21
![Page 49: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/49.jpg)
Multi-modal Chess Player
Multi-modal Human-Computer Interaction - 22
![Page 50: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/50.jpg)
Turk 2 – Multi-modal Chess Player
Multi-modal Human-Computer Interaction - 23
![Page 51: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/51.jpg)
Turk 2 – System Components
Multi-modal Human-Computer Interaction - 24
![Page 52: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/52.jpg)
Face Detection, Facial GesturesRecognition
Multi-modal Human-Computer Interaction - 25
![Page 53: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/53.jpg)
Introduction
➡ Faces are our interfaces in our emotional and
social live.
![Page 54: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/54.jpg)
Introduction
➡ Faces are our interfaces in our emotional and
social live.
➡ Automatic analysis of facial gestures is rapidly be-
coming an area of interest in multi-modal human-
computer interaction.
![Page 55: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/55.jpg)
Introduction
➡ Faces are our interfaces in our emotional and
social live.
➡ Automatic analysis of facial gestures is rapidly be-
coming an area of interest in multi-modal human-
computer interaction.
➡ Basic goal of this area of research is a human-like
description of shown facial expression.
Multi-modal Human-Computer Interaction - 26
![Page 56: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/56.jpg)
➡ The solution of this problem can be based on the
idea of some face detection approaches.
Multi-modal Human-Computer Interaction - 27
![Page 57: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/57.jpg)
Related Research Topics
➡ Face detection (one face/image)
![Page 58: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/58.jpg)
Related Research Topics
➡ Face detection (one face/image)
➡ Face localization (more faces/image)
![Page 59: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/59.jpg)
Related Research Topics
➡ Face detection (one face/image)
➡ Face localization (more faces/image)
➡ Facial feature detection (eyes, mouth, etc.)
![Page 60: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/60.jpg)
Related Research Topics
➡ Face detection (one face/image)
➡ Face localization (more faces/image)
➡ Facial feature detection (eyes, mouth, etc.)
➡ Facial expression recognition
![Page 61: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/61.jpg)
Related Research Topics
➡ Face detection (one face/image)
➡ Face localization (more faces/image)
➡ Facial feature detection (eyes, mouth, etc.)
➡ Facial expression recognition
➡ Face recognition, face identification
Multi-modal Human-Computer Interaction - 28
![Page 62: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/62.jpg)
➡ Face tracking
Multi-modal Human-Computer Interaction - 29
![Page 63: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/63.jpg)
Problems of the Face Detection
➡ Pose: The images of a face vary due to the relative
camera-face pose.
![Page 64: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/64.jpg)
Problems of the Face Detection
➡ Pose: The images of a face vary due to the relative
camera-face pose.
➡ Presence or absence of structural components (be-
ards, mustaches, glasses etc.).
![Page 65: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/65.jpg)
Problems of the Face Detection
➡ Pose: The images of a face vary due to the relative
camera-face pose.
➡ Presence or absence of structural components (be-
ards, mustaches, glasses etc.).
➡ Facial expression: The appearance of faces are
directly affected by the facial expression.
Multi-modal Human-Computer Interaction - 30
![Page 66: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/66.jpg)
➡ Occlusion: Faces may be partially occluded by
other objects.
![Page 67: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/67.jpg)
➡ Occlusion: Faces may be partially occluded by
other objects.
➡ Image orientation: Face images vary for different
rotations about the optical axis of the camera.
![Page 68: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/68.jpg)
➡ Occlusion: Faces may be partially occluded by
other objects.
➡ Image orientation: Face images vary for different
rotations about the optical axis of the camera.
➡ Imaging conditions (lighting, background, camera
characteristics).
Multi-modal Human-Computer Interaction - 31
![Page 69: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/69.jpg)
Detecting Faces in a Single Image
➡ Knowledge-based methods (G. Yang and T.S. Hu-
ang, 1994).
![Page 70: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/70.jpg)
Detecting Faces in a Single Image
➡ Knowledge-based methods (G. Yang and T.S. Hu-
ang, 1994).
➡ Feature invariant approaches (T. K. Leung, M. C.
Burl, and P. Perona, 1995), (K. C. Yow and R.
Cipolla, 1996).
![Page 71: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/71.jpg)
Detecting Faces in a Single Image
➡ Knowledge-based methods (G. Yang and T.S. Hu-
ang, 1994).
➡ Feature invariant approaches (T. K. Leung, M. C.
Burl, and P. Perona, 1995), (K. C. Yow and R.
Cipolla, 1996).
➡ Template matching methods (A. Lanitis, C. J.
Taylor, and T. F. Cootes, 1995).
Multi-modal Human-Computer Interaction - 32
![Page 72: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/72.jpg)
➡ Appearance-based methods (E. Osuna, R. Freund,
and F. Girosi, 1997), (A. Fazekas, C. Kotropoulos,
I. Pitas, 2002).
Multi-modal Human-Computer Interaction - 33
![Page 73: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/73.jpg)
Detecting Faces in a Single Image
➡ Scanning of the picture by a running window in a
multiresolution pyramid.
![Page 74: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/74.jpg)
Detecting Faces in a Single Image
➡ Scanning of the picture by a running window in a
multiresolution pyramid.
➡ Normalize of the window.
![Page 75: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/75.jpg)
Detecting Faces in a Single Image
➡ Scanning of the picture by a running window in a
multiresolution pyramid.
➡ Normalize of the window.
➡ Hide some parts of the face.
![Page 76: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/76.jpg)
Detecting Faces in a Single Image
➡ Scanning of the picture by a running window in a
multiresolution pyramid.
➡ Normalize of the window.
➡ Hide some parts of the face.
➡ Normalize of the local variance of the brightness
on the picture.
Multi-modal Human-Computer Interaction - 34
![Page 77: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/77.jpg)
➡ Equalization of the histogram.
![Page 78: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/78.jpg)
➡ Equalization of the histogram.
➡ Localization of the face (decision).
Multi-modal Human-Computer Interaction - 35
![Page 79: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/79.jpg)
Face Gesture Recognition like BinaryClassification Problem
➡ Let us consider a set of the facial pictures.
![Page 80: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/80.jpg)
Face Gesture Recognition like BinaryClassification Problem
➡ Let us consider a set of the facial pictures.
➡ Let us set up a finite system of some features
related the pictures.
![Page 81: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/81.jpg)
Face Gesture Recognition like BinaryClassification Problem
➡ Let us consider a set of the facial pictures.
➡ Let us set up a finite system of some features
related the pictures.
➡ It is known any pictures is related to only one
class:
![Page 82: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/82.jpg)
Face Gesture Recognition like BinaryClassification Problem
➡ Let us consider a set of the facial pictures.
➡ Let us set up a finite system of some features
related the pictures.
➡ It is known any pictures is related to only one
class: face with the given gesture,
![Page 83: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/83.jpg)
Face Gesture Recognition like BinaryClassification Problem
➡ Let us consider a set of the facial pictures.
➡ Let us set up a finite system of some features
related the pictures.
➡ It is known any pictures is related to only one
class: face with the given gesture, face without
the given gesture.
Multi-modal Human-Computer Interaction - 36
![Page 84: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/84.jpg)
➡ The problem to find a method to determine the
class of the examined picture.
![Page 85: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/85.jpg)
➡ The problem to find a method to determine the
class of the examined picture.
➡ One possible way to solve this problem: Support
Vector Machine.
Multi-modal Human-Computer Interaction - 37
![Page 86: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/86.jpg)
Support Vector Machine
➡ Statistical learning from examples aims at selec-
ting from a given set of functions {fα(x) | α ∈ Λ},the one which predicts best the correct response.
![Page 87: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/87.jpg)
Support Vector Machine
➡ Statistical learning from examples aims at selec-
ting from a given set of functions {fα(x) | α ∈ Λ},the one which predicts best the correct response.
➡ This selection is based on the observation of l
pairs that build the training set:
(x1, y1), . . . , (xl, yl), xi ∈ Rm, yi ∈ {+1,−1}Multi-modal Human-Computer Interaction - 38
![Page 88: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/88.jpg)
which contains input vectors xi and the associated
ground ”truth” given by an external supervisor.
![Page 89: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/89.jpg)
which contains input vectors xi and the associated
ground ”truth” given by an external supervisor.
➡ Let the response of the learning machine fα(x)belongs to a set of indicator functions {fα(x) | x ∈Rm, α ∈ Λ}.
![Page 90: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/90.jpg)
which contains input vectors xi and the associated
ground ”truth” given by an external supervisor.
➡ Let the response of the learning machine fα(x)belongs to a set of indicator functions {fα(x) | x ∈Rm, α ∈ Λ}.
➡ If we define the loss-function:
L(y, fα(x)) ={
0, if y = fα(x),1, if y 6= fα(x).
Multi-modal Human-Computer Interaction - 39
![Page 91: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/91.jpg)
The expected value of the loss is given by:
R(α) =∫
L(y, fα(x))p(x, y)dxdy,
where p(x, y) is the joint probability density func-
tion of random variables x and y.
![Page 92: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/92.jpg)
The expected value of the loss is given by:
R(α) =∫
L(y, fα(x))p(x, y)dxdy,
where p(x, y) is the joint probability density func-
tion of random variables x and y.
➡ We would like to find the function fα0(x) which
minimizes the risk function R(α).
![Page 93: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/93.jpg)
The expected value of the loss is given by:
R(α) =∫
L(y, fα(x))p(x, y)dxdy,
where p(x, y) is the joint probability density func-
tion of random variables x and y.
➡ We would like to find the function fα0(x) which
minimizes the risk function R(α).
➡ The basic idea of SVM to construct the optimal
separating hyperplane.
Multi-modal Human-Computer Interaction - 40
![Page 94: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/94.jpg)
➡ Suppose that the training data can be separated
by a hyperplane, fα(x) = αTx + b = 0, such that:
yi(αTxi + b) ≥ 1, i = 1, 2, . . . , l
where α is the normal to the hyperplane.
![Page 95: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/95.jpg)
➡ Suppose that the training data can be separated
by a hyperplane, fα(x) = αTx + b = 0, such that:
yi(αTxi + b) ≥ 1, i = 1, 2, . . . , l
where α is the normal to the hyperplane.
➡ For the linearly separable case, SVM simply se-
eks for the separating hyperplane with the largest
margin.
Multi-modal Human-Computer Interaction - 41
![Page 96: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/96.jpg)
➡ For linearly nonseparable data, by mapping the in-
put vectors, which are the elements of the training
set, into a high-dimensional feature space through
so-called kernel function.
![Page 97: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/97.jpg)
➡ For linearly nonseparable data, by mapping the in-
put vectors, which are the elements of the training
set, into a high-dimensional feature space through
so-called kernel function.
➡ We construct the optimal separating hyperplane
in the feature space to get a binary decision.
Multi-modal Human-Computer Interaction - 42
![Page 98: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/98.jpg)
Experimental Results
➡ For all experiments the package SVMLight deve-
loped by T. Joachims was used. For complete test,
several routines have been added to the original
toolbox.
![Page 99: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/99.jpg)
Experimental Results
➡ For all experiments the package SVMLight deve-
loped by T. Joachims was used. For complete test,
several routines have been added to the original
toolbox.
➡ The database recorded by our institute was used.
Multi-modal Human-Computer Interaction - 43
![Page 100: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/100.jpg)
➡ Training set of 40 images (20 faces with the given
gesture, 20 faces without the given gesture.).
![Page 101: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/101.jpg)
➡ Training set of 40 images (20 faces with the given
gesture, 20 faces without the given gesture.).
➡ All images are recorded in 256 grey levels.
![Page 102: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/102.jpg)
➡ Training set of 40 images (20 faces with the given
gesture, 20 faces without the given gesture.).
➡ All images are recorded in 256 grey levels.
➡ They are of dimension 640× 480.
![Page 103: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/103.jpg)
➡ Training set of 40 images (20 faces with the given
gesture, 20 faces without the given gesture.).
➡ All images are recorded in 256 grey levels.
➡ They are of dimension 640× 480.
➡ The procedure for collecting face patterns is as
follows.
Multi-modal Human-Computer Interaction - 44
![Page 104: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/104.jpg)
➡ A rectangle part of dimension 256×320 pixels has
been manually determined that includes the actual
face.
![Page 105: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/105.jpg)
➡ A rectangle part of dimension 256×320 pixels has
been manually determined that includes the actual
face.
➡ This area has been subsampled four times. At
each subsampling, non-overlapping regions of 2×2pixels are replaced by their average.
Multi-modal Human-Computer Interaction - 45
![Page 106: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/106.jpg)
➡ The training patterns of dimension 16 × 20 are
built.
![Page 107: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/107.jpg)
➡ The training patterns of dimension 16 × 20 are
built.
➡ The class label +1 has been appended to each
pattern.
![Page 108: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/108.jpg)
➡ The training patterns of dimension 16 × 20 are
built.
➡ The class label +1 has been appended to each
pattern.
➡ Similarly, 20 non-face patterns have been collected
from images in the same way, and labeled −1.
Multi-modal Human-Computer Interaction - 46
![Page 109: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/109.jpg)
Facial Gesture Database
Surprising face Smiling face
Sad face Angry face
Multi-modal Human-Computer Interaction - 47
![Page 110: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/110.jpg)
Classification Error on Facial GestureDatabase
Angry Happy Sad Serial Suprised22.4% 10.3% 11.8% 9.4% 18.9%
Multi-modal Human-Computer Interaction - 48
![Page 111: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/111.jpg)
Examples
Multi-modal Human-Computer Interaction - 49
![Page 112: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/112.jpg)
Multi-modal Human-Computer Interaction - 50
![Page 113: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/113.jpg)
Multi-modal Human-Computer Interaction - 51
![Page 114: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/114.jpg)
Multi-modal Human-Computer Interaction - 52
![Page 115: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/115.jpg)
Multi-modal Human-Computer Interaction - 53
![Page 116: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/116.jpg)
Multi-modal Human-Computer Interaction - 54
![Page 117: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/117.jpg)
Multi-modal Human-Computer Interaction - 55
![Page 118: Multi-modal Human-Computer Interaction · Multi-modal Human-Computer Interaction - 18. ted in spatial context. ted in spatial context. Naturalness follows from the free choice of](https://reader030.vdocuments.us/reader030/viewer/2022041103/5f0258207e708231d403ceb6/html5/thumbnails/118.jpg)
Multi-modal Human-Computer Interaction - 56