toyoaki nishida kyoto university...[kendon 2004] a context-of-use study of open hand prone gestures...
TRANSCRIPT
Artificial Intelligence Adv., October 2nd, 2013
Copyright © 2013 Toyoaki Nishida All Rights Reserved.
1. Introduction
Toyoaki Nishida Kyoto University
Conversational Informatics: a field of study that centers on understanding and augmenting conversation
Prologue
At a marketplace
Under cherry blossom
At a seminar room
Conversation is everywhere in our life
Challenge: A robot that can participate in conversation
Long-term goal
Challenge: Synthetic character who guides the user through the virtual world
Long-term goal
What is AI?
Artificial intelligence
Artificial mind
Example → Enthiran (The Robot), 2010
AI in the movies Year Title Who Wikipedia
1968 2001: A Space Odyssey HAL9000 http://ja.wikipedia.org/wiki/2001%E5%B9%B4%E5%AE%87%E5%AE%99%E3%81%AE%E6%97%85
1977 Star Wars C-3PO, R2-D2 http://ja.wikipedia.org/wiki/%E3%82%B9%E3%82%BF%E3%83%BC%E3%83%BB%E3%82%A6%E3%82%A9%E3%83%BC%E3%82%BA%E3%83%BB%E3%82%B7%E3%83%AA%E3%83%BC%E3%82%BA
1982 Blade Runner Replicants http://ja.wikipedia.org/wiki/%E3%83%96%E3%83%AC%E3%83%BC%E3%83%89%E3%83%A9%E3%83%B3%E3%83%8A%E3%83%BC
1984 The Terminator The terminator http://ja.wikipedia.org/wiki/%E3%82%BF%E3%83%BC%E3%83%9F%E3%83%8D%E3%83%BC%E3%82%BF%E3%83%BC_(%E6%98%A0%E7%94%BB)
1987 RoboCop RoboCop (cyborg) http://ja.wikipedia.org/wiki/%E3%83%AD%E3%83%9C%E3%82%B3%E3%83%83%E3%83%97
1993 War Games WOPR: War Operation Plan Response
http://ja.wikipedia.org/wiki/%E3%82%A6%E3%82%A9%E3%83%BC%E3%83%BB%E3%82%B2%E3%83%BC%E3%83%A0_(%E6%98%A0%E7%94%BB)
1994 Disclosure Angel http://ja.wikipedia.org/wiki/%E3%83%87%E3%82%A3%E3%82%B9%E3%82%AF%E3%83%AD%E3%83%BC%E3%82%B8%E3%83%A3%E3%83%BC_(%E6%98%A0%E7%94%BB)
1998 Bicentennial Man Andrew, a new NDR-114 robot http://ja.wikipedia.org/wiki/%E3%82%A2%E3%83%B3%E3%83%89%E3%83%AA%E3%83%A5%E3%83%BCNDR114
1999 The Matrix Computer http://ja.wikipedia.org/wiki/%E3%83%9E%E3%83%88%E3%83%AA%E3%83%83%E3%82%AF%E3%82%B9_(%E6%98%A0%E7%94%BB)
2001 A.I. Artificial Intelligence David , achild Mecha http://ja.wikipedia.org/wiki/A.I.
2002 Minority Report Insect robots (user interface is interesting, too)
http://ja.wikipedia.org/wiki/%E3%83%9E%E3%82%A4%E3%83%8E%E3%83%AA%E3%83%86%E3%82%A3%E3%83%BB%E3%83%AA%E3%83%9D%E3%83%BC%E3%83%88
2004 I, ROBOT V.I.K.I . , Sonny http://ja.wikipedia.org/wiki/%E3%82%A2%E3%82%A4,%E3%83%AD%E3%83%9C%E3%83%83%E3%83%88
2009 ATOM Astro boy Astro boy http://ja.wikipedia.org/wiki/ATOM_(%E6%98%A0%E7%94%BB)
2009 Avatar (tele-existence) http://ja.wikipedia.org/wiki/%E3%82%A2%E3%83%90%E3%82%BF%E3%83%BC
2009 Surrogate (tele-existence) http://ja.wikipedia.org/wiki/%E3%82%B5%E3%83%AD%E3%82%B2%E3%83%BC%E3%83%88_(%E6%98%A0%E7%94%BB)
Year AI ICT 1940~ 1936: Turing Machine, 1947: von Neumann Computer, 1948: Information
Theory, by C. Shannon and W. Weaver, 1948: Cybernetics by Wiener
1950~ 1952-62: Checker program by A.Samuel 1956: Dartmouth Conference
1957: FORTRAN by J.Backus
1960~ 1961: Symbolic Integration program SAINT by J.Slagle 1962: Perceptron by F.Rosenblatt 1966: The ALPAC report against Machine Translation by R. Pierce 1967: Formula Manipulation System Macsyma by J.Moses 1967: Dendral for Mass Spectrum Analysis by E.Feigenbaum
1961: Mathematical theory of Packet Networks by L. Kleinrock 1963: Interactive Computer Graphics by I.Sutherland 1968: Mouse and Bitmap display for oN Line System (NLS) by D.C.Engelbart 1969: ARPA-net
1970~
1971: Natural Language Dialogue System SHRDLU, by T.Winograd 1973: Combinatorial Explosion problem pointed out in The Lighthill report 1974: MYCIN by T.Shortliffe Mid 1970’s: Prial Sketch and Visual Perceptron by D.Marr 1976: Automated Mathematician (AM) by D.Lenat 1979: Autonomous Vehicle Stanford Cart by H.Moravec
1970: ALOHAnet 1970: Relational Database Theory by E.F.Codd 1972: Theory of NP-completeness by S.Cook and R.Karp Mid 1970’s: Alto Machine by A.Kay and A.Goldberg 1976: Ethernet 1979: Spreadsheet Program Visicalc by D.Bricklin
1980~
1982: Fifth Generation Computer Project 1984: The CYC Project by D.Lenat Mid 1980’s: Back-propagation algorithm was widely used 1985: the Cybernetic Artist Aaron by H.Cohen 1986: Subsumption Architecture by R.Brooks 1989: An Autonomous Vehicle ALVINN by D.Pomerleau
1982:TCP/IP Protocol by B.Kahn and V.Cerf Mid 1980’s: First Wireless Tag Products 1987: UUNET started the Commercial UUCP Network Connection Service 1988: Internet worm (Morris Worm) 1989: World Wide Web by T.Berners-Lee 1989: The number of hosts on the Internet has exceeded 100,000.
1990~
1990: Genetic Programming by J.R.Koza Early1990’s: TD-Gammon by G.Tesauro Mid 1990’s: Data Mining Technology 1997: DeepBlue defeated the World Chess Champion G.Kasparov 1997: The First Robocup by H.Kitano 1999: Robot pets became commercially available
1992: The number of hosts on the Internet has exceeded 1,000,000. 1994: Shopping malls on the Internet 1994: W3C was founded by T. Berners-Lee 1997: Google Search 1998: XML1.0(eXtensible Markup Language) by W3C 1998: PayPal
2000~ 2000: Honda Asimo 2004: The Mars Exploration Rovers (Spirit & Opportunity)
2001: Wikipedia. 2003: Skype / iTunes store 2004: Facebook 2005: YouTube / Google Earth 2006: Twitter 2007: Google Street View
2010~ 2010: Google Driverless Car / Kinect 2011: IBM Watson Jeopardy defeated two of the greatest champions 2012: Siri
History of AI research in contrast with ICT
1960 Prof. Toshiyuki Sakai’s group at Kyoto
1961 Sonotype: phonetic typewriter by Shuji Doshita
1970 The world’s first face recognition system by Takeo Kanade
1979 Japanese Word Processor JW-10 by Toshiba
1997 1st Robocup games and conference at Nagoya, Japan
1979 IJCAI 79 (Tokyo)
1997 IJCAI 97 (Nagoya)
1982-1992 Fifth Generation Project 1986 JSAI (Japanese Society for AI)
1990 1st PRICAI (Nagoya, Japan)
1970’s Research groups for AI were formed in Tokyo, Osaka, Kyushu
1960’s Image Understanding and NLP by Makoto Nagao
Successful Topics of AI
Large-scale Search Knowledge-based Systems Language, Speech, Vision Planning Machine Learning and Data Mining Using AI in Creating Works of Art
AARON (1985)
The Mars Exploration Rovers (2003)
Experiments in Musical Intelligence (1987)
DeepBlue (1997) http://www.research.ibm.com/deepblue/
http://marsrovers.jpl.nasa.gov/mission/spacecraft_surface_rover.html
http://www.kurzweilcyberart.com/
http://arts.ucsc.edu/faculty/cope/ http://artsites.ucsc.edu/faculty/cope/mp3page.htm
IBM Watson (2011) http://www-03.ibm.com/innovation/us/watson/index.shtml
ALVINN (An Autonomous Land Vehicle in a Neural Network) on Navlab (1989) http://dl.acm.org/citation.cfm?id=89891
Autonomous Vehicles - Lego Robots (1998 -) - European Land-Robot Trial (2006-) (original source not found) - DARPA Urban Challenge (2007-) -The Microtransat Challenge: a transatlantic race of fully autonomous sailing boats. (2006-) - Aerial Robotics (Sept. 28, 2011) Chess (competition ended) - Deep Blue defeated Gary Kasparov on May 11th, 1997 Education - STEM Grand Challenge to develop adaptive, generalizable intelligent tutors (June 5, 2011-) - Hewlett Foundation: Improve Automated Scoring of Student Essays (Jan, 2012) Face Recognition - the Face Recognition Vendor test (FRVT) (2002-) Go - Entertainment and Cognitive Science, The University of Electro-Communication Language Learning and Understanding - Loebner Prize (1990-) - IBM Watson Lunar Robotics - NASA Prize for Digging Moon Dirt (September 20, 2005) - Google Lunar X Prize (2007-) Medical Diagnosis and Monitoring - Nokia Sensing X Challenge (may 24, 2012) Rescue Operations - Robocup Rescue (2001-) Robotics - The DARPA Robotics Challenge (October 2012-) Japanese Chess (Shogi) - World Computer Shogi Championship,IPSJ Computer Shogi Project Shredder Challenge - DARPA Shredder Challenge (October 27th-December 2nd, 2011; solved) Soccer - Robocup (1997 -) Entrance examination of the University of Tokyo - NII Artificial Brain Project: “U Tokyo Robot”
Conceptual Turing Test (1950)
Adapted from AAAI>AITopics>Grand Challenges in AI
Grand Challenges in AI
Recent Trends
1997: Deep Blue defeated G. Kasparov … Source: IBM Deep Blue, wikipedia 1997: Official opening of Robocup (H. Kitano) … Source: robocup.org, robocup.or.jp, Wikipedia 1997: Mars Pathfinder … Source: NASA Mars Pathfinfer, Wikipedia 1999: Robot pet SONY AIBO … Source: SONY AIBO, Wikipedia 1999: OpenCV Project … Source: OpenCVWiki,wikipedia 2000: Honda ASIMO … Source: Honda ASIMO, Wikipedia 2004: Mars Exploration Rovers … Source: NASA-JPL 2010: Google Driverless Car … Source: Wikipedia, Sebastian Thrun’s home page, 3P 2010: Kinect … Source: xbox.com, Wikipedia 2011: IBM Watson defeated two Jeopardy! champions … Source: IBM Watson, Wikipedia 2011: iPhone Siri … Source: Apple, Wikipedia 2011: Google Voice Search … Source: Google 2012: Zen Takemiya Masaki with 4 stones … Source: Sig ECS, UEC 2012: Google Glass Project … Source: Project Glass 2012: NTT DOCOMO’s Shabette Concier … NTT DOCOMO Other: Smarter than You Think (New York Times)
Landmarks
- From philosophy to science-technology
- Toy problems to the real world
- Powerful tools
- Grand challenges and competitions
Traditional AI Wow! Wow!
Wow! Wow!
Wow!
Wow! Wow!
Wow!
Wow! Wow!
Wow!
Wow! Wow!
Wow! Wow!
Wow!
Wow! Wow!
Wow!
Wow! Wow!
Wow!
Dark side of super intelligence
Technology abuse
Responsibility flaw
Moral in crisis
Over-dependency on technology
(Technological) singularity
The day when machine intelligence surpasses human intelligence.
IEEE Spectrum June 2008 issue http://spectrum.ieee.org/biomedical/ethics/signs-of-the-singularity
“The AI Scenario: We create superhuman artificial intelligence (AI) in computers.”
“The IA Scenario: We enhance human intelligence through human-to-computer interfaces--that is, we achieve intelligence amplification (IA).”
The fear of utopia
So what?
Dismiss as nonsense.
Neo-Luddism.
Invent a better solution.
Endow AI with empathic capabilities.
Communicative Intelligence for Bridging People and CI
Communicative Intelligence
Computational Intelligence People
Towards Empathic Agents
Glad to stay with you!
• Service • Presence • Empathy
Embedding AI in the society
DeepBlue Entertain with a game
IBM Watson Entertain with a game
Siri Conversation partner
(AI that can pass entrance exam) Effective and affective tutor
Traditional AI: High competence
Future AI: High empathy
The ability to understand others’ emotions and/or perspectives and, often, to resonate with others’ emotional states.
An affective response that is identical, or very similar, to what the other person is feeling or might be expected to feel given the context: a response stemming from an understanding of another’s emotional state or condition.
or …
[Eisenberg 2010]
Empathy
The more common ground is shared, the more empathy will be gained.
... the universe of discourse, first-person view, knowledge and skills, the communication style and rituals, the value system, ...
[Nishida 2013]
Sharing hypothesis
Tell me your problem
I have a problem
B A
Towards Empathic Agents
Eye gaze
Hand gesture Posture Para linguistic
Asking Negotiating
Proposing
Conviviality Social networks Trust
Conversation is a complex business
[Kendon 2004]
A context-of-use study of Open Hand Prone gestures suggests that they all share the semantic theme of stopping or interrupting a line of action that is in progress.
Gestures as a part of speaker’s utterances
The Open Hand Prone Pointing Gesture specifying size, shape and structure of an object
Stop / interrupt
Index Finger Extended Supine (palm down).
Size-shape-specifier gesture serving as a referent for the deictic pronoun ‘that’.
[Clark 1996]
Adam: Sit down here [pointing at a chair] would you Bart: ...
A
B
A+B
Adam Bart
Sit down here would you
Conversation is a complex business
[Brown & Yule 1983] [Clark 1996]
Is it very rough down there though?
There are no cobbles
Rough in terms of people
Oh I see
Town
Is it very rough down there though There
are no cobbles There are no
cobbles
Rough in terms of people?
Ar e t he peopl
e r ough
?
Town
Town
I s it ver y r ough down t her e t hough
Are the road rough?
There are no cobbles
There are no cobbles
Ther e ar e no cobbles
Fix
Rough in t er m s of people? Sync
I s it ver y r ough down t her e t hough
Are the people rough?
Town
Town
Is it very rough down there though
Ar e t he r oad r ough?
There are no cobbles Ther e ar e no cobbles
There are no cobbles
Fix Rough in terms of people?
Is it very rough down there though
A B
Conversation is a complex business
The Architecture of Empathic Agent?
Self
Internal theater
Interactive
Social
Cultural
Perception
Sensors
Embodiment
Motors
cognition
Language
Memory
Imagination
Consciousness
Learning skills
Evolutional system
Mind
Judgment
t 1990 2000 2010 1980 1970
Natural language dialogue systems
Speech dialogue systems
Multi-modal dialogue systems
Embodied Conversational Agents Intelligent Virtual Human
The Knowledge Navigator
Conversation Analysis
Discourse Analysis
Story Understanding systems
Conversational Informatics
Transactional systems
Interactional systems
History of conversational systems development
Our approach
Application
Platform
Evaluation
Content production Model building
Analysis
Conversational interactions
Theory
Measurement
Our approach: Building a Primordial Soup of Conversation
Content provision Training data
Acquired content
Conversation management - Conversation environment management - Sensing human behaviors - Controlling agents
Conversation environment #1
Conversation environment #2
Conversation environment #3
Partially mechanized humans & Partially humanized machines live together and converse with each other.
materialization embodied conversational agent
conversation quantum
a conversational situation
accumulation dematerialization
another conversational situation
conversation quantum
conversation quantum
conversation quantum
conversation quantum
conversation quantum
conversion / spatialization
The Idea of Conversation Quantization
[Nishida DNIS 2005]
Augmented Conversation Environment We aim at building a smart environment that integrates conversation in physical and virtual spaces.
Conversation in the physical environment Conversation in the virtual environment ICIE: Immersive Collaborative Interaction Environment
Multiparty conversation recorder
Eye tracker
Eye tracker (wearable)
Polygraph Optical motion capture systems
Cluster computer
IMADE: Interaction Measurement, Analysis, and Design Environment
SmartInFill
Conversation in physical world Conversation in cyber world
Platform for supporting cyber-physical conversation
Cyber-physical conversation environment
a conversation robot cameras for varying purposes a computer vision and a cluster computer (to be installed)
IMADE: Interaction Measurement, Analysis, and Design Environment
Optical motion capture systems
Cluster computer
SmartInFill
Polygraph
Multiparty conversation recorder
Eye tracker Eye tracker (wearable)
IMADE: Interaction Measurement, Analysis, and Design Environment
- experiment profiles - timeline - motion data
- visual data - audio data (PCM)
Storage - experiment record - recording operation
Web server
Audio capture system ・8ch preamp
Camera system data streams
Motion data server
Wearable sensors
- Wireless head set microphones - Eye mark recorders - Motion capture devices - Biological sensors
Ethernet Wireless
Motion capture tag
Camera
Microphone
Conversation field
- time stamps
NTP server
[Sumi]
Database
IMADE: Interaction Measurement, Analysis, and Design Environment
[Yano 2011]
3D conversation capture
[Yano 2011]
3D conversation capture – from first person view
[Yano 2011]
3D conversation capture – from first person view
[Yano 2012]
3D conversation capture – over the shoulder
ICIE: Immersive Collaborative Interaction Environment
[Ohashi 2011]
Immersive Interaction Environment
[Nitschke 2013] https://www.youtube.com/watch?v=V-9SKpcMrzk
Projecting the real world into the virtual world
[Mori 2011]
[Mori 2012]
Virtualized physical world
ICIE: Immersive Collaborative Interaction Environment
Projecting operator’s behavior on a robot [Saiga 2012]
iDEAL: a software platform for ICIE [Ohmoto 2012]
ICIE: Immersive Collaborative Interaction Environment
Immersive WOZ environment
[Ohmoto, Ohashi, Saiga]
Feedback generation Motion mapping User motion sensing Head recognition Gesture recognition Face model Human body model
WOZ operating environment
WOZ operator Tele-operated robot
The conversation place
Immersive WOZ environment
The user in ICIE is asking for a clerk in the remote shop to explain a product.
[Saiga 2011] http://www.youtube.com/watch?v=YTVYwwZcGhU
ICIE: Immersive Collaborative Interaction Environment
[Ohmoto 2011]
Interactions from observation
Interaction Corpus
Observation/measurement of human-human
interactions
Realizing autonomous robot with estimated parameters
North conversation field shifts South
S1 S2
S3 S1
S2
S3 S1
S2
S3 S1
S2
S3
scene 1 scene 2 scene 3 scene 4
[Sumi, Bono, Kijima]
Analysis using annotation tool (iCorpusStudio)
[Yano and Sumi]
Conversation visualizer
Learning by imitation
[Mohammad 2009]
[Mohammad 2010]
Gesture stream
Action stream
The problem formulation
Learning by imitation
[Mohammad 2009]
Fluid Learning
[Mohammad 2013]
[Lala 2013]
Virtual Basketball
[Nakazawa Nitschke 2013]
Corneal Image Capture
1. This course centers on conversations. 2. Why conversations?
-> Foundation of thought and communication 3. Why do people converse with each other?
-> As a part of social interaction, for creating and maintaining stories, for fun.
4. The complexity of conversations: -> Multi-level, multi-layered, polysemy, polymorphism, coordination of multi-modality.
5. People are not only proficient in expressing ideas but also skillful in interpreting utterances, thereby they learn from each other.
6. Building a conversationally intelligent agents helps people effectively communicate with each other by conversations.
7. Conversational intelligence can be counted as an important branch of artificial intelligence.
8. We will integrate the engineering and scientific approached to conversation.
Summary
Agenda
Credits: Will be awarded based on one or more reports on subjects given at the class.
Calendar (tentative)
1. Introduction (October 2rd) 2. History of Conversational System Development (October 9th) 3. Smart space for conversation and the analysis (October 16th) 4. Mark Up Language (October 23rd) 5. Corpus Based Approach (October 30th) 6. Measuring, Analysis and Modeling (November 6th) 7. Affective Computing (November 13th) 8. Motif Discovery (November 20th) 9. Gesture (December 4th) 10. Computer Vision (December 11th) 11. Language Use - 1 (December 18th) 12. Language Use - 2 (December 25th) 13. Cognitive Design (January 8th) 14. Speaking Turn Taking System (January 22nd) 15. Cognitive Process (January 29th)
References
[Brown & Yule 1983] Gillian Brown and George Yule. Discourse Analysis, Cambridge University Press, 1983. [Clark 1996] Herbert H. Clark. Using Language. Cambridge, Cambridge University Press (1996) [Kendon 2004] Kendon, A.: Gesture, Cambridge University Press, 2004 [Nishida 2012a] Toyoaki Nishida: The Best of AI in Japan - Prologue. AI Magazine 33(2): 108-111, 2012
http://www.aaai.org/ojs/index.php/aimagazine/article/view/2358/2288 [Nishida 2012b] Toyoaki Nishida: Artificial intelligence research in the second half century, Journal of Information Processing and Management 55(7):
461-471, 2012 (in Japanese) http://dx.doi.org/10.1241/johokanri.55.461
[Nishida 2013a] Toyoaki Nishida: What’s AI, Journal of Japanese Society for Artificial Intelligence 28(2): 326-335, 2013 (in Japanese) [Nishida 2013b] Toyoaki Nishida: Conversation Quantization as a Foundation of Conversational Intelligence. DNIS 2013: 230-245, 2013.
http://dx.doi.org/10.1007/978-3-642-37134-9_18