1 simulation semantics: a framework to explore the links between language, cognition, and action...
TRANSCRIPT
1
Simulation Semantics: A Framework to explore the links between Language, Cognition, and Action
Jerome A. FeldmanICSI/UC Berkeley
ONR Cognitive Science & Human Robot Interaction6.1 and 6.2 Program Review
August 2015
2
OBJECTIVES
Perform and integrate basic research and converging evidence from cognitive linguistics, computation, and neural information processing to build cognitive models of language acquisition and use.
Provide an operational computational framework of action/simulation semantics to investigate the interaction between language, action, and cognition.
Action/Simulation semantics offers the possibility to build systems for computing with natural language that come close to human performance levels. This is necessary for joint action in complex naval scenarios with a mix of human and artificial agents.
ECG - NLU Beyond the 1980s
1. Much more computation
2. NLP technology
3. Construction Grammar: form-meaning pairs
Conceptual compositionality + Idioms, etc.
4. Cognitive Linguistics: Conceptual primitives, Metaphor, etc.
ECG = Embodied Construction Grammar; 6 uses of formalism
5. Constrained Best Fit : Analysis, Simulation, Learning
Analysis uses Bayesian (form, meaning and context) best fit
6. Under-specification: Meaning involves context, goals, etc.
SemSpec = Semantic/Simulation Specification
7. Simulation Semantics: Meaning as action/simulation
8. CPRM= Coordinated Probabilistic Relational Models; Petri Nets ++
Action formalism works as a generative model
9. Domain Semantics; Need rich semantics on the Action side
10. General NLU front end: Modest effort to link to a new Action side
Slide 3
Active representations
• Many inferences about actions derive from what we know about executing them
• X-net representation based on stochastic Petri nets captures dynamic, parameterized nature of actions
• Used for acting, recognition, planning, and language
Walking: bound to a specific walker with a direction or goal
consumes resources (e.g., energy)may have termination condition
(e.g., walker at goal) ongoing, iterative actionwalker=Harry
goal=home
energy
walker at goal
How do we specify an event? Formalized event schema
• Key elements– preconditions, resources, effects, sub-events– evoked by frames (alternatively: predicates, words)
• Contrast with Event Recognition/Extraction, other NLP work
5
ISA
hasFrame hasParameter
construedAs
composedBy
EVENT
COMPOSITE EVENT
FRAMEActorThemeInstrumentPatient
CONSTRUALPhase (enable, start, finish, ongoing, cancel)Manner (scales, rate, path) Zoom (expand, collapse)
RELATION(E1,E2)SubeventEnable/DisableSuspend/ResumeAbort/Terminate Cancel/StopMutually ExclusiveCoordinate/Synch
EventRelation
CONSTRUCTSequenceConcurrent/Conc. SyncChoose/AlternativeIterate/RepeatUntil(while)If-then-Else/Conditional
PARAMETERPreconditionsEffectsResources - In, OutInputsOutputsDurationGrounding
Time, Location
AccomplishmentsAction Language Understanding System
• Demonstrate utility through a series of scalable prototypes – that show the ability of the system to handle increasingly complex
language– in a general way across multiple tasks and environments – to support communication in communities comprised of both
human and artificial agents
• Current Goal: Implement a prototype system that can follow instructions and synthesize actions and procedures expressed in natural language. – This requires the system to analyze natural language and
translate this language in context into a coordinated network of actions and complex commands.
Slide 6
Integrated Pilot System for Action Synthesis
7
DiscourseAnalyzer ECG Grammar
SemSpecSpecializer
ApplicationProblem Solver
API ~ Morse etc.
Actions
CompiledN-Tuples
World
Situation
Ontology
Robot1, move North!
Videos
8
9
ISSUES
• The 2013 PI, Professor Srinivas Narayanan, joined Google in Zurich as of February 1, 2014.
• Professor Jerome Feldman, who was Co-PI, has become the PI. Prof. Narayanan continues to advise on the project and there has been no significant disruption.
• In early 2015, the companion IARPA program was abruptly cancelled, causing significant curtailment of the metaphor work, but not the core ONR mission.
10
CONCLUSIONS & NAVAL RELEVANCE
• The main conclusion is that it is now feasible to build practical LCAS (Language Communication with Autonomous Systems) realizations. ICSI has received a provisionary patent on this technology.
• Our NLU work is deeply interdisciplinary and is having effects on several fields.
• One explicit Naval requirement is an improved ability to communicate with robots and other complex equipment using natural language. We have made significant progress in this area and propose to directly address Navy tasks.
11
CURRENT AND FUTURE WORK
We will continue to develop both the theory of action/simulation semantics and its application to Naval mission-relevant LCAS. Our actionability theory is being discussed in a wide range of forums and is potentially disruptive.
We will expand our general working LCAS interface to complex systems and the specific applications to multi-agent robot control and spoken language integration.
One main effort moving forward is to expand the full system to industrial scale. Analysis and staffing for this challenge has begun.
12
RECENT PUBLICATIONS, PATENTS, PRESENTATIONS, & AWARDS
• The team had five papers accepted at the International Cognitive Linguistics Conference held in June 2013.
The 2013 PI, Professor Narayanan and collaborators won the Semantic Web Science Association’s ten year highest impact paper award. Prof. Narayanan, was the keynote speaker at the First Workshop on Metaphor in NLP at NAACL in June 2103
PI Prof. Jerome Feldman, Invited seminar talk, Redwood Center for Theoretical Neuroscience, UC Berkeley.Jerome Feldman gave invited talks on our Actionability theory early in 2014 at: The Conference on Theory of Mind, The Soft Computing Round Table, and at the UC Irvine Cognitive Science seminar.
CoPI Jerome Feldman and PI Srini Narayanan organized a workshop on Actionability held at ICSI Berkeley in 2013.
Katia Shutova organized a workshop on metaphor in NLP at NAACL in 2013 and again in 2014.
13
RECENT PUBLICATIONS,
Srini Narayanan (2013). “A neural computational model of metaphor learning using spiking networks and STDP”, Neural Computation (accepted)
Dodge, Ellen. “A Compositional Constructional Analysis of ‘Hitting’ Verb Argument Realization Patterns and Their Meanings”. Talk presented at the International
Cognitive Linguistics Conference (ICLC) 12, Edmonton, AB. June 27, 2013. Srini Narayanan (2013). A Cognitive Model of Scales. Proceedings of the
International Conference Cognitive Linguistics Conference, June 2013.
Dodge, Ellen K.; Petruck, Miriam R. L. “Representing Caused Motion in Embodied Construction Grammar”. Association for Computational Linguistics, Baltimore, MD. June 26, 2014
Sweetser, E. (with Dancygier, B.): Figurative Language Cambridge U. Press 2014
Feldman, Jerome “Affordances, Actionability, and Simulation” Affordances in Vision for Cognitive Robotics, Berkeley, July 2014
14
TRANSITION PLAN
Our applied efforts in semantics –based Natural Language Understanding (NLU) have reached a stage where transition to military and/or commercial use is feasible.
It is possible that the current LCAS pilot system could be used for focused development of Naval systems and we would be happy to assist in such efforts. We are actively working with ONR staff to identify Navy and performer partners.
For wide spread military and/or commercial use, we need to scale up and harden the current system and also expand to spoken language and multi-agent tasks. We have begun this effort in cooperation with other research and industrial groups .
Simulation SemanticsJerome Feldman ICSI/UC Berkeley
Award Period 01/2011 – 01/2016Project Objectives:
Technical Approach: Accomplishments/Impact/Transitions: ECG Workbench and Grammar Analyzer Computational Model of Simulation
Semantics Extension to metaphor, other languages Integrated System - Language control of
mobile robots, etc. - LCAS LCAS Provisional Patent granted in 2015 Transition to Spoken Language Many publications, invited talks, workshops Collaborations with government, industry,
academics
Develop and test cognitive models of language use.Operational framework of action/simulation semantics LCAS- Language Communication with Autonomous Systems
Action/Simulation uses probabilistic best-fit a) Coordinated Probabilistic Relational Models (CPRM) b) Embodied Construction Grammar Message-based LCAS applicationsDemonstrate utility with a series of scalable prototypes Build community based on action/simulation semantics
Situation model
Problem Solver
n-tuplesSpecializer
Task API (MORSE simulator)
SemSpec
Analyzer(ECG)
TEXT
USERObservations
Ontology
Simulated
World
Extras
16
Computing with Natural Language
• An integrated system combining –Deep semantic analysis of language in context with–A scalable simulation model
• Best-fit Language Analyzer– Embodied Construction Grammar (ECG)
• Construction Parser – John Bryant PhD Thesis 2008– Eva Mok PhD Thesis 2009– Ellen Dodge PhD Thesis 2010
• Scalable Domain Representation– Event Models
– Steve Sinha PhD Thesis 2008– Joe Makin PhD Thesis 2008
– Coordinated Probabilistic Relational Models – Leon Barrett PhD Thesis 2010– Steve Doubleday Thesis 2015
Embodied Construction Grammar: ECG
• ECG serves:1. as a technical tool for linguistic analysis
2. to specify shared grammar, conceptual conventions of a linguistic community
3. as a computer specification for implementing linguistic theories
4. as a representation for models and theories of language acquisition
5. as a front-end system for applied language-understanding tasks
6. as a high-level functional description for biological and behavioral experiments
ECG Workbench
ECG Workbench:● Based on Eclipse● Takes advantage of and fully integrates with
Eclipse RCP (Rich Client Platform)● Makes it easy to enter, edit and check
consistency of ECG grammars● Can analyze text licensed by the grammar,
producing a SemSpec (Semantic Specification)
● Download: http://www.icsi.berkeley.edu/~lucag
Additional ExamplesSimple Commands:Robot1, move North!Robot1, move to the big red box!Robot1, move behind a red box!Robot1, push the blue box North!
Serial Processes:Robot1, move to the blue box then return!Robot1, move to the big red box then move the small red box North!Robot1, move to the green box then push another one East!
Conditionals + Referent Resolution:Robot1, if the green box is near the small red box, move to it!Robot1, if the blue box is near the small red box, move to the green box then push it!
Questions:which boxes are near the green box?which boxes are red?is the box near the blue box green?where is the green box?
Definitions:define tour QL1 and QL2 and QL3 as move to QL1 then move to QL2 then move to QL3.Robot1, tour Box1 and Box2 and Box3!
Same Examples in SpanishSimple Commands:Robot1, muévete al norte!Robot1, muévete detrás de una caja roja!Robot1, empuje la caja azul al norte!
Serial Processes:Robot1, muévete a la caja azul y regresa!Robot1, muévete a la caja roja y grande y muévete a la caja roja y pequeña!
Conditionals + Referent Resolution:Robot1, si la caja verde está cerca de la caja roja y pequeña, la empuje al norte!Robot1, si la caja pequeña es roja, muévete a la caja verde y la empuje!
Questions:es la caja cerca de la caja azul verde?está la caja verde cerca de la caja roja y pequeña?cuáles cajas son rojas? cuáles cajas están cerca de la caja roja y pequeña?dónde está la caja verde?
Robot1, move North then return!
Robot1, move North then return!
Analyzer
Robot1, move North then return!
Integrated Pilot System for Action Synthesis
25
DiscourseAnalyzer ECG Grammar
SemSpecSpecializer
ApplicationProblem Solver
API ~ Morse etc.
Actions
CompiledN-Tuples
World
Situation
Ontology
Robot1, move North!
Parse serial processes
Initialize executable template
Assign values in SemSpec to template
Specializer
Parameters of message
First part of N-tuple received by Solver; contains information for first part of serial process (“move North”)
Second part of N-tuple received by Solver; contains information for second part of serial process (“return”)
Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading='north', goal=None)
N-Tuple
[Struct(protagonist='robot1_instance', heading='north', goal=None, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing'),
Struct(protagonist='robot1_instance', heading=None, goal={'location': 'home'}, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing')]
Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading=None, goal={'location': 'home'})
Solver
Unpacks N-tuple passed from Specializer.
Determines heading of movement vector (“move North”).
Returns to “home” position, stored in prior state variable.
Morse (Robot Simulator)
Moves robot
Gets the new state of the world
Updates internal model of the world
Gets the updated state of the world
Moves the robot in simulation (and waits for it to arrive)