understanding naturally conveyed explanations of device behavior
DESCRIPTION
Understanding Naturally Conveyed Explanations of Device Behavior. Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab. Roadmap. The problem Our approach Implementation System architecture How ASSISTANCE interprets descriptions Demonstrating understanding - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/1.jpg)
Understanding Naturally Conveyed Explanations of
Device Behavior
Michael Oltmans and Randall Davis
MIT Artificial Intelligence Lab
![Page 2: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/2.jpg)
Michael Oltmans
Roadmap
The problem• Our approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
• Evaluation and contributions• Related and future work
![Page 3: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/3.jpg)
Michael Oltmans
Sketches Models
• We have a sketch of a device
• A simulation model can be generated from the sketch
• Life is good… or is it?
![Page 4: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/4.jpg)
Michael Oltmans
![Page 5: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/5.jpg)
Michael Oltmans
The Problem
• No representation of intended behavior
• People talk and sketch but the computer doesn’t understand
![Page 6: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/6.jpg)
Michael Oltmans
Task
• Understand descriptions of device behavior:– Given:
• A model of the device’s structure• A natural explanation of the behavior
– Generate a causal model of behavior
![Page 7: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/7.jpg)
Michael Oltmans
![Page 8: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/8.jpg)
Michael Oltmans
Roadmap
• The problemOur approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
• Evaluation and contributions• Related and future work
![Page 9: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/9.jpg)
Michael Oltmans
Naturally Conveyed Explanations
• Natural input modalities– Sketched devices– Sketched gestures– Speech
• Natural content of descriptions– Causal– Behavioral
![Page 10: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/10.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:
![Page 11: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/11.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:
Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
![Page 12: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/12.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:
Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
Qualitative Reasoner
(< (length spring) (rest-length spring))
![Page 13: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/13.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:
Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
Qualitative Reasoner
(< (length spring) (rest-length spring))
ASSISTANCE“The spring pushes the block”
![Page 14: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/14.jpg)
Michael Oltmans
Sources of power
• Conventions in explanations aide interpretation– Description order suggests causal order
• Constrained vocabulary
• Overlapping descriptions provide constraints on interpretations
![Page 15: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/15.jpg)
Michael Oltmans
Roadmap
• The problem• Our approach Implementation
System architectureHow ASSISTANCE interprets descriptionsDemonstrating understanding
• Evaluation and contributions• Related and future work
![Page 16: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/16.jpg)
Michael Oltmans
ASSIST
•Recognize sketch
ASSIST
•Recognize sketch
Sketch Speech
Causal Model
and Simulation
Causal Model
and Simulation
ViaVoice™•Recognize speech•Parse
ViaVoice™•Recognize speech•Parse
LTRE•Truth Maintenance•Rule System
LTRE•Truth Maintenance•Rule System
ASSISTANCE
•Interpret explanation
![Page 17: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/17.jpg)
Michael Oltmans
Outputs
• Consistent causal model– Tree– Nodes are events– Links indicate causal relationships
• Demonstration of understanding– Natural language descriptions of causality– Parameter constraints
![Page 18: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/18.jpg)
Michael Oltmans
The Representation of Utterances
• Input comes from ViaVoice™ :– Grammar constructed based on observed
explanations– Tagged with parts of speech and semantic
categories
![Page 19: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/19.jpg)
Michael Oltmans
Representing the parse tree
SENTENCE SIMPLE_SENTENCE
(… “body 1 pushes body 2” (S0) t1)
SENTENCE SIMPLE_SENTENCE
(… “body 1 pushes body 2” (S0) t1)
DIRECT_OBJECT NOUN NOUN-PHRASE
(… “body 2” (S0 t1 t3) t5)
DIRECT_OBJECT NOUN NOUN-PHRASE
(… “body 2” (S0 t1 t3) t5)PROPELS VERB
(… “pushes” (S0 t1 t3) t4)
PROPELS VERB
(… “pushes” (S0 t1 t3) t4)
SUBJECT NOUN NOUN-PHRASE
(… “body 1” (S0 t1) t2)
SUBJECT NOUN NOUN-PHRASE
(… “body 1” (S0 t1) t2)VERB_PHRASE
(… “pushes body 2” (S0 t1) t3)
VERB_PHRASE
(… “pushes body 2” (S0 t1) t3)
“body 1 pushes body 2”
![Page 20: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/20.jpg)
Michael Oltmans
Steps In Interpreting Explanations:
• Infer motions from annotations and build event representations
• Find causal connections
• Search for consistent causal structures
• Pick best causal structure
![Page 21: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/21.jpg)
Michael Oltmans
Step 1: Inferring Motions from Annotations
• Inputs:– Arrows– Utterances
• “moves,” “pushes,” “the spring releases”
• Outputs:– (moves body-1 moves-body-1-394)– (describes arrow-2 moves-body-1-394)
![Page 22: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/22.jpg)
Michael Oltmans
•Rule triggers:–Arrow–Arrow referent (i.e. a body)–The body is mobile
•Rule body records that:–The body moves–The arrow describes the path
Inferring Motion From Arrows
![Page 23: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/23.jpg)
Michael Oltmans
(rlet ((?id (new-id “Moves” ?name))) (rassert! (:implies (:AND ?f1 ?f2 ?f3) (:AND (moves ?body ?id) (describes ?arrow ?id)))
:ARROW-IS-MOTION)))
Inferring Motion From Arrows(rule ((:TRUE (arrow ?arrow) :VAR ?f1) (:TRUE (arrow-referent ?arrow ?body)
:VAR ?f2) (:TRUE (can-move ?body) :VAR ?f3) (:TRUE (name ?name ?body)))
![Page 24: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/24.jpg)
Michael Oltmans
Multi-Modal References
• Match a sentence whose subject is “this” and a pointing gesture
• Assert the referent as the subject of the sentence
• Limitations:– User must point at referent before the
utterance– Allow one “this” per utterance
![Page 25: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/25.jpg)
Michael Oltmans
Redundant Events
• Redundant explanations lead to multiple move statements for some events
• Merge them into a unique event statement
“Body 1” falls
(moves body-1 id-1)
(moves body-1 id-2)
Event 1
![Page 26: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/26.jpg)
Michael Oltmans
Step 2: Find Causal Connections
• Plausible causes– Arrow indicating motion near another
object– Exogenous forces
• Definite causes– “When … then …” utterances– “Body 1 pushes body 2”
![Page 27: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/27.jpg)
Michael Oltmans
Step 3: Search for Consistent Causal Structures
• Some events have several possible causes
• Find consistent causal chains
• Search– Forward looking depth-first-search– Avoids repeating bad choices by recording
bad combinations of assumptions
![Page 28: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/28.jpg)
Michael Oltmans
Step 4: Find the Best Interpretation
• Filter out interpretations that have unnecessary exogenous causes
• Pick the interpretation that most closely matches the explanation order
• While there are multiple valid interpretations– Choose one event with multiple possible causes– Assume the causal relation whose cause has the
earliest description time
![Page 29: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/29.jpg)
Michael Oltmans
Answer Queries and Adjust Parameters
• Queries: – Designer: What is body 2 involved in?– ASSISTANCE: The motion of body 3 causes
the motion of body 2 which causes the motion of body 5
• Parameter Adjustment– Set spring length
![Page 30: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/30.jpg)
Michael Oltmans
Roadmap
• The problem• Our approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
Evaluation and contributions• Related and future work
![Page 31: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/31.jpg)
Michael Oltmans
Limitations of the Implementation
• Scope of applicability restricted– State transitions are one step deep– Cannot handle conjunctions of causes
• Limited knowledge about common device patterns– Latches, linkages, etc…– Supports and prevents
• Natural language limitations– Use a full featured NL system like START– Formally determine the grammar
![Page 32: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/32.jpg)
Michael Oltmans
Evaluation of the Approach
• Advantages– Focus on behavior in accordance with survey
results– Move away from rigidity of WIMP interfaces– Similar to person-to-person interaction
• Alternatives– More dialog and feedback– Natural vs. efficient– Open claim that the domain is adequately
constrained
![Page 33: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/33.jpg)
Michael Oltmans
Contributions
• Understanding naturally conveyed descriptions of behavior
• Generating representations of device behavior– Match the designer’s explanation– Generate simple explanations of causality– Allow the calculation of simulation
parameters
![Page 34: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/34.jpg)
Michael Oltmans
Related Work• Understanding device sketches
– Alvarado 2000
• Multimodal interfaces– Oviatt and Cohen
• Causality– C. Rieger and M. Grinberg 1977
![Page 35: Understanding Naturally Conveyed Explanations of Device Behavior](https://reader035.vdocuments.us/reader035/viewer/2022062422/56813fef550346895daafbf7/html5/thumbnails/35.jpg)
Michael Oltmans
Future Work
• Direct manipulation
• Dialog
• Expand natural language capabilities
• Smart design tools