understanding naturally conveyed explanations of device behavior michael oltmans and randall davis...
DESCRIPTION
Michael Oltmans Sketches Models We have a sketch of a device A simulation model can be generated from the sketch Life is good… or is it?TRANSCRIPT
![Page 1: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/1.jpg)
Understanding Naturally Conveyed Explanations of
Device Behavior
Michael Oltmans and Randall DavisMIT Artificial Intelligence Lab
![Page 2: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/2.jpg)
Michael Oltmans
Roadmap
The problem• Our approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
• Evaluation and contributions• Related and future work
![Page 3: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/3.jpg)
Michael Oltmans
Sketches Models
• We have a sketch of a device
• A simulation model can be generated from the sketch
• Life is good… or is it?
![Page 4: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/4.jpg)
Michael Oltmans
![Page 5: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/5.jpg)
Michael Oltmans
The Problem
• No representation of intended behavior• People talk and sketch but the computer
doesn’t understand
![Page 6: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/6.jpg)
Michael Oltmans
Task
• Understand descriptions of device behavior:– Given:
• A model of the device’s structure• A natural explanation of the behavior
– Generate a causal model of behavior
![Page 7: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/7.jpg)
Michael Oltmans
![Page 8: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/8.jpg)
Michael Oltmans
Roadmap
• The problemOur approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
• Evaluation and contributions• Related and future work
![Page 9: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/9.jpg)
Michael Oltmans
Naturally Conveyed Explanations
• Natural input modalities– Sketched devices– Sketched gestures– Speech
• Natural content of descriptions– Causal– Behavioral
![Page 10: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/10.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:
![Page 11: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/11.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
![Page 12: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/12.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
Qualitative Reasoner
(< (length spring) (rest-length spring))
![Page 13: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/13.jpg)
Michael Oltmans
Example: Describing the Behavior of a Spring
Tool: Description:Mechanical CAD
Spring length = 2.3cm Rest length = 3.0cm
Qualitative Reasoner
(< (length spring) (rest-length spring))
ASSISTANCE“The spring pushes the block”
![Page 14: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/14.jpg)
Michael Oltmans
Sources of power
• Conventions in explanations aide interpretation– Description order suggests causal order
• Constrained vocabulary• Overlapping descriptions provide
constraints on interpretations
![Page 15: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/15.jpg)
Michael Oltmans
Roadmap
• The problem• Our approach Implementation
System architectureHow ASSISTANCE interprets descriptionsDemonstrating understanding
• Evaluation and contributions• Related and future work
![Page 16: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/16.jpg)
Michael Oltmans
ASSIST•Recognize sketch
Sketch Speech
Causal Modeland Simulation
ViaVoice™•Recognize speech•Parse
LTRE•Truth Maintenance•Rule System
ASSISTANCE•Interpret explanation
![Page 17: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/17.jpg)
Michael Oltmans
Outputs
• Consistent causal model– Tree– Nodes are events– Links indicate causal relationships
• Demonstration of understanding– Natural language descriptions of causality– Parameter constraints
![Page 18: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/18.jpg)
Michael Oltmans
The Representation of Utterances
• Input comes from ViaVoice™ :– Grammar constructed based on observed
explanations– Tagged with parts of speech and semantic
categories
![Page 19: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/19.jpg)
Michael Oltmans
Representing the parse tree
SENTENCE SIMPLE_SENTENCE(… “body 1 pushes body 2” (S0) t1)
DIRECT_OBJECT NOUN NOUN-PHRASE(… “body 2” (S0 t1 t3) t5)
PROPELS VERB(… “pushes” (S0 t1 t3) t4)
SUBJECT NOUN NOUN-PHRASE(… “body 1” (S0 t1) t2)
VERB_PHRASE(… “pushes body 2” (S0 t1) t3)
“body 1 pushes body 2”
![Page 20: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/20.jpg)
Michael Oltmans
Steps In Interpreting Explanations:
• Infer motions from annotations and build event representations
• Find causal connections• Search for consistent causal
structures• Pick best causal structure
![Page 21: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/21.jpg)
Michael Oltmans
Step 1: Inferring Motions from Annotations
• Inputs:– Arrows– Utterances
• “moves,” “pushes,” “the spring releases”
• Outputs:– (moves body-1 moves-body-1-394)– (describes arrow-2 moves-body-1-394)
![Page 22: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/22.jpg)
Michael Oltmans
•Rule triggers:–Arrow–Arrow referent (i.e. a body)–The body is mobile
•Rule body records that:–The body moves–The arrow describes the path
Inferring Motion From Arrows
![Page 23: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/23.jpg)
Michael Oltmans
(rlet ((?id (new-id “Moves” ?name))) (rassert! (:implies (:AND ?f1 ?f2 ?f3) (:AND (moves ?body ?id) (describes ?arrow ?id)))
:ARROW-IS-MOTION)))
Inferring Motion From Arrows(rule ((:TRUE (arrow ?arrow) :VAR ?f1) (:TRUE (arrow-referent ?arrow ?body)
:VAR ?f2) (:TRUE (can-move ?body) :VAR ?f3) (:TRUE (name ?name ?body)))
![Page 24: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/24.jpg)
Michael Oltmans
Multi-Modal References
• Match a sentence whose subject is “this” and a pointing gesture
• Assert the referent as the subject of the sentence
• Limitations:– User must point at referent before the
utterance– Allow one “this” per utterance
![Page 25: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/25.jpg)
Michael Oltmans
Redundant Events
• Redundant explanations lead to multiple move statements for some events
• Merge them into a unique event statement
“Body 1” falls
(moves body-1 id-1)
(moves body-1 id-2)
Event 1
![Page 26: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/26.jpg)
Michael Oltmans
Step 2: Find Causal Connections
• Plausible causes– Arrow indicating motion near another
object– Exogenous forces
• Definite causes– “When … then …” utterances– “Body 1 pushes body 2”
![Page 27: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/27.jpg)
Michael Oltmans
Step 3: Search for Consistent Causal Structures
• Some events have several possible causes
• Find consistent causal chains• Search
– Forward looking depth-first-search– Avoids repeating bad choices by recording
bad combinations of assumptions
![Page 28: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/28.jpg)
Michael Oltmans
Step 4: Find the Best Interpretation
• Filter out interpretations that have unnecessary exogenous causes
• Pick the interpretation that most closely matches the explanation order
• While there are multiple valid interpretations– Choose one event with multiple possible causes– Assume the causal relation whose cause has the
earliest description time
![Page 29: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/29.jpg)
Michael Oltmans
Answer Queries and Adjust Parameters
• Queries: – Designer: What is body 2 involved in?– ASSISTANCE: The motion of body 3 causes
the motion of body 2 which causes the motion of body 5
• Parameter Adjustment– Set spring length
![Page 30: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/30.jpg)
Michael Oltmans
Roadmap
• The problem• Our approach• Implementation
– System architecture– How ASSISTANCE interprets descriptions– Demonstrating understanding
Evaluation and contributions• Related and future work
![Page 31: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/31.jpg)
Michael Oltmans
Limitations of the Implementation
• Scope of applicability restricted– State transitions are one step deep– Cannot handle conjunctions of causes
• Limited knowledge about common device patterns– Latches, linkages, etc…– Supports and prevents
• Natural language limitations– Use a full featured NL system like START– Formally determine the grammar
![Page 32: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/32.jpg)
Michael Oltmans
Evaluation of the Approach
• Advantages– Focus on behavior in accordance with survey
results– Move away from rigidity of WIMP interfaces– Similar to person-to-person interaction
• Alternatives– More dialog and feedback– Natural vs. efficient– Open claim that the domain is adequately
constrained
![Page 33: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/33.jpg)
Michael Oltmans
Contributions
• Understanding naturally conveyed descriptions of behavior
• Generating representations of device behavior– Match the designer’s explanation– Generate simple explanations of causality– Allow the calculation of simulation
parameters
![Page 34: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/34.jpg)
Michael Oltmans
Related Work• Understanding device sketches
– Alvarado 2000• Multimodal interfaces
– Oviatt and Cohen• Causality
– C. Rieger and M. Grinberg 1977
![Page 35: Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab](https://reader036.vdocuments.us/reader036/viewer/2022062909/5a4d1ba57f8b9ab0599c8dfc/html5/thumbnails/35.jpg)
Michael Oltmans
Future Work
• Direct manipulation• Dialog• Expand natural language capabilities• Smart design tools