![Page 1: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/1.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Technical Cognitive Systems
Michael Beetz
Intelligent Autonomous SystemsTechnische Universitat Munchen
Summer Term 2012
![Page 2: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/2.jpg)
Part X
Reasoning
![Page 3: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/3.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Outline
Motion Planning
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems204
![Page 4: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/4.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Outline
Motion Planning
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems205
![Page 5: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/5.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Making pancakes from web instructions
From web-instructions:
• Mix flour, sugar, egg and milk together in a largebowl
• Pour mixture into frying pan
• Flip your pancake using the long handled spatula
Challenge: how to infer missing knowledge?
• underspecified actions like mixing, pouring, andflipping?
• where to stand?
• where to put objects?
• . . .
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems206
![Page 6: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/6.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Outline
Motion Planning
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems207
![Page 7: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/7.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Cracking an egg
“A cook is cracking a raw egg against a glass bowl. Properly performed, theimpact of the egg against the edge of the bowl will crack the eggshell in half.Holding the egg over the bowl, the cook will then separate the two halves ofthe shell with his fingers, enlarging the crack, and the contents of the eggwill fall gently into the bowl. The end result is that the entire contents ofthe egg will be in the bowl, with the yolk unbroken, and that the two halvesof the shell are held in the cook’s fingers.” (Common Sense Problem Page)
Variants: What happens if. . .
• the cook brings the egg to impact very quickly? Very slowly?
• the bowl is made of looseleaf paper? of soft clay?
• the bowl is smaller than the egg? bowl is upside down?
• the cook tries this procedure with a hard-boiled egg? a coconut?
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems208
![Page 8: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/8.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Cracking an egg
“A cook is cracking a raw egg against a glass bowl. Properly performed, theimpact of the egg against the edge of the bowl will crack the eggshell in half.Holding the egg over the bowl, the cook will then separate the two halves ofthe shell with his fingers, enlarging the crack, and the contents of the eggwill fall gently into the bowl. The end result is that the entire contents ofthe egg will be in the bowl, with the yolk unbroken, and that the two halvesof the shell are held in the cook’s fingers.” (Common Sense Problem Page)
Variants: What happens if. . .
• the cook brings the egg to impact very quickly? Very slowly?
• the bowl is made of looseleaf paper? of soft clay?
• the bowl is smaller than the egg? bowl is upside down?
• the cook tries this procedure with a hard-boiled egg? a coconut?
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems209
![Page 9: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/9.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems210
![Page 10: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/10.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems211
![Page 11: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/11.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
Theory is based on the event calculus (Kowalski 86)
courtesy of (Shanahan 99)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems212
![Page 12: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/12.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
Axiom ECF1 (InitiallyP(f) ∧¬Clipped(0,f,t2)) ∨(Happens(a,t1) ∧Initiates(a,f,t1) ∧t1 ¡t2 ∧¬Clipped(t1,f,t2)) ⇒Holds(f,t2)Axiom ECF2 (InitiallyN(f) ∧¬Declipped(0,f,t2)) ∨(Happens(a,t1) ∧Terminates(a,f,t1)∧t1 ¡ t2 ∧¬Declipped(t1,f,t2)) ⇒¬Holds(f,t2)Axiom ECF3 Clipped(t1,f,t2)⇔∃a,t (Happens(a,t) ∧(Terminates(a,f,t) ∨Releases(a,f,t))∧t1 ¡ t ∧t ¡ t2)Axiom ECF4 Declipped(t1,f,t2)⇔∃a,t (Happens(a,t) ∧(Initiates(a,f,t) ∨Releases(a,f,t))∧t1 ¡ t ∧t ¡ t2)
Liquid and Solid Objects, Materials:Axiom 1 Holds(Solid(o), t) ∨ Holds(Liquid(o), t)Axiom 2 (Mutual Exclusivity of Material Properties) Hard(m) ∨Soft(m) ∨ Fragile(m)∨Hard-to-Break(m) ∨Unbreakable(m)Axiom 3 Unbreakable(m1) ∧Hard-to-Break(m2) ⇒ HardertoBreak(m1,m2)Hard-to-Break(m1) ∧ Fragile(m2) ⇒ HardertoBreak(m1,m2)Axiom 4 HardertoBreak(m1,m2) ∧ HardertoBreak(m2,m3) ⇒ HardertoBreak(m1,m3)Axiom 5 Glass(m) ⇒ Hard-to-Break(m)Axiom 6 Glass(m) ⇒ Hard(m)
Containers:Definition 1 ClosedContainerShape(r) is true iff r is an ordinary point set and thecomplement of r has exactly two connected components.Definition 2 insideof(r) is the closure of the finite component of the complement of r.Definition 3 outsideof(r) is the closure of the infinite component of the complement of r.Definition 4 Holds(ClosedContainer(c),t) ⇔(Holds(shape(c)=r,t) ∧ ClosedContainer-Shape(r))
. . .
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems213
![Page 13: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/13.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
In total, the theory includes > 60 axioms, > 20 definitions and lemmas.
Example:
Axiom 48: Holds(HardertoBreak(MaterialOf(o2),MaterialOf(o1)),t) ∧Holds(Hard(MaterialOf(o2)),t) ∧ Holds(Hard(MaterialOf(o1)),t) ∧Holds(ObjPart(pt1,o1),t) ∧ Holds(ObjPart(pt2,o2),t) ⇒Initiates(Hit(pt1,pt2,crackforce(o1,pt1,pt2)),CrackedContainer(o1),t)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems214
![Page 14: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/14.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
• Simplifications:
– ignore role and position of hands– shape of the crack is ignored– a container does not separate into two (or more) fragments– only 180-degree rotations are considered– . . .
• Variants the theory cannot handle:
– cooks brings the egg to impact very quickly/slow– bowl is upside down– cook uses a hard-boiled-egg, a coconut, or M&M– cook puts egg in bowl and exerts steady pressure with his hand– . . .
Important for robot manipulation: abstraction level, variants,non-deterministic and concurrent actions (not isolated problems)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems215
![Page 15: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/15.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
• Simplifications:
– ignore role and position of hands– shape of the crack is ignored– a container does not separate into two (or more) fragments– only 180-degree rotations are considered– . . .
• Variants the theory cannot handle:
– cooks brings the egg to impact very quickly/slow– bowl is upside down– cook uses a hard-boiled-egg, a coconut, or M&M– cook puts egg in bowl and exerts steady pressure with his hand– . . .
Important for robot manipulation: abstraction level, variants,non-deterministic and concurrent actions (not isolated problems)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems216
![Page 16: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/16.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Logical Axiomatization by (Morgenstern 2001)
• Simplifications:
– ignore role and position of hands– shape of the crack is ignored– a container does not separate into two (or more) fragments– only 180-degree rotations are considered– . . .
• Variants the theory cannot handle:
– cooks brings the egg to impact very quickly/slow– bowl is upside down– cook uses a hard-boiled-egg, a coconut, or M&M– cook puts egg in bowl and exerts steady pressure with his hand– . . .
Important for robot manipulation: abstraction level, variants,non-deterministic and concurrent actions (not isolated problems)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems217
![Page 17: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/17.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Animation Movies / Computer Games
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems218
![Page 18: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/18.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Physics Engines (ODE, Bullet, . . . )
Physics engines don’t have problems with:
• abstraction level (fine level of granularity)
• variants (alternative configurations)
• non-deterministic actions (noise)
• concurrent actions
Example: Dropping an Egg from 0.5 meters
Falling egg Hitting ground Breaking of joints Broken egg
time
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems219
![Page 19: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/19.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Outline
Motion Planning
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems220
![Page 20: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/20.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Robots Learning from InstructionsMaking a pancake
From the web-instructions:
• Mix flour, sugar, egg and milktogether in a large bowl
• Pour mixture into frying pan
• Flip your pancake using the longhandled spatula
Challenge:
How to infer missing knowledge forunderspecified actions like mixing,pouring, and flipping?
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems221
![Page 21: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/21.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Robots Learning from InstructionsMaking a pancake
From the web-instructions:
• Mix flour, sugar, egg and milktogether in a large bowl
• Pour mixture into frying pan
• Flip your pancake using the longhandled spatula
Challenge:
How to infer missing knowledge forunderspecified actions like mixing,pouring, and flipping?
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems222
![Page 22: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/22.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Bridging the Knowledge Gapbetween abstract instructions and robot control parameters
Learning from observations:
• Problem: almost only successful and goodperformances
; How can we derive flexible, robust, and failure-aware robot controlprograms?
Claim
Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effects
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems223
![Page 23: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/23.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Bridging the Knowledge Gapbetween abstract instructions and robot control parameters
Learning from observations:
• Problem: almost only successful and goodperformances
; How can we derive flexible, robust, and failure-aware robot controlprograms?
Claim
Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effects
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems224
![Page 24: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/24.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Bridging the Knowledge Gapbetween abstract instructions and robot control parameters
Learning from observations:
• Problem: almost only successful and goodperformances
; How can we derive flexible, robust, and failure-aware robot controlprograms?
Claim
Robots need an understanding of Naive Physics in order to choose controlparameters that lead to desired effectsMotion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems225
![Page 25: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/25.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Proposed ApproachBasic idea
worldsituation
simulationperceptionlogging
logicalaxiomatization
axiomatization+ timelines
proof
worldsituation′
translatesparametrizes
represents/formalizes
naive physics inference(including mental
simulation)
answers
deduces
???
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems226
![Page 26: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/26.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Framework Overview
parametrizes
translates Logged Simulations
monitors
Augmented Simulation
breaking
detailed 3D models
+controllers
+monitors
rigid body dynamicsfor everyday objects
absorbing
Simulation (partial)
PROLOG
Timeline
Representations
Domain
Knowledge
Description LogicOWL
Fact-basePROLOG
First-order Representations Subsymbolic Representations
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems227
![Page 27: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/27.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Domain KnowledgePhysical aspects about Eggs
Egg
Liquid Solid Container
EggWhiteYolk EggShell
Viscosity Mass Extensions Rigidity Fragility Shape
hasPart hasPart
isType isType isType
hasProp hasProp hasProp hasProp hasProp hasProp hasProp
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems228
![Page 28: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/28.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Physics-based SimulationModelling Shape and Properties of an Egg
Body id: #42x: -0.082 y: 0.297 z: 0.265type: {solid, eggshell} mass: 0.1
Joint id: #88body1: #42 body2: #18x: -0.123 y: 0.322 z: 0.199roll: 0.0 pitch: -54.092 yaw: -31.626type: {solid, eggshell} forcelimit: 150 broken: false
Body id: #71x: -0.053 y: 0.305 z: 0.082type: {liquid, egg-yolk} mass: 0.1
Object modelling similar to work by Johnston and Williams [?]
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems229
![Page 29: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/29.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Augmenting Simulationwith Everyday Phenomena
We need more than rigid body dynamics for:
• breaking
• absorbing liquids
• melting
Augmenting object descriptions with:
• graph-based shape models
• controllers that compute physical effects
– perform computation in each simulation step– e.g.: breaking, absorbing, etc
• monitors that log physical properties:
– position, velocity, acceleration, forces, contact points, etc
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems230
![Page 30: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/30.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Translating Logged Simulationsinto Timelines
Fluents and events at points in time / time intervals
• Holds(f,t) / Holds-tt(f,[t1,t2])
• Occurs(ev,t) / Occurs-tt(f,[t1,t2])
How predicates are grounded in simulations:
• Holds(supports(o1,o2),t) ⇔Holds(Contacts(o1,o2), t) ∧p1 = SimulatorValueAt(Pose(o1),t) ∧p2 = SimulatorValueAt(Pose(o2),t) ∧isDirectlyBelow(p1,p2) ∧gravityForceIsCancelledOut(o2)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems231
![Page 31: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/31.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Predicates Grounded in Simulations
Static configurations:
fluent intuitive descriptioncontacts(o1, o2) object o1 and object o2 contact each otherattached(o1, o2) object o2 is attached to object o1
supports(o1, o2) object o1 supports object o2
contains(o1, o2) container o1 contains object (or stuff) o2
broken(o1) object o1 is brokenspilled(o1) object o1 is spilled
Events:
events intuitive descriptioncolliding(o1, o2) object o1 and object o2 are collidingfalling(o1) object o1 is fallingmoving(o1) object o1 is movingopeningGripper(o1) robot is opening gripper o1
closingGripper(o1) robot is closing gripper o1
breaking(o1) object o1 is breakingspilling(o1) object o1 is spilling over a surface
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems232
![Page 32: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/32.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Language
Asserting a scenario:
?- create scenario(egg scenario),
add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems233
![Page 33: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/33.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Language
Asserting a scenario:
?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),
add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems234
![Page 34: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/34.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Language
Asserting a scenario:
?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),
add to scenario(egg scenario,egg1).
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems235
![Page 35: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/35.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Language
Asserting a scenario:
?- create scenario(egg scenario),add to scenario(egg scenario,kitchen env),add to scenario(egg scenario,pr2 robot),add to scenario(egg scenario,egg1).
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems236
![Page 36: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/36.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Language (cont’d)
Making temporal projections:
?- make projection(egg scenario,grasp obj,[(P1, force, [1, 3, 5, 7, 9, 11]),(P2, position, [-0.02, 0.00, 0.02])],
Timeline),not(occurs(break, Time, Timeline)),not(occurs(slip, Time, Timeline)).
P1 = 5,P2 = 0.00
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems237
![Page 37: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/37.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal Projection Process
• make projection:sample parameters
• simulate:setup simulatorrun simulation
• translate:ground predicates inlogged simulations
• evaluate:events/fluentsspecialized predicates
make_projection(scenario,pogram,params,Timeline)
occurs(event,Time,Timeline)
egg slips
away
egg is hold
in gripper
egg is
broken
simulate(scenario,program,params,Log)
translate(Log,Timeline)
occurs(moving(egg), 12)
occurs(moving(egg), 13).
occurs(moving(egg), 14).
occurs(falling(egg), 15).
occurs(falling(egg), 16).
occurs(moving(egg), 17).
occurs(moving(egg), 18).
occurs(moving(egg), 19)..
occurs(moving(egg), 20).
occurs(moving(egg), 21).
occurs(moving(egg), 22).
occurs(moving(egg), 12)
occurs(moving(egg), 13).
occurs(moving(egg), 14).
occurs(falling(egg), 15).
occurs(falling(egg), 16).
occurs(moving(egg), 17).
occurs(moving(egg), 18).
occurs(moving(egg), 19)..
occurs(moving(egg), 20).
occurs(moving(egg), 21).
occurs(moving(egg), 22).
occurs(moving(egg), 12)
occurs(moving(egg), 13).
occurs(moving(egg), 14).
occurs(falling(egg), 15).
occurs(falling(egg), 16).
occurs(moving(egg), 17).
occurs(moving(egg), 18).
occurs(moving(egg), 19)..
occurs(moving(egg), 20).
occurs(moving(egg), 21).
occurs(moving(egg), 22).
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems238
![Page 38: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/38.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Pouring pancake mix
• Parameters: position, time, angle• Outcomes: number of particles on pan (spilled on table)
• Specialized predicates on particle sets: round/centered
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems239
![Page 39: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/39.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Flipping a pancake
• Parameters: angle of spatula• Outcomes: turned, not turned
• Common failures:
break push off fold stick onMotion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems240
![Page 40: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/40.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Summary
We propose simulation-based temporal projection:
worldsituation
simulationperceptionlogging
logicalaxiomatization
axiomatization+ timelines
proof
worldsituation′
translatesparametrizes
represents/formalizes
naive physics inference(including mental
simulation)
answers
deduces
• Framework for making predictions about manipulation actions
• Temporal projection language, e.g. make projection
• Extended rigid-body simulation with everyday phenomena
• Grounding predicates in data structures of simulation
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems241
![Page 41: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/41.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Outline
Motion Planning
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems242
![Page 42: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/42.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Motivation: Put the pancake mix on the counter
Whereshould Iput it???
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems243
![Page 43: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/43.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Motivation
• Infer plan parameters at run-time.
• Integrate state of the current world.
• Don’t abstract away from important geometric properties.
• Generation of parameters under geometric constraints vs. verification.
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems244
![Page 44: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/44.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan parameterizations: Designators
• Symbolic descriptions of Objects, Locations and Actions
• Examples:
– (a location (on table) (reachable-by Rosie))– (an object (type cup))– (an action (to reach) (the object (name Cup1)))
• Example plan:
(with-designators
((cup-loc (a location (on counter-top)))
(cup (an object (type cup) (at cup-loc)))
(pick-up-loc (a location (to reach) (obj cup))))
(at-location pick-up-loc
(achieve (obj-grasped cup))))
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems245
![Page 45: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/45.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Location to put-down pancake mix
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
Concepts
• Generative model
• Stability reasoning
• Reachability reasoning
• Visibility reasoning and perspective taking
• Light-weight temporal projection
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems246
![Page 46: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/46.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Location to put-down pancake mix
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
Concepts
• Generative model
• Stability reasoning
• Reachability reasoning
• Visibility reasoning and perspective taking
• Light-weight temporal projection
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems247
![Page 47: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/47.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Location to put-down pancake mix
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
Concepts
• Generative model
• Stability reasoning
• Reachability reasoning
• Visibility reasoning and perspective taking
• Light-weight temporal projection
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems248
![Page 48: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/48.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Location to put-down pancake mix
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
Concepts
• Generative model
• Stability reasoning
• Reachability reasoning
• Visibility reasoning and perspective taking
• Light-weight temporal projection
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems249
![Page 49: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/49.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Example: Location to put-down pancake mix
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
Concepts
• Generative model
• Stability reasoning
• Reachability reasoning
• Visibility reasoning and perspective taking
• Light-weight temporal projection
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems250
![Page 50: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/50.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
First-order representation of Designators
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
⇓
posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems251
![Page 51: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/51.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
First-order representation of Designators
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
⇓
posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems252
![Page 52: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/52.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
First-order representation of Designators
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
⇓
posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems253
![Page 53: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/53.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
First-order representation of Designators
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
⇓
posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems254
![Page 54: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/54.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
First-order representation of Designators
(a location(for pancake-mix)(on counter)(reachable-for Rosie)(visible-for James)(not-hindering (the activity (type pancake-making))))
⇓
posesOn(Counter, PancakeMix, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(PancakeMix, ?P) ∧ stable(PancakeMix)∧ reachable(Rosie, PancakeMix) ∧ blockingObjects(Rosie, PancakeMix, ∅)∧ visible(James, PancakeMix)∧ projectPlan(MakePancakes, ?Tl) ∧ bagof(?F, flawsInTimeline(?Tl, ?F), ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems255
![Page 55: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/55.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems256
![Page 56: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/56.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Create distribution for sampling poses
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems257
![Page 57: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/57.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Draw a pose sample
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems258
![Page 58: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/58.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Place the mug
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems259
![Page 59: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/59.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Simulate for 5s, fail!
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems260
![Page 60: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/60.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Backtrack, draw another pose sample
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems261
![Page 61: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/61.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Place the mug
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems262
![Page 62: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/62.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Inference algorithm
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ stable(Cup)
1. posesOn(Counter, Cup, ?Poses)
2. member(?P, ?Poses)
3. assertPose(Cup, ?P)
4. stable(Cup)
Simulate for 5s, succeed!
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems263
![Page 63: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/63.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Built-in Predicates of CRAM Reasoning
Stabilitycontact(O1, O2) Contact between objectsstable(O) Stability of object
Visibilityvisible(P, O) Object visible from pose Poccluding(P, O1, O2) Object O2 occludes object O1
Reachabilityreachable(R, O) Object O is reachable by robot RblockingObjects(R, O, B) B is the list of objects that “block” O
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems264
![Page 64: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/64.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Built-in Predicates of CRAM Reasoning
Stabilitycontact(O1, O2) Contact between objectsstable(O) Stability of object
Visibilityvisible(P, O) Object visible from pose Poccluding(P, O1, O2) Object O2 occludes object O1
Reachabilityreachable(R, O) Object O is reachable by robot RblockingObjects(R, O, B) B is the list of objects that “block” O
Stability reasoning
• Bullet physics engine
• Infer collisions and stability of scenes
• Simulate for a short period of time (5s)
• Example: mugs can be placed on plates but not the other way around
5s
5s
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems265
![Page 65: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/65.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Built-in Predicates of CRAM Reasoning
Stabilitycontact(O1, O2) Contact between objectsstable(O) Stability of object
Visibilityvisible(P, O) Object visible from pose Poccluding(P, O1, O2) Object O2 occludes object O1
Reachabilityreachable(R, O) Object O is reachable by robot RblockingObjects(R, O, B) B is the list of objects that “block” O
Visibility reasoning
• OpenGL to infer visibility of objects
• Render scene from camera frame, each object in a different color and countpixels
• Infer visibility of objects and occluding objects
See also “Geometric Tools for Perspective Taking for Human-Robot Interaction” [Marin-Urias 2008]
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems266
![Page 66: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/66.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Built-in Predicates of CRAM Reasoning
Stabilitycontact(O1, O2) Contact between objectsstable(O) Stability of object
Visibilityvisible(P, O) Object visible from pose Poccluding(P, O1, O2) Object O2 occludes object O1
Reachabilityreachable(R, O) Object O is reachable by robot RblockingObjects(R, O, B) B is the list of objects that “block” O
Reachability
• Use inverse kinematics to infer reachability
• No grasp planning, reach for object center
• Blocking objects = objects in collision with IK solution
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems267
![Page 67: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/67.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples
Find a pose for the cup...
... on the counter.
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)
∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems268
![Page 68: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/68.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples
Find a pose for the cup...
... on the counter that is visible for the robot.
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)
∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems269
![Page 69: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/69.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples
Find a pose for the cup...
... on the counter that is visible and reachable for the robot.
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)
∧ blockingObjects(Rosie, PancakeMix, ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems270
![Page 70: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/70.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples
Find a pose for the cup...
... on the counter that is visible and reachable for the robot and there are noblocking objects.
posesOn(Counter, Cup, ?Poses) ∧ member(?P, ?Poses)∧ assertPose(Cup, ?P) ∧ visible(Pr2, Cup)∧ reachable(Pr2, Cup)∧ blockingObjects(Rosie, PancakeMix, ∅)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems271
![Page 71: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/71.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal projection
(a location ...(not-hindering (the activity (type (pick-up Cup1)))))
⇓
projectPlan(PickUp(Cup1), ?Tl) ∧ bagof(?F, flawsInTimeline(?F), ∅)
1. Execute plan in projection mode
2. Projection generates a timeline
3. Match pre-defined flaws on the timeline
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems272
![Page 72: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/72.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Temporal projection
(a location ...(not-hindering (the activity (type (pick-up Cup1)))))
⇓
projectPlan(PickUp(Cup1), ?Tl) ∧ bagof(?F, flawsInTimeline(?F), ∅)
1. Execute plan in projection mode
2. Projection generates a timeline
3. Match pre-defined flaws on the timeline
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems273
![Page 73: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/73.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Reasoning based on Execution Traces
• Why did you leave the cup on thetable while clearing it?
• Where did you stand whileperforming a task?
• What did you see?
• How did you move?
• How did you move the arm whilegrasping the bottle?
• . . .
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems274
![Page 74: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/74.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Our approach
1. Record execution trace
– Belief state– State of plan execution, tasks, activation, deactivation, results
2. Provide an interface to the execution trace through a first-orderrepresentation
– Symbolic annotations of plans– Causal relations through plan hierarchy– Symbolic representation of objects in plans
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems275
![Page 75: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/75.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Our approach
1. Record execution trace
– Belief state– State of plan execution, tasks, activation, deactivation, results
2. Provide an interface to the execution trace through a first-orderrepresentation
– Symbolic annotations of plans– Causal relations through plan hierarchy– Symbolic representation of objects in plans
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems276
![Page 76: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/76.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Our approach
1. Record execution trace
– Belief state– State of plan execution, tasks, activation, deactivation, results
2. Provide an interface to the execution trace through a first-orderrepresentation
– Symbolic annotations of plans– Causal relations through plan hierarchy– Symbolic representation of objects in plans
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems277
![Page 77: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/77.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
System Overview
Logging
Beliefstate
SensorsTaskTree
...
......
Interpreter Plan Actuators
Plan Execution
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems278
![Page 78: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/78.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan projection and Timeline generation
Achieve(ObjInHand(Cup1))
Perceive(Cup1)
......
AtLocation(ReachLoc)
ObjectGrasped(Cup1) ObjectLifted(Cup1)
t
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems279
![Page 79: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/79.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan projection and Timeline generation
Achieve(ObjInHand(Cup1))
Perceive(Cup1)
......
AtLocation(ReachLoc)
ObjectGrasped(Cup1) ObjectLifted(Cup1)
t
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems280
![Page 80: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/80.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan projection and Timeline generation
Achieve(ObjInHand(Cup1))
Perceive(Cup1)
......
AtLocation(ReachLoc)
ObjectGrasped(Cup1) ObjectLifted(Cup1)
t
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems281
![Page 81: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/81.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan projection and Timeline generation
Achieve(ObjInHand(Cup1))
Perceive(Cup1)
......
AtLocation(ReachLoc)
ObjectGrasped(Cup1) ObjectLifted(Cup1)
t
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems282
![Page 82: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/82.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan projection and Timeline generation
Achieve(ObjInHand(Cup1))
Perceive(Cup1)
......
AtLocation(ReachLoc)
ObjectGrasped(Cup1) ObjectLifted(Cup1)
t
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems283
![Page 83: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/83.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Concepts for Reasoning
To reason about plans and their effects, we define
• Events
• Occasions
• Plan representation
• Intentions and the purpose of tasks.
• Helper predicates to access the execution trace.
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems284
![Page 84: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/84.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
System Design
High LevelPlans
Belief stateOccasions
Perception Navigation ManipulationProcessModules
Designators
Events
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems285
![Page 85: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/85.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Occasions and Events
• Events
– Occur at points in time.– We assume that they are caused by actions of the robot.– Describe changes in the (believed) world state.– Examples: PickUp(obj), PerceiveObject(obj)
⇒ Occurs(ev , t)
• Occasions
– Describe states.– Can hold at intervals or points in time:
At(t), During(t1, t2), Throughout(t1, t2)– Inferred based on events– Examples: Loc(obj, loc), ObjectInHand(obj)
⇒ Holds(occ, t)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems286
![Page 86: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/86.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Occasions and Events
• Events
– Occur at points in time.– We assume that they are caused by actions of the robot.– Describe changes in the (believed) world state.– Examples: PickUp(obj), PerceiveObject(obj)
⇒ Occurs(ev , t)
• Occasions
– Describe states.– Can hold at intervals or points in time:
At(t), During(t1, t2), Throughout(t1, t2)– Inferred based on events– Examples: Loc(obj, loc), ObjectInHand(obj)
⇒ Holds(occ, t)
LocChange(obj) An object changed its locationLocChange(Robot) The robot changed its locationCollision(obj1, obj2) obj1 and obj2 started collidingCollisionEnd(obj1, obj2) obj1 and obj2 stopped collidingPickUp(obj) obj has been picked upPutDown(obj) obj has been put downObjectPerceived(obj) The object has been perceived
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems287
![Page 87: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/87.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Occasions and Events
• Events
– Occur at points in time.– We assume that they are caused by actions of the robot.– Describe changes in the (believed) world state.– Examples: PickUp(obj), PerceiveObject(obj)
⇒ Occurs(ev , t)
• Occasions
– Describe states.– Can hold at intervals or points in time:
At(t), During(t1, t2), Throughout(t1, t2)– Inferred based on events– Examples: Loc(obj, loc), ObjectInHand(obj)
⇒ Holds(occ, t)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems288
![Page 88: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/88.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Occasions and Events
• Events
– Occur at points in time.– We assume that they are caused by actions of the robot.– Describe changes in the (believed) world state.– Examples: PickUp(obj), PerceiveObject(obj)
⇒ Occurs(ev , t)
• Occasions
– Describe states.– Can hold at intervals or points in time:
At(t), During(t1, t2), Throughout(t1, t2)– Inferred based on events– Examples: Loc(obj, loc), ObjectInHand(obj)
⇒ Holds(occ, t)
Loc(obj, loc) The location of an objectLoc(Robot, loc) The location of the robotObjectVisible(obj) The object is visible to the robotObjectInHand(obj) The object is carried by the robotObjectPlacedAt(obj, loc) The object has been placed at locationTaskStatus(task, status) The status of a task
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems289
![Page 89: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/89.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Plan Representation
Achieve(Loc(Obj, Loc))
Perceive(Loc(Obj, Loc))
Achieve(ObjInHand(Obj))
......
Achieve(ObjPlacedAt(Obj, Loc))
......
...
• Semantic annotation of sub-plans: Achieve(Loc(Obj, Loc))• Representation of objects, locations and action parametrized by designators
table = LocationDesig(On(Table))mug = ObjectDesig(Type(Mug), Color(White), At(table))grasp = TrajectoryDesig(To(Grasp), Obj(mug))
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems290
![Page 90: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/90.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Intentions
Achieve(Loc(bottle, table))
Achieve(ObjectOpened(fridge))
Achieve(ObjInHand(bottle))
Achieve(ObjectClosed(fridge))
Achieve(ObjPlacedAt(bottle, table))
• Plan interpretation creates a task tree.
• The intention of a task is to achieve a state in the world, i.e. an occasion.
• Intention of task tsk is to achieve ObjectInHand(Obj)⇔ TaskGoal(tsk,Achieve(ObjectInHand(Obj)))
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems291
![Page 91: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/91.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples (1)
Which object did you see in the drawer?
Query for the object designator at the end of a perceive goal.
Task(tsk) ∧ TaskGoal(tsk, Perceive(obj))∧ TaskEnd(tsk, t)∧ DesignatorTypeAt(obj, type, t)⇒ type = Plate
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems292
![Page 92: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/92.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples (1)
Which object did you see in the drawer?
Query for the object designator at the end of a perceive goal.
Task(tsk) ∧ TaskGoal(tsk, Perceive(obj))∧ TaskEnd(tsk, t)∧ DesignatorTypeAt(obj, type, t)⇒ type = Plate
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems293
![Page 93: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/93.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples (2)
Are there unachieved plans?
Query for top-level plans where occasions do not hold at the end.
UnachievedGoalError(o) ⇔TopLevel(tt) ∧ Task(tsk)∧ Subtask(tt, tsk) ∧ TaskGoal(tsk, o)∧ TaskEnd(tt, t) ∧ ¬Holds(o, t)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems294
![Page 94: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/94.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Examples (2)
Are there unachieved plans?
Query for top-level plans where occasions do not hold at the end.
UnachievedGoalError(o) ⇔TopLevel(tt) ∧ Task(tsk)∧ Subtask(tt, tsk) ∧ TaskGoal(tsk, o)∧ TaskEnd(tt, t) ∧ ¬Holds(o, t)
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems295
![Page 95: Technical Cognitive Systems - ias.cs.tum.eduDepartment of Informatics Intelligent Autonomous Systems Technische Universität München Example: Cracking an egg \A cook is cracking a](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b1aeaa3d0693546d7e4a68/html5/thumbnails/95.jpg)
Department of InformaticsIntelligent Autonomous Systems Technische Universität München
Questions?
Motion Planning
Michael BeetzSummer Term 2012
Technical Cognitive Systems296