p l a nn i ng (a i m a ch. 10 )

Planning (AIMA Ch. 10)

1

• Planning problem defined

• Simple planning agent• Types of plans• Graph planning

What we have so far

2

• Can TELL KB about new percepts about the world

• KB maintains model of the current world state

• Can ASK KB about any fact that can be inferred from KB

How can we use these components to build a planning agent?

i.e., an agent that constructs plans that can achieve its goals, and that then executes these plans?

Planning Problem

3

• Find a sequence of actions that achieves a given goal when executed from a given initial world state. That is, given◦a set of operator descriptions (defining the possible

primitive actions by the agent),◦an initial state description, and◦a goal state description or predicate,

• compute a plan, which is◦a sequence of operator instances, such that

executing them in the initial state will change the world to a state satisfying the goal-state description.

• Goals are usually specified as a conjunction of subgoals to be achieved

Planning vs. Problem Solving

4

• Planning and problem solving methods can often solve the same sorts of problems

• Planning is more powerful because of the representations and methods used

• States, goals, and actions are decomposed into sets of sentences (usually in first-order logic)

• Search often proceeds through plan space rather than state space (though there are also state-space planners)

• Subgoals can be planned independently, reducing the complexity of the planning problem

Remember: Problem-Solving Agent

5

function SIM PLE-PROBLEM -SOLV ING-A GENT ( percept) ret urns an actionst at ic: seq, an action sequence, init ial ly empt y

state, some descript ion of t he cur rent wor ld st at egoal, a goal, init ial ly nullproblem, a problem formulat ionstate f.. U PDA T E-ST A T E(state,

percept)if seq is empt y t hen

goal f.. FORM ULA T E-GOA L(state)

// What is the current state?

problem f.. FORM U LA T E-PROB LEM (state, goal) seq f.. SEA RCH ( problem)

action f.. RECOM M EN D A T I ON (seq, state)seq f.. REM A I N D ER (seq, state)ret urn action

Note: This is offline problem-solving. Online problem-solving involves acting w/o complete knowledge of the problem and environment

// From LA to San Diego (given curr. state)

// e.g., Gas usage

// If fails to reach goal, update

Simple planning agent

6

• Use percepts to build model of current world state

• IDEAL-PLANNER: Given a goal, algorithm generates plan of action

• STATE-DESCRIPTION: given percept, return initial state description in format required by planner

• MAKE-GOAL-QUERY: used to ask KB what next goal should be

A Simple Planning Agent

7

function SIMPLE-PLANNING-AGENT(percept) returns an actionstatic: KB, a knowledge base (includes action descriptions)

p, a plan (initially, NoPlan)t, a time counter (initially 0)

local variables:G, a goalcurrent, a current state description

TELL(KB, MAKE-PERCEPT-SENTENCE(percept, t)) current STATE-DESCRIPTION(KB, t)if p = NoPlan then

G ASK(KB, MAKE-GOAL-QUERY(t))p IDEAL-PLANNER(current, G, KB)

if p = NoPlan or p is empty thenaction NoOp

elseaction FIRST(p) p REST(p)

TELL(KB, MAKE-ACTION-SENTENCE(action, t))t t+1return action

Like popping from a stack

Goal of Planning

8

• Choose actions to achieve a certain goal• But isn’t it exactly the same goal as for

problem solving?• Some difficulties with problem solving:

◦The successor function is a black box: it must be “applied” to a state to know which actions are possible in that state and what are the effects of each one

Goal of Planning

9

• Choose actions to achieve a certain goalut isn’t it exactly the same goal as for oblem solving?

ome difficulties with problem solving:

• B • Suppose that the goal is HAVE(milk).prthe successor function must be repeatedly applied

to• S eventually generate a state where HAVE(milk) is satisfied.

• From some initial state where HAVE(milk) is not satisfied,

• An explicit representation of the possible actions and◦ their effects would help the problem solver select

the relevant actions.possible in that state and what are the effects ofeac

h oneOtherwise, in the real world an agent would be overwhelmed by irrelevant actions

Goal of Planning

• Choose actions to achieve a certain goal• But isn’t it exactly the same goal as for

problem solving?• Some difficulties with problem solving:

◦The goal test is another black-box function, states are domain-specific data structures, and heuristics must be supplied for each new problem

10

Goal of Planning

• Choose actions to achieve a certain goal• But isn’t it exactly the same goal as for

problem solving?• Some difficulties with problem solving:

◦The goal test is another black-box function, states are domain-specific data structures, and heuristics must be supplied for each new problem

11

• Suppose that the goal is HAVE(milk) ^ HAVE(book).• Without an explicit representation of the goal, theproblem solver cannot know that a state where HAVE(milk)is already achieved is more promising than a state whereneither HAVE(milk) nor HAVE(book) is achieved.

Goal of Planning

• Choose actions to achieve a certain goal• But isn’t it exactly the same goal as for

problem solving?• Some difficulties with problem solving:

◦The goal may consist of several nearly independent subgoals, but there is no way for the problem solver to know it

12

Goal of Planning

13

• Choose actions to achieve a certain goal• But isn’t it exactly the same goal as for

pro• So

blem solving?me difficulties with problem solving:

HAVE(milk) and HAVE(book) may be achieved by two nearly independent

sequences of actions

◦The goal may consist of several nearly independent subgoals, but there is no way for the problem solver to know it

Representations in Planning

14

Planning opens up the black boxes by usinglogic to represent:• Actions• States• Goals

Problem solving Logic representation

Planning

How to represent Planning Problem

• Classical planning: we will start with problems that are full observable, deterministic, statics environment with single agents

15

PDDL

PDDL = Planning Domain Definition Language ← standard encoding language for “classical” planning tasksComponents of a PDDL planning task:• Objects: Things in the world that interest us.• Predicates: Properties of objects that we are interested in; can be

true or false.• Initial state: The state of the world that we start in.• Goal specification: Things that we want to be true.• Actions/Operators: Ways of changing the state of the world.PDDL can describe what we need for search algorithm: 1) initial state 2) actions that are available in a state 3) result of applying an action 4) and goal test

16

PDDL operators

States: Represented as a conjunction of fluent (ground, functionless atoms).Following are not allowed: At(x, y) (non-ground), ￢ Poor (negation), and At(Father(Fred), Sydney) (use function symbol)

• Use closed-world assumption: Any fluent that are not mentioned are false.• Use unique names assumption: States named differently are distinct.

*Fluents: Predicates and functions whose values vary from by time (situation). *Ground term: a term with no variables. *Literal: atomic sentence.Initial state is conjunction of ground atoms.Goal is also a conjunction ( ) of literal (positive or negative) that may ∧contain variables.

17

PDDL operators cont.

• Actions: described by a set of action schemas that implicitly define the ACTIONS(s) and RESULT(s, a).

• Action schema: composed of 1) action name, 2) list of all the variable used in the schema, 3)a precondition, and 4) an effect

• Action(Fly(p, from, to),PRECOND: At(p, from) Plane(p) Airport(from) Airport(to)∧ ∧ ∧EFFECT: ￢ At(p, from) At(p, to)∧

• Value substituted for the variables: Action(Fly(P1, SFO, JFK),PRECOND: At(P1, SFO) Plane(P1) Airport(SFO) Airport(JFK)∧ ∧ ∧EFFECT: ￢ At(P1, SFO) At(p, JFK)∧

Action a is applicable in state s if the preconditions are satisfied by s. *preconditions: Condition which makes an action possible.

18

PDDL operators cont.

• Result of executing action a in state s is defined as a state s which is ′represented by the set of fluents formed by starting with s, removing the fluents what appear as negative literals in the action’s effects.

• For example: in Fly(P1, SFO, JFK) we remove At(P1, SFO) and add At(P1, JFK).

a Actions(s) s |= Precond(a)∈ ⇔Result(s, a) = (s − Del(a)) Add(a)∪

where Del(a) is the list of literals which appear negatively in the effect of a, and Add(a) is the list of positive literals in the effect of a.• * The precondition always refers to time t and the effect to time t + 1.

19

PDDL: Cargo transportation planning problem

20

Major approaches

21

• Planning graphs• State space planning• Situation calculus• Partial order planning• Hierarchical decomposition (HTN planning)• Reactive planning

Basic representations for planning

22

• Classic approach first used in STRIPS planner circa 1970• States represented as a conjunction of ground literals

◦ at(Home)• Goals are conjunctions of literals, but may have

existentially quantified variables◦ at(?x) ^ have(Milk) ^ have(bananas) ...

• Do not need to fully specify state◦ Non-specified either don’t-care or assumed false◦ Represent many cases in small storage◦ Often only represent changes in state rather than entire

situation• Unlike theorem prover, not seeking whether the goal is

true, but is there a sequence of actions to attain it

C

A B

Start State

On(C, A)On(A, Table) On(B, Table) Clear(C) Clear(B)

Sparse encoding, but

complete state spec

A

C

B

Goal

Planning Problems

23

Which goal first?

ACTION: MoveToTable(b,x)PRECONDITIONS: On(b,x), Clear(b), Block(b), Block(x), (bx) POSTCONDITIONS: On(b,Table), Clear(x)

On(b,x)

Action schema, instantiates to give

specific ground actions

On(B, C)On(A, B)

Set of goal states, only requirements

specified (think unary constraints)

Operator/action representation

24

• Operators contain three components:◦ Action description◦ Precondition - conjunction of positive literals◦ Effect - conjunction of positive or negative literals which

describe how situation changes when operator is applied• Example:

Op[Action: Go(there),

Precond: At(here) ^ Path(here,there), Effect: At(there) ^ ~At(here)]

• All variables are universally quantified

• Situation variables are implicit

At(here) Path(here,there)

Go(there)At(there) ~At(here)

◦ preconditions must be true in the state immediately before operator is applied; effects are true immediately after

C

A B

Start State

Sequential Plan

MoveToTable(C,A) > Move(B,Table,C) > Move(A,Table,B)

Types of Plans

25

On(C, A)On(A, Table) On(B, Table) Clear(C) Clear(B) Block(A)…

Partial-Order Plan

MoveToTable(C,A)> Move(A,Table,B)

Move(B,Table,C)

Forward Search

Forward (progression) state-space search: ♦ Search through the space of states, starting in the initial

state and using the problem’s actions to search forward for a member of the set of goal states.

♦ Prone to explore irrelevant actions ♦ Planning problems often have large state space. ♦ The state-space is concrete,

i.e. there is no unassigned variables in the states.

26

C

A B

Start State

On(C, A)On(A, Table)On(B, Table)Clear(C)Clear(B)Block(A)

Forward Search

27

A CB

MoveToBlock(C,A,B)

Applicable actions

Backward Search

• Backward (regression) relevant-state search: ♦ Search through set of relevant states, staring at the set of

states representing the goal and using the inverse of the actions to search backward for the initial state.

♦ Only considers actions that are relevant to the goal (or current state).

♦Works only when we know how to regress from a state description to the predecessor state description.

♦ Need to deal with partially instantiated actions and state, not just ground ones.

(Some variable may not be assigned values.)

28

A

C

B

Goal State

MoveToBlock(A,Table,B) MoveToBlock(A,x’,B)

ACTION: MoveToBlock(b,x,y) PRECONDITIONS: On(b,x), Clear(b), Clear(y),

Block(b), Block(y), (bx), (by), (xy) POSTCONDITIONS: On(b,y), Clear(x)

On(b,x), Clear(y)

Backward Search

29

A C

B?

On(B, C)On(A, B)Relevant

actions

Start:HaveCake

Goal: AteCake, HaveCake

Action: Eat Pre: HaveCake Add: AteCake

Have=T, Ate=F

Have=F, Ate=T

{Eat} {}

Have=T, Ate=F

Planning “Tree”

30

Del: HaveCake

Action: Bake Pre: HaveCake Add: HaveCake

Have=T, Ate=T

{Bake} {}

Have=F, Ate=T

Have=F, Ate=T

{Eat}

Have=T, Ate=F

{}

Have=T, Ate=F

{Eat} {}

Have=T, Ate=F

Reachable State Sets

31

Have=F, Ate=T

Have=T Ate=F

Have=T, Ate=T

{Bake} {}

Have=F, Ate=T

Have=F, Ate=T

{Eat}

Have=T, Ate=F

{}

Have=T, Ate=F

Have=F, Ate=T

Have=T, Ate=T

Have=F, Ate=T

Have=T, Ate=F

Have=T, Ate=F

Have={T}, Ate={F}

Approximate Reachable Sets

32

Have=T, Ate=F

Have=F, Ate=T

Have=T, Ate=T

Have=F, Ate=T

Have=T, Ate=F

Have={T,F},

Ate={T,F}

Have={T,F},Ate={T,F}

(Have, Ate) not (T,T)(Have, Ate) not (F,F)

(Have,Ate) not (F,F)

Planning Graphs

33

Start:HaveCake

Goal: AteCake, HaveCake

Action: EatPre: HaveCake Add: AteCake Del: HaveCake

HaveCake

Eat

HaveCake

HaveCake

Action: BakePre: HaveCake Add: HaveCake

S0

AteCake AteCake

AteCake

A0 S1

Mutual Exclusion (Mutex)

34

HaveCake

Eat

HaveCake

HaveCake

NEGATIONLiterals and their negations can’t be true at the

S0

AteCake AteCake

AteCake

A0 S1

same time

P

P

Mutual Exclusion (Mutex)

35

INCONSISTENT EFFECTSAn effect of one negates the effect of the other

HaveCake

Eat

HaveCake

HaveCake

S0

AteCake AteCake

AteCake

A0 S1

Mutual Exclusion (Mutex)

36

INCONSISTENT SUPPORTAll pairs of actions that achieve two literals are mutex

HaveCake

Eat

HaveCake

HaveCake

S0

AteCake AteCake

AteCake

A0 S1

Planning Graph

37

Eat

HaveCake

HaveCake

Bake

HaveCake

Eat

HaveCake

HaveCake

AteCake

AteCake

A1 S2S0

AteCake AteCake

AteCake

A0 S1

Mutual Exclusion (Mutex)

38

COMPETITIONPreconditions are mutex; cannot both hold

INCONSISTENT EFFECTSAn effect of one negates the effect of the other

Mutual Exclusion (Mutex)

39

INTERFERENCEOne deletes a precondition of the other

Planning Graph

40

Observation 1

41

Observation 2

42

Actions monotonically increase

(if they applied before, they still do)

Observation 3

43

p

q

rA

p

q

r

p

q

r

Proposition mutex relationships monotonically decrease

… … …

Observation 4

44

p

q

…B

p

q

r

p

q

r

A

B

A

p

q

rB

A

Action mutex relationships monotonically decrease

s

…

s

…

C C s

…

C

Observation 5

45

• Claim: planning graph “levels off”◦ After some time k all levels are identical◦ Because it’s a finite space, the set of literals

cannot increase indefinitely, nor can the mutexes decrease indefinitely

• Claim: if goal literal never appears, or goal literals never become non-mutex, no plan exists◦ If a plan existed, it would eventually achieve all goal literals

(and remove goal mutexes – less obvious)◦ Converse not true: goal literals all appearing non-mutex

does not imply a plan exists

• Relax problem by ignoring preconditions

• Can drop all or just some preconditions• Can solve in closed form or with set-cover methods

Heuristics: Ignore Preconditions

46

• Relax problem by not deleting falsified literals

• Can’t undo progress, so solve with hill-climbing (non-admissible)

COn(C, A)On(A, Table) On(B, Table)

Heuristics: No-Delete

47

A BClear(C) Clear(B)

ACTION: MoveToBlock(b,x,y) PRECONDITIONS: On(b,x), Clear(b), Clear(y),

Block(b), Block(y), (bx), (by), (xy) POSTCONDITIONS: On(b,y), Clear(x)

On(b,x), Clear(y)

• Independent subgoals?• Partition goal literals

• Find plans for each subset

• cost(all) < cost(any)?

• cost(all) < sum-cost(each)?

A

C

B

Goal State

Heuristics: Independent Goals

48

On(B, C)On(A, B)

On(A, B) On(B, C)

Heuristics: Level Costs

49

S1

• Planning graphs enable powerful heuristics◦ Level cost of a literal is the smallest S in which it appears◦ Max-level: goal cannot be realized before largest goal

conjunct level cost (admissible)◦ Sum-level: if subgoals are independent, goal cannot be

realized faster than the sum of goal conjunct level costs (not admissible)

◦ Set-level: goal cannot be realized before all conjuncts are non-mutex (admissible)

Bake

HaveCake HaveCake HaveCake

Eat HaveCake Eat HaveCake

AteCake AteCake

AteCake

A1 S2

AteCake

S0

AteCake

A0

Graphplan

50

• Graphplan directly extracts plans from a planning graph

• Graphplan searches for layered plans (often called parallel plans)◦ More general than totally-ordered plans, less general than

partially-ordered plans• A layered plan is a sequence of sets of actions

◦ actions in the same set must be compatible◦ all sequential orderings of compatible actions gives same

resultA C ? D

B D A C B

move(A,B,TABLE) move(C,D,TABLE)

move(B,TABLE,A) move(D,TABLE,C);

Layered Plan: (a two layer plan)

Search problem:Start state: goal set at last level Actions: conflict-free ways of

achieving the current goal set Terminal test: at S0 with goal set

entailed by initial planning state

Note: may need to start much deeper than the leveling-off point!

Caching, good ordering is important

Solution Extraction: Backward Search

51

An example domain: the blocks world

The domain consists of:1. A table, a set of cubic blocks and a robot arm;2. Each block is either on the table or stacked on top of

another block3. The arm can pick up a block and move it to another

position either on the table or on top of another block;4. The arm can only pick up one block at time, so it

cannot pick up a block which has another block on top.

A goal is a request to build one or more stacks of blocks specified in terms of what blocks are on top of what other blocks

52

State descriptions

•Blocks are represented by constants A, B, C, . . . etc. and an additional constant T able representing the table.

•The following predicates are used to describe states:

On(b, x) block b is on x, where x is either another block or the table

Clear (x) there is a clear space on x to hold a block

53

PDDL: Block-world problem

54

Start State Goal State

C

B A

Initial state:On(C, A) ∧ OnT able(A) ∧ OnT able(B ) ∧ Clear (B ) ∧ Clear (C )

Goal:On(A, B ) ∧ On(B , C ) ∧ OnT able(C )

A

B

C

Actions schemas

• Actions schemas:MOVE(b, x, y):

Precond: On(b, x) Clear(b) Clear(y)∧ ∧Effect: On(b, y) Clear(x)∧ ∧

￢ On(b, x)∧ ￢ Clear(y)MOVE-TO-TABLE(b, x):

Precond: On(b, x) Clear(b)∧Effect: On(b, Table) Clear(x) ∧ ∧

￢ On(b, x)

55

PDDL: Block-world problem

56

Start State Goal State

C

B A

Initial state:On(C, A) ∧ OnT able(A) ∧ OnT able(B ) ∧ Clear (B ) ∧ Clear (C )

Goal:On(A, B ) ∧ On(B , C ) ∧ OnT able(C )

A

B

C

Goal stack planning

• We work backwards from the goal, looking for an operator which has one or more of the goal literals as one of its effects and then trying to satisfy the preconditions of the operator. The preconditions of the operator become sub-goals that must be satisfied. We keep doing this until we reach the initial state.

• Goal stack planning uses a stack to hold goals and actions to satisfy the goals, and a knowledge base to hold the current state, action schemas and domain axioms

• Goal stack is like a node in a search tree; if there is a choice of action, we create branches

57

Goal stack planning pseudo-code

• Push the original goal on the stack. Repeat until the stack is empty:• If stack top is a compound goal, push its unsatisfied sub-goals on the

stack.• If stack top is a single unsatisfied goal, replace it by an action that

makes it satisfied and push the actions precondition on the stack.• If stack top is an action, pop it from the stack, execute it and change

the knowledge base by the actions effects.• If stack top is a satisfied goal, pop it from the stack.

58

Goal stack planning 1

• (We do pushing the sub-goals at the same step as the compound goal.) The order of sub-goals is up to us; we could have put On(B,C) on top

On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(C,A),OnTable(A),OnTable(B),Clear(B), Clear(C)}plan = [ ]The top of the stack is a single unsatisfied goal. We push the action which would achieve it, and its preconditions.

59

Goal stack planning 2

On(A, Table)Clear(B)Clear(A)On(A, Table) Clear(A) Clear(B)∧ ∧MOVE(A, x,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(C,A),OnTable(A),OnTable(B),Clear(B), Clear(C)}plan = [ ]The top of the stack is a satisfied goal. We pop the stack (twice).

60

Goal stack planning 3

Clear(A)On(A, Table) Clear(A) Clear(B)∧ ∧MOVE(A, Table,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(C,A),OnTable(A),OnTable(B),Clear(B), Clear(C)}plan = [ ]

The top of the stack is an unsatisfied goal. We push the action which wouldachieve it, and its preconditions. 61

Goal stack planning 4

On(C,A)Clear(C)On(C,A) Clear(C)∧MOVE-TO-TABLE(C,A)Clear(A)On(A, Table) Clear(A) Clear(B)∧ ∧MOVE(A, Table,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(C,A),OnTable(A),OnTable(B),Clear(B), Clear(C)}plan = [ ]

The top of the stack is a satisfied goal. We pop the stack (three times).62

Goal stack planning 5

MOVE-TO-TABLE(C,A)Clear(A)On(A, Table) Clear(A) Clear(B)∧ ∧MOVE(A, Table,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(C,A),OnTable(A),OnTable(B),Clear(B), Clear(C)}plan = [ ]

The top of the stack is an action. We execute it, update the KB with itseffects, and add it to the plan. 63

Goal stack planning 6

Clear(A)On(A, Table) Clear(A) Clear(B)∧ ∧MOVE(A, Table,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {OnTable(C),OnTable(A),OnTable(B),Clear(A), Clear(B), Clear(C)}plan = [MOVE-TO-TABLE(C,A)]

The top of the stack is a satisfied goal. We pop the stack (twice).

64

Goal stack planning 7

MOVE(A, Table,B)On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {OnTable(C),OnTable(A),OnTable(B),Clear(A), Clear(B), Clear(C)}plan = [MOVE-TO-TABLE(C,A)]

The top of the stack is an action. We execute it, update the KB with itseffects, and add it to the plan.

65

Goal stack planning 8

On(A,B)On(B,C)OnTable(C)On(A,B) On(B,C) OnTable(C)∧ ∧

KB = {On(A,B), OnTable(C), OnTable(B), Clear(A), Clear(C)}plan = [MOVE-TO-TABLE(C,A),MOVE(A, Table,B)]the current state is

66

Goal stack planning 9

If we follow the same process for the On(B,C) goal, we end up in the state

On(B,C) OnTable(A) OnTable(C)∧ ∧The remaining goal on the stack,

On(A,B) On(B,C) OnTable(C)∧ ∧is not satisfied. So On(A,B) will be pushed on the stack again.

67

Goal stack planning 10

Now we finally can move A on top of B, but the resulting plan is redundant:

MOVE-TO-TABLE(C,A)MOVE(A, Table,B)MOVE-TO-TABLE(A,B)MOVE(B, Table,C)MOVE(A, Table,B)

There are techniques for ‘fixing’ inefficient plans (where something is done and then undone) but it is difficult in general (when it is not straight one after another)

68