Heuristic Search: Intro
• Blind search can be applied to all problems
• Is inefficient, as does not incorporate knowledge about the problem to guidethe search
• Such knowledge can be used when deciding which node to expand next
• Heuristic search (aka informed, best first search) uses problem knowledge toselect state that appears to be closest to goal state as next to expand
• This should produce a solution faster than systematically expanding all nodestil a goal is stumbled upon
• The general state evaluation function: f(n) = h(n) + g(n)
– f(n)) is estimated cost of getting from start state to goal state via state n
– g(n) is actual cost of getting from start state to n found so far
– h(n) is heuristic function: estimated cost of going from n to goal state
∗ This function is where knowledge about the problem comes into play
∗ h(goal) = 0
– f ∗(n) = h∗(n) + g∗(n) represents the minimal cost of getting from startstate to goal state via state n when all paths through n are considered
∗ h∗(n) is actual cost to goal from n
∗ g∗(n) is cost of best path to n from initial state
• There are categories of heuistic search, based on variations in f(n))
1
Heuristic Search: Intro (2)
• General algorithm (tree):
function GIS-TREE (problem, evalFn) returns solution, or failure
{
node <- new(node)
node.STATE <- problem.INITIAL-STATE
node.COST <- apply(evalFn, INITIAL-STATE)
frontier <- new(priorityQueue)
INSERT(node, frontier)
explored <- new(set)
loop {
if (EMPTY?(frontier))
return failure
node <- POP(frontier)
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node)
for (each action in problem.ACTIONS(node.STATE)) {
child <- CHILD-NODE(problem, node, action)
frontier <- INSERT(child, frontier)
}
}
}
2
Heuristic Search: Intro (3)
• General algorithm (graph):
function GIS-GRAPH (problem, evalFn) returns solution, or failure
{
node <- new(node)
node.STATE <- problem.INITIAL-STATE
node.COST <- apply(evalFn, INITIAL-STATE)
frontier <- new(priorityQueue)
INSERT(node, frontier)
explored <- new(set)
loop {
if (EMPTY?(frontier))
return failure
node <- POP(frontier)
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node)
ADD(node.STATE, explored)
for (each action in problem.ACTIONS(node.STATE)) {
child <- CHILD-NODE(problem, node, action)
if ((child.STATE !in explored) AND
((!STATE-FOUND(child.STATE, frontier)) {
frontier <- INSERT(child, frontier)
else if (STATE-FOUND(child.STATE, frontier)) {
node <- RETRIEVE-NODE(child.STATE, frontier)
if (child.COST < node.COST))
frontier <- REPLACE(node, child, frontier)
}
}
}
}
3
Heuristic Search: Intro (4)
– The above is a modified version of the uniform cost algorithm
– it accepts an additional argument: the function used to assign a value to anode
– Function apply(function, node) evaluates function for a given node
– PATH-COST has been replaced by a more generic PATH attribute
– Attributes that need to be stored on a node are dependent on the searchstrategy
4
Heuristic Search: Greedy Search
• The greedy search algorithm selects the node that appears to be closest to thegoal
– f(n) = h(n)
– I.e., g(n) = 0: We ignore cost to get to a node
• Algorithm GS:
function GS (problem) returns solution, or failure
{
return GIS-TREE (problem, h)
// OR return GIS-GRAPH (problem, h)
}
• Characteristics:
– Attempts to get to goal as quickly as possible
– May reach dead ends
– May expand nodes unnecessarily
– Not complete for tree rep (due to reversible actions)
– Complete for graph rep of finite state space
– Not optimal
• Complexity:
– Storage requirements ∈ O(bd) (where d is max depth of structure
– Time requirements ∈ O(bd)
• Note that quality of heuristic greatly affects search
5
Heuristic Search: A∗
• Combines aspects of uniform cost and greedy seraches
– UCS is optimal and complete, but inefficient
– UCS based on g(n)
– GS is neither optimal nor complete, but efficient
– GS based on h(n)
– For A∗, f(n) = h(n) + g(n)
• Algorithm A∗ (for trees)
function ASTAR-TREE (problem) returns solution, or failure
{
return GIS-TREE (problem, h + g)
}
• A∗ is optimal and complete, providing that
1. For trees, h(n) is admissible
– Admissible means that h(n) ≤ h∗(n)
– I.e., h(n) never over-estimates cost from n to g
– Such an algorithm is optimistic
2. For graphs, h(n) is consistant (monotonic)
– h(n) is consistant if for every node n and successor nodes n′ generatedby action a, the estimated cost of getting from n to the goal is nevergreater than the actual cost of getting fom n to a successor state n′ plusthe estimated cost of getting from n′ to the goal
– I.e., h(n) ≤ c(n, a, n′) + h(n′) for all n′
6
Heuristic Search: A∗ (2)
– This is reflected in the following diagram:
– This is referred to as the triangle inequality
– Relevance: Estimated cost to goal cannot decrease by more than cijwhen move from ni to nj
– Consistancy is a stronger requirement than admissiblity
– Consider
∗ The above represents a nonmonotonic function
∗ Since h(ni) underestimates the cost of getting from n to g, the costmust be at least 4
∗ Since c(ni, a, nj) = 1, the cost from nj to g must be at least 3
∗ h(nj) = 2 is unrealistic (i.e., As you get closer to the goal, the overallcost should be increasing, not decreasing)
7
Heuristic Search: A∗ Graphs
• As with blind search, using graphs instead of trees will result in extra effort
• Handling already-visited nodes more complex in A∗
– When encounter node that appears on frontier or closed, may have foundan alternate path to the node
– New path may be cheaper than original
∗ If new path is not cheaper, just ignore newly found path
∗ If not, must adjust f of the node, and adjust path pointers between itand parent
∗ If the node is on closed, will also have to cascade updated cost of paththrough descendants
8
Heuristic Search: A∗ Graphs (2)
• Algorithm:
function ASTAR-GRAPH (problem) returns solution, or failure
{
node <- new(node)
node.STATE <- problem.INITIAL-STATE
node.COST <- apply(evalFn, INITIAL-STATE)
frontier <- new(priorityQueue)
INSERT(node, frontier)
explored <- new(set)
loop {
if (EMPTY?(frontier))
return failure
node <- POP(frontier)
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node)
INSERT(node, explored-nodes)
for (each action in problem.ACTIONS(node.STATE)) {
child <- CHILD-NODE(problem, node, action)
//** If state has not been generated yet, add it to frontier**
if ((!STATE-FOUND(child.STATE, explored-nodes)) AND
((!STATE-FOUND(child.STATE, frontier)) {
frontier <- INSERT(child, frontier)
//** Replace existing node in frontier with new, cheaper node
else if (STATE-FOUND(child.STATE, frontier)) {
node <- RETRIEVE-NODE(child.STATE, frontier)
if (child.COST < node.COST))
frontier <- REPLACE(node, child, frontier)
}
//** Must propagate cheaper cost to children
else if (STATE-FOUND(child.STATE, explored-nodes)) {
node <- RETRIEVE-NODE(child.STATE, explored-nodes)
if (child.COST < node.COST))
explored-nodes <- REPLACE(node, child, explored-nodes)
UPDATE-PATH(child)
}
}}}}
9
Heuristic Search: A∗ Graphs (3)
– Note that this algorithm is essentially the same as GIS-GRAPH, except fortwo changes:
1. The last if-else:
else if (child.STATE in explored)
UPDATE-PATH(child)
∗ If child appears in explored, this means that it has been expandedand we have generated any number of ancestors for this state
∗ If we have found a cheaper path to child, then the cost associatedwith those ancestors will need to be adjusted, since the paths to theancestors go through child
∗ Function UPDATE-PATH will id those states reached via child
· These nodes may be in frontier or explored-nodes
· The g values will be updated (to a lesser value), which will decreasetheir f values
2. explored-nodes
∗ The text’s formulation maintains states - not nodes - on the exploredlist
∗ Since A∗ graph search requires us to be able to adjust costs and tra-verse paths to explored nodes, we must retain nodes - not just stateinformation - on the explored list
∗ The name explored-nodes emphasizes this change from graph searchalgorithms presented earlier
10
Heuristic Search: A∗ Optimality
• This examines the optimality of A∗ graph search
1. If h(n) is consistant, then values of f(n) are nondecreasing along any path
Given g(n′) = g(n) + c(n, a, n′)
Then, f(n′) = g(n′)+h(n′) = g(n)+c(n, a, n′)+h(n′) ≥ g(n)+h(n) =f(n)
2. At every step prior to termination of A∗, there is always a node n∗ onfrontier with the following properties:
– n∗ is on optimal path to a goal
– A∗ has found an optimal path to n∗
– f(n∗) ≤ f ∗(n0)
3. Proof (by induction):
(a) Base case:
– At start, only n0 on frontier
– n0 on optimal path to goal
– f(n∗) ≤ f(n0) = h(n0∗)(b) Inductive assumption:
– Assume m ≥ 0 nodes expanded and above holds for each
(c) Proof:
– Consider expansion of (m+ 1)st node from frontier
– Call this node n∗
– Let n∗ be on optimal path
– Then either
i. n∗ is not selected as (m+ 1)st nodeRegardless, it still has properties as noted
11
Heuristic Search: A∗ Optimality (2)
ii. n∗ is selected
– In the second case, let np be one of n∗’s successors on an optimal path
– This node is on frontier
– Path to np must be optimal, otherwise, there would be a better pathto the goal
– np becomes the new n∗ for the next iteration of the algorithm
(d) Proof that f(n∗) ≤ f(n0)
– Let n∗ be on optimal path and assume A∗ has found optimal pathfrom n0 to n∗
– Then
f(n∗) = g(n∗) + h(n∗) (1)
≤ g(n∗) + h(n∗) (since g(n∗) = g(n∗) and h(n∗) ≤ h(n∗)(2)
≤ f(n∗) (since f(n∗) = g(n∗) + h(n∗)) (3)
≤ f(n0) (since f(n∗) = f(n0) and n∗ on optimal path) (4)
(5)
12
Heuristic Search: A∗ Optimality (3)
4. Given the conditions specified for A∗ and h, and providing there is a pathwith finite cost from n0 to goal, A∗ is guaranteed to terminate with aminimal cost path to a goal
5. Proof (by contradiction): A∗ terminates if there is an accessible goal
– Assume A∗ doesn’t terminate
– Then a point will be reached where f(n) > f(n0) for some n ∈ frontier,since ε > 0
– This contradicts assumption
6. Proof (by contradiction): Termination of A∗
(a) A∗ terminates when either frontier empty
– This contradicts assumption that there is an accessible goal
(b) Or when a goal node is id’d
– Suppose there is an optimal goal g1 where f(g1) = f(n0),and A∗ finds a non-optimal goal g2 with f(g2) > f(n0) where g1 6= g2
– When A∗ terminates, f(g2) ≥ f(g2) > f(n0)
– But prior to selection of g2, there was a node n∗ on frontier on anoptimal path with f(n∗) ≤ f(n0) by previous lemma
– This contradicts assumptions
• Because f(n) is nondecreasing, contours can be drawn in the state space
– All nodes within a given contour fi have an f value less than those oncontour fi, and the f value of contour fi < fj, where i < j
13
Heuristic Search: A∗ Optimality (4)
– As the search progresses, the contours narrow and stretch toward the goalalong the cheapest path
– The more accurate h(n), the more focused the contours become
– Note that UCS generates circular contours centered on the initial state
• A∗ is optimally efficient for any given h function:
– No other algorithm is guaranteed to expand fewer nodes than A∗
14
Heuristic Search: A∗ Complexity
• The number of states within the goal contour is exponential wrt the solutionlength
• For problems of constant step costs, analysis is based on
1. Absolute error: ∆ = h∗ − h2. Relative error: ε = (h∗ − h)/h∗
• Complexity analysis depends on the characteristics of the state space
• When there is a single goal and reversible actions
– Time complexity ∈ O(b∆) = O(bεd), where d is the solution depth
– For heuristics of practical use, ∆ ∝ h∗, so ε is constant or growing and timecomplexity is exponential in d
– O(bεd) = O((bε)d) which means the effective branching factor is bε
• If there are many goal states (especially near-optimal goal states), path mayveer from the optimal path
– This will result in additional cost proportional to the number of goal stateswithin a factor of ε of the optimal cost
• With a graph, there can be exponentially many states with f(n) < C∗
• The search usually runs out of space before time becomes an issue
15
Heuristic Search: Variations to A∗
• The main issue with A∗ is memory usage
– As noted above, the algorithm usually uses up available memory beforetime becomes an issue
• The following algorithms attempt to limit memory usage while retaining theproperties of A∗
16
Heuristic Search: Variations to A∗ - Iterative Deepening A∗ Search (IDA∗)
• This is the A∗ version of the depth first iterative deepening algorithm
– On each iteration, perform a ”depth first” search
– Instead of using a depth limit, use a bound on f
• Algorithm IDA∗
function IDASTAR (problem) returns solution, or failure
{
root <- new(node)
root.STATE <- problem.INITIAL-STATE
root.FCOST <- problem.GET-INITIAL-STATE-H()
f-limit <- root.FCOST
while (TRUE) {
[result, f-limit] <- DFS-CONTOUR(root, problem, f-limit)
if (result != NULL)
return result
else if (f-limit == infinity)
return failure
}
function DFS-CONTOUR (node, problem, f-limit) returns [solution, f-limit]
{
nextF <- infinity
if (node.FCOST > f-limit)
return [node, node.FCOST]
if (problem.GOAL-TEST(node.STATE))
return [node, f-limit]
for (each action in problem.ACTIONS(node.STATE)) {
child <- CHILD-NODE(problem, node, action)
[solution, newF] <- DFS-CONTOUR(child, f-limit)
if (solution NOT NULL)
return [solution, f-limit]
nextF <- min(nextF, newF)
}
return [NULL, nextF]
17
Heuristic Search: Variations to A∗ - Iterative Deepening A∗ Search (IDA∗) (2)
• Expands nodes along contour lines of equal f
• IDA∗ complete and optimal
• Complexity
1. Space
– Let δ = smallest operator cost
– Let f ∗ = optimal solution cost
– Worst case requires bf∗
δ nodes
2. Time
– Dependent on range of values that f can assume
– Best case
∗ The fewer the values, the fewer the contours ⇒ fewer iterations
∗ Thus, IDA∗ → A∗
∗ Also has less overhead as does not require priority queue
– Worst case
∗ When have many unique values for h(n) (absolute worst is every hvalue unique)
∗ Requires many iterations
∗ ∼ 1 + 2 + ...+ n ∈ O(n2)
• To reduce time complexity
– increase f − limit by ε on every iteration
– Number of iterations ∝ 1/ε
– While reduces search cost, may produce non-optimal solution
– Such a solution is worse than optimal by at most ε
– Such algorithms called ε-admissible
18
Heuristic Search: Variations to A∗ - Recursive Best First Search (RBFS)
• This uses more memory than IDA∗ but generates fewer nodes
• Uses same approach as A∗ with following differences
– Assigns a backed up value to each node
∗ Let n be a node with mi successors
∗ Backed up value of n = b(n) = min(b(mi))
∗ Backed up value of leaf node (one on frontier) is f(n)
∗ Backed up value of a node represents descendant with lowest f value intree rooted at node
– Only most promising path to goal maintained at any time
• Algorithm (description)
– When node is expanded, f for all successors computed
– If one of these (m) has f less than b value of any node on frontier
∗ Back up values of all ancestors of m based on this value
∗ Continue from m
– Otherwise
∗ Let n′ be node on frontier with b < f(mi)
∗ Back up ancestors of n based on lowest successor f
∗ Find common ancestor of n′ and n; call this node k
∗ Let kn be the child of k that is the root of the subtree containg n
∗ Delete everything below kn (kn will have backed up value of this subtree)
∗ n′ will be next node to expand
19
Heuristic Search: Variations to A∗ - Recursive Best First Search (RBFS) (2)
• Algorithm
function RECURSIVE-BFS (problem) returns solution, or failure
{
return RBDF(problem, MAKE-NODE(problem.INITIAL-STATE), infinity)
}
function RBFS (problem, node, f-limit) returns solution, or failure
{
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node.STATE)
successors <- new(set)
for (each action in problem.ACTIONS(node.STATE)) {
successors <- INSERT(CHILD-NODE(problem, node, action), successors)
if (EMPTY(successors)
return failure
for (each s in successors)
s.f <- MAX(s.g + s.h, node.f)
loop {
best <- FIND-MIN-NODE(successors) //find node with smallest f value
if (best.f > f.limit
return failure
//find second smallest f value
alternative <- FIND-MIN-2-NODE-VALUE(successors, best.f)
result <- RBFS(problem, best, min(f-limit, alternative)
if (result != failure)
return result
}
}
20
Heuristic Search: Simplified Memory-bounded A∗ Search (SMA∗)
• IDA* and RBFS have problems due to using too little memory
– IDA∗ holds only current f-cost limit between iterations
– RBFS holds more info, using linear space
– Both have no memory of what went before
∗ May expand same nodes multiple times
∗ May experience redundant paths and the computational overhead re-quired to deal with them
• SMA∗ uses all memory available
• Characteristics:
– Avoids repeated states within memory constraints
– Complete if memory sufficient to store shallowest solution path
– Optimal if memory sufficient to store optimal solution path
– Otherwise, finds best solution possible within memory constraints
– If entire search tree fits in memory, is optimally efficient
• Algorithm overview:
– A∗ applied to problem until run out of memory
– To generate a successor when no memory available, will need to remove onefrom queue
– Removed nodes called forgotten nodes
– Remove node with highest FCOST
– Want to remember cost of best path so far through a forgotten node (incase need to return to it later)
– This info retained in root of forgotten subtree
– These vales called backed-up values
21
Heuristic Search: Simplified Memory-bounded A∗ Search (SMA∗) (2)
• Algorithm SMA∗
function SMASTAR (problem) returns solution, or failure
{
Q <- makeNode(initialState [problem])
node <- new(node)
node.STATE <- problem.INITIAL-STATE
node.FCOST <- problem.GET-INITIAL-STATE-H()
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node)
frontier <- new(queue)
INSERT(node, frontier)
loop {
if (EMPTY?(frontier))
return failure
node <- deepest, least-FCOST node in frontier
if (problem.GOAL-TEST(node.STATE))
return SOLUTION(node)
child <- CHILD-NODE(problem, node, problem.NEXT-ACTION(node.STATE))
if (NOT problem.GOAL-TEST(node.STATE) AND MAX-DEPTH(child))
child.F <- infinity
else
child.F <- max(node.F, child.G + child.H)
if (no more successors of node)
update node’s FCOST and those of ancestors to least cost path
through node
if (successors of node all in memory)
pop(node)
if (full(memory)) {
delete shallowest, highest-FCOST node r in frontier
remove r from parent successor list
frontier <- INSERT(r.PARENT, frontier) \\if necessary
insert r’s parent on frontier if necessary
}
frontier <- INSERT(child, frontier)
}
}
22
Heuristic Search: Simplified Memory-bounded A∗ Search (SMA∗) (3)
• Issues:
– If all leaf nodes have same f value is problemmatic
– To preclude cycles of selecting and deselecting the same node
∗ Always expand newest best leaf
∗ always delete oldest worst leaf
• Evaluation:
– SMA∗ can handle more difficult problems than A∗ without the memoryoverhead
– Better than IDA∗ on graphs
– For very hard problems, may have significant regeneration of paths
∗ May make solution intractable where A∗ with limited memory wouldfind a solution
23
Heuristic Search: Heuristic Function Evaluation
• Branching factor
– Let n be number of nodes expanded by A∗ for a given problem
– Let d be solution depth
– Let B be effective branching factorI.e., average branching factor over whole problem
– Then,
n =d∑i=1
B =(Bd − 1)B
B − 1
– B generally constant over a large range of instances for a given problemand generally independent of path length
– Ideal case is B = 1: converge directly to goal
– Want heuristic with smallest branching factor
– Can be used to estimate number of nodes expanded for a given B and depth
• Domination
– ha dominates hb if ha(n) ≥ hb(n)
– Dominating heuristic will always expand fewer nodes than dominated oneThe larger h(n), the more accurate it is
• Computational cost of h
– Must consider cost of computing h
– Frequently, the more accurate, the more expensive
– If computational cost outweighs cost of generating nodes, may not be worth-while
24
Heuristic Search: Designing Heuristics
• There are a number of techniques that can be used to design heuristics:
1. Problem relaxation
– Relaxed problem is one with fewer constraints
– For example, consider the eight puzzle
(a) A tile can move from A → B if they are adjacent: Corresponds tothe Manhatten distance
(b) A tile can move from A→ B if B is empty: Gaschnig’s heuristic
(c) A tile can move from A→ B: Corresponds to the number of tiles outof place
– The state space graph of a relaxed problem is a super graph of theoriginal
∗ It will have more edges than the original
∗ Any optimal solution in the original space will be a solution in therelaxed space
∗ Since the relaxed space has additional edges, some of its solutionsmay be better
– Cost of the optimal solution to a relaxed problem is an admissible heuris-tic in the original
– The derived heuristic must obey the triangle inequality, and so is con-sistent
– Good heuristics often represent exact costs to relaxed problem
2. Composite functions
– If have several heuristic functions, and none is dominant, useh(n) = max(h1(n), h2(n), ..., hm(n))
– If each hi is admissible, so is h(n)
– h(n) dominates each individual hi
3. Statistical info
– By generating random problem instances, can gather data about real vestimated costs
– If find that when h(n) = x ⇒ true cost = y z% of the time, use y inthose cases
– Admissability lost
25
Heuristic Search: Designing Heuristics (2)
4. Pattern databases
– Find the cost of generating a subproblem of the original
∗ This cost will be a lower bound on the cost of the full problem
– A pattern database stores exact solutions for every subproblem of theoriginal
– When solving the original, look up the solutions of the correspondingmatching subproblems in the DB
∗ This is an admissible heuristic
– Take maximum of all possible matches for a given configuration
5. Features
– Identify those features that (should) contribute to h
– Base heuristic on them
– The agent learns which features are valuable over time
6. Use weighted functions
– Let f(n) = g(n) + wh(n)
– w > 0
– As w decreases, f → optimal cost search
– As w increases, f → greedy search
– Experimental evidence suggests varying w inversely with tree depth
26
Heuristic Search: Learning Heuristics
• Can an agent learn a better search strategy?
• To do so, need a meta-level state space
– This space represents the internal state of the agent program during thesearch process
• The actual problem state space is the object-level state space
• A meta-level learning algorithm monitors the steps of the search process at themeta-level and compares them with properties in teh object-level to id whichsteps are not worthwhile
27