![Page 1: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/1.jpg)
Adversarial Search
CS 486/686: Introduction to Artificial IntelligenceFall 2013
1
![Page 2: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/2.jpg)
Introduction
• So far have only been concerned with single agents
• Today
- Multiple agents planning against each other- Adversarial settings
2
![Page 3: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/3.jpg)
Outline
• Games
• Minimax search
• Alpha-beta pruning
• Evaluation functions
• Coping with chance
• Game programs
3
![Page 4: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/4.jpg)
Games• Games are the oldest, most well-studied domain in AI
• Why?
- They are fun
- Easy to represent, rules are clear
- State spaces can be very large- In chess, the search tree has ~10154 nodes
- Like the “real world” in that decisions have to be made and time is important
- Easy to determine when a program is doing well
4
![Page 5: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/5.jpg)
Types of Games• Perfect vs Imperfect Information
- Perfect information: You can see the entire state of the game
- Imperfect information:
• Deterministic vs Stochastic
- Deterministic: change in state is fully controlled by the players
- Stochastic: change in state is partially determined by chance
5
![Page 6: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/6.jpg)
Games as Search Problems
• 2-player perfect information game
• State: board configuration plus the player whoʼs turn it is to move
• Successor function: given a state, returns a list of (move,state) pairs indicating legal move and resulting state
• Terminal state: states where there is a win/loss/draw
• Utility function: assigns a numerical value to terminal states
• Solution: a strategy (way of picking moves) that wins the game
6
![Page 7: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/7.jpg)
Game Search Challenge
• What makes game search challenging?
- There is an opponent
- The opponent is malicious- it wants to win (by making you lose)
- We need to take this into account when choosing moves
• Notation:
- MAX player wants to maximize its utility
- MIN player wants to minimize its utility
7
![Page 8: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/8.jpg)
Example
MAX’s job is to use the search tree to determine the best move
8
![Page 9: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/9.jpg)
Optimal Strategies• In standard search
- Optimal solution is sequence of moves leading to winning terminal state
• Strategy (from MAXʼs perspective)
- Specify a move for the initial state
- Specify a move for all possible states arising from MINʼs response
- Then all possible responses to all of MINʼs responses to MAXʼs previous move
- ...
9
![Page 10: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/10.jpg)
Optimal Strategies• Goal: Find optimal strategy
• What do we mean by optimal?
- Strategy that leads to outcomes at least as good as any other strategy, given that MIN is playing optimally
- Equilibrium (game theory)
• Today we focus mainly on zero-sum games of perfect information
- Easy games according to game theory
10
![Page 11: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/11.jpg)
Minimax Value
Utility(n) if n is a terminal state
Maxs in Succ(n) MINIMAX-VALUE(s) if n is a MAX node
Mins in Succ(n) MINIMAX-VALUE(s) is n is a MIN node
MINIMAX-VALUE(n) =
ply
11
![Page 12: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/12.jpg)
Properties of Minimax• Complete if tree is finite
• Time complexity: O(bm)
- m is depth of tree
• Space complexity: O(bm)
- It is DFS
• Optimal against an optimal opponent
- If opponent is not playing optimally, then may be better off doing something else
12
![Page 13: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/13.jpg)
Minimax and Multi-Player Games
13
![Page 14: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/14.jpg)
Question
• Can we now write a program that will play chess reasonably well?
14
![Page 15: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/15.jpg)
Question
• Can we now write a program that will play chess reasonably well
- For chess b~35 and m~100
15
![Page 16: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/16.jpg)
Alpha-Beta Pruning
• If we are smart (and lucky) we can do pruning
- Eliminate large parts of the tree from consideration
• Alpha-beta pruning applied to a minimax tree
- Returns the same decision as minimax
- Prunes branches that cannot influence final decision
16
![Page 17: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/17.jpg)
Alpha-Beta Pruding• Alpha:
- Value of best (highest value) choice we have found so far on path for MAX
• Beta:
- Value of best (lowest value) choice we have found so far on path for MIN
• Update alpha and beta as search continues
• Prune as soon as value of current node is known to be worse than current alpha or beta values for MAX or MIN
17
![Page 18: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/18.jpg)
ExampleMAX
MIN
≥ 3
12 8
3
3
18
![Page 19: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/19.jpg)
ExampleMAX
MIN
[3,∞]
12 8
3
3 2
≥2
Prune remaining children
19
![Page 20: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/20.jpg)
ExampleMAX
MIN
[3,∞]
12 8
3
3 2
≤2
14
≤14
20
![Page 21: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/21.jpg)
ExampleMAX
MIN
[3,∞]
12 8
3
3 2
≤2
14
≤5
5
21
![Page 22: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/22.jpg)
ExampleMAX
MIN
[3,∞]
12 8
3
3 2
≤2
14
2
5 2
22
![Page 23: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/23.jpg)
Properties of Alpha-Beta
• Pruning does not affect the final result
- Why?
• Move ordering is important
• Alpha-beta demonstrates the value of reasoning about which computations are important
23
![Page 24: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/24.jpg)
Real-Time Decisions• Alpha-Beta can be a huge improvement
over minimax
- Still not good enough- Need to search to terminal states for at least part of search
space
- Need to make decisions quickly
• Solution
- Heuristic evaluation function + cutoff tests
24
![Page 25: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/25.jpg)
Evaluation Functions
• Apply an evaluation function to a state
- If terminal state, function returns actual utility
- If non-terminal, function returns estimate of the expected utility
• Function must be fast to compute
25
![Page 26: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/26.jpg)
Evaluation Functions
• How do we get evaluation functions?
- Expert knowledge
- Learned from experience
• Look for features of states
- Weighted linear function Eval(s)=∑i wifi(s)
26
![Page 27: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/27.jpg)
Cutting Off Search• Do we have to search to terminal states?
- No! Cut search early and apply evaluation function
• When?
- Arbitrarily (but deeper is better)
- Quiescent states- States that are “stable”
- Singular extensions- Searching deeper when you have a move that is “clearly better”
- Can be used to avoid the horizon effect
27
![Page 28: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/28.jpg)
Cutting Off Search• How deep?
- Novice player- 5-ply (minimax)
- Master player- 10-ply (alpha-beta)
- Grandmaster- 14-ply + fantastic evaluation function, opening and endgame
databases,...
28
![Page 29: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/29.jpg)
Stochastic Games
29
![Page 30: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/30.jpg)
Stochastic Games
• Need to consider best/worst cases + probability they will occur
• Recall: Expected value of a random variable x E[x]=∑x in X P(x)x
• Expectiminimax: minimax but at chance nodes compute the expected value
30
![Page 31: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/31.jpg)
Expectiminimax
31
![Page 32: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/32.jpg)
Expectiminimax
WARNING: exact values do matter! Order-preserving transformations of the evaluation function can change the choice of moves. Must have positive linear transformations only
32
![Page 33: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/33.jpg)
Summary• Games pose lots of fascinating challenges for
AI researchers
• Minimax search allows us to play optimally against an optimal opponent
• Alpha-beta pruning allows us to reduce the search space
• A good evaluation function is key to doing well
• Games are fun!
33
![Page 34: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/34.jpg)
Some Game Programs
34
![Page 35: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/35.jpg)
Checkers
Mr. Tinsley suffered his 4th and 5th losses ever against Chinook
35
![Page 36: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/36.jpg)
Checkers• Chinook (University of Alberta)
- World Man-Machine Checkers Champion
- Alpha-beta search
- Opening database
• Secret weapon: Endgame database
- Perfect knowledge into the search
• Checkers is now dominated by computers
- Checkers is (weakly) solved
36
![Page 37: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/37.jpg)
Chess: Kasparov vs. Deep Blue1997: Deep Blue wins by 3 wins, 1 loss, and 2 draws
Kasparov
5’10” 176 lbs 34 years50 billion neurons
2 pos/secExtensiveElectrical/chemicalEnormous
Deep Blue
6’ 5”2,400 lbs
4 years32 RISC processors
+ 256 VLSI chess engines200,000,000 pos/sec
PrimitiveElectrical
None
HeightWeightAge
Computers
SpeedKnowledge
Power SourceEgo
Jonathan Schaeffer37
![Page 38: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/38.jpg)
Chess• Its secret:
- Specialized chess processor + special-purpose memory optimization
- Very sophisticated evaluation function- Expert features and hand-tuned weights
- Opening and closing books
- Alpha-beta + improvements (searching up to 40 ply deep)
- Searched over 200 million positions per second
38
![Page 39: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/39.jpg)
Chess
• There are now apps that are on par with human champions
• Is Chess still a human game or have computers conquered it?
39
![Page 40: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/40.jpg)
Backgammon• TD-Gammon (Gerry Tesauro at IBM)
• One of the top players in the world
• Searches only two moves ahead!
• Its secret: One amazing evaluation function
- Neural network trained with reinforcement learning during ~1 million games played against itself
- Humans play backgammon differently now, based on what TD-Gammon learned about the game
- Very cool AI ☺
40
![Page 41: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/41.jpg)
Go• Large branching factor
makes Go too large to solve by classic search methods
• pieces added to the board
• evaluation function
• ...
• Limited progress for decades
41
![Page 42: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/42.jpg)
Go
• BUT computer Go has undergone a revolution in the past ~5 years
- Close to perfection on 7x7 games
- Reached top human level on 9x9 games
- Still weaker than top humans on 19x9 boards
42
![Page 43: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/43.jpg)
Go• Monte-Carlo Tree Search (MCTS)
- Build search tree according to outcomes of simulated plays
Upper Confidence Bounds for Trees (UCT): “Minimax search” using UCB
43
![Page 44: Adversarial Search - Cheriton School of Computer … • Games are the oldest, most well-studied domain in AI • Why?-They are fun-Easy to represent, rules are clear-State spaces](https://reader031.vdocuments.us/reader031/viewer/2022030801/5b0ac91e7f8b9a61448b7fbb/html5/thumbnails/44.jpg)
Card Games
• Focus has been on Bridge and Poker
- Humans are still winning...
- But machines are catching up!
• Issues
- Stochastic and partially observable- Ideas discussed today donʼt work well
- New approaches are being developed
44