Outline
1. Introduction
2. Parameter tuning
1. Supervised learning
2. Comparison training
3. Reinforcement learning
3. Conclusion
Game Playing Program
Simple implementation of real-world problem- If one player wins, the other must lose.
Very large search spaces- We can’t get complete information in limited time.
Game tree search
7
9 8 7 6 5 4 3 2 1
7 4 1
Root node: current position Node: position Branch: legal move
MINIMAX SEARCH
Requirements for a evaluation function accuracy efficiency
Static Evaluation Function
4 5 2 -1 6 3 7 -2 1
Static Evaluation FunctionEvaluated by
Features and Weights
Feature f(The number of pieces of each side, etc.)
Weight w(The weight of a important feature must be big.)
Linear
Non-linear Neural Network , etc.
nn fwfw 11
Machine learning in games
In simple game like Othello and backgammon, parameter tuning by machine learning has been successful.
In complex games like Shogi, hand-crafted evaluation function is still better. Machine learning is used only in limited domains.(only value of materials, etc.)
Outline
1. Introduction
2. Parameter tuning
1. Supervised learning
2. Comparison training
3. Reinforcement learning
3. Conclusion
Training sample:
Minimize the error of the evaluation function on these positions.
Supervised learning
(Position, Score)
Supervised learning(1)
Backgammon program [Tesauro 1989] Score is given by human experts. Standard back-propagation Far from human expert-level. Out
In1 In2 In458 In459
・・・
・・・
Input : Position and move (459 hand-crafted feature of Boolean value)
Score of Move
w1 w2
w3 w5w4
Supervised learning
Difficulties in supplying training data by experts Consuming much time of experts to create a
database. Human experts don’t think in terms of absolute
scores.
Supervised learning(2)
Bayesian learning [Lee et. al. 1988] Training position is labeled win or lose. Estimate the mean feature vector and the covariance
matrix for each label from training data.
x1
x3
x4
x2
μwin
μlose
Test sample
Supervised learning(3)
LOGISTELLO [Buro 1998] Build different classifiers for different stages of the game.
(Othello is a game of finite plies.) last stage → middle/first stage.
(Scores of last stage is more reliable than middle/first stage.)
Outline
1. Introduction
2. Parameter tuning
1. Supervised learning
2. Comparison training
3. Reinforcement learning
3. Conclusion
Comparison training
Training sample:
Evaluation function learns to satisfy the constraint of these training sample.
Expert’s move is preferred above all other moves.
(Position_1 , Position_2, which is preferable)
Backgammon program [Tesauro 1989] Consistency Transitivity Standard back-propagation Simpler and stronger than
preceding versions of supervisedlearning.
Comparison training
Final position (a) Final position (b)
W1 W2
W3 W4
W1=W2
W3=-W4
Which is preferable
)( abba )( cacbba
Comparison training
Problems of Comparison training Is the assumption “human expert’s move is the
best“ correct? A program trained on experts’ games will imitate a
human playing style, which makes it harder for the program to surprise a human being.
Outline
1. Introduction
2. Parameter tuning
1. Supervised learning
2. Comparison training
3. Reinforcement learning
3. Conclusion
Reinforcement learning
No training information from a domain expert.
Program explores the different actions.
It will receive feedback from the environment (reward). Win or Lose By which margin the
program won/lost. Program (Learner)
Environment
action reward position
TD(λ)
Temporal Difference Learning
t
kk
kttttt xFxFxFww
111 )())()((
wt : Weight vector at time t.
F: Evaluation function (Function of vector W and position x)
xt : Position at time t.
α: Learning rate.
λ: Influence of the current evaluation function value for weight updates of previous moves
Temporal Difference Learning
F(xt-3) F(xt-2) F(xt-1) F(xt) F(xt+1)
TD(λ)
t
kk
kttttt xFxFxFww
111 )())()((
0)()( 1 tt xFxFλ= 0
λ= 0.5
λ= 1
F(xt+1)
F(xt+1)
F(xt)
F(xt)
F(xt-1)
F(xt-1)
F(xt-2)
F(xt-2)
F(xt-3)
F(xt-3)
Temporal Difference Learning(1)
TD-Gammon [Tesauro 1992-] Neural Network (Input: raw board information) TD(λ) Self-play (300,000games) Human expert level
Program
Program
action action
Self-play in other games
None of those successors achieved a performance as impressive as TD-Gammon’s.
In case of backgammon, dice before each move ensured a sufficient variety.
exploration-exploitation dilemmaProgram
Program
action action
Temporal Difference learning(2)
KNIGHT CAP [Baxter et al. 1998] Learned on Internet chess server 1468 features (linearly combined) × 4 stages TDLeaf (λ)
Knight Cap’s rating
All but the material parameters are initially set to zero.
After about 1000 games on the server, its rating had improved to exceed 2150, which is an improvement from an average amateur to a strong expert.