machine learning in computer games marc ponsen 11/29/04
TRANSCRIPT
![Page 1: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/1.jpg)
Machine Learning in Computer Games
Machine Learning in Computer Games
Marc Ponsen
11/29/04
![Page 2: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/2.jpg)
2
Game AI: The Last Frontier
“Progress in graphics and sound has slowed in recent years… Now, more than ever, good game play is at the forefront and AI is one of the most critical components (Rabin 2004).”
“… Interactive computer games provide a rich environment for incremental research on human-level AI… Populating games with realistic, human-level characters will lead to fun, challenging games with great game play (Laird 2000).”
![Page 3: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/3.jpg)
3
Why Learning of Game AI?
The process of learning in games generally implies the adaptation of behavior for opponent players in order to improve performance
Self-correction Automatically fixing exploits
Creativity Responding intelligently to new situations
Scalability Better entertainment for strong players Better entertainment for weak players
![Page 4: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/4.jpg)
4
Online – during gameplayAdapt to player tacticsAvoid repetition of mistakesRequirements: computationally cheap, effective,
robust, fast learning (Spronck 2004) Offline - before the game is released
Devise new tacticsDiscover exploits
Offline vs. Online Learning
![Page 5: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/5.jpg)
5
Machine Learning Techniques
![Page 6: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/6.jpg)
6
Overview
Reinforcement Learning (RL) Introduction to RLRL application: Dynamic ScriptingDemo NeverWinter Nights ©
Evolutionary Algorithms (EA) Introduction to EAEA application: EA in Stratagus/WargusDemo Wargus
![Page 7: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/7.jpg)
7
Introduction to Reinforcement Learning Techniques Framework based on punishments and rewards Maximize the frequency of rewards Minimize the frequency of punishments RL is popular research area because:
RL can solve wide variety of complex problems RL will find close to optimal solution RL learns as it interacts with the environment
![Page 8: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/8.jpg)
8
How to provide Punishments & Rewards Weight adaptation function based on fitness
score(s) Lookup tables Neural networks
![Page 9: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/9.jpg)
9
When to provide Punishments & Rewards Reward when AI achieves objective or the opponent
finds itself in a state where it can’t achieve its objective
Reward when AI does something to increase the chance of achieving objective (guided rewards)
Punish when opponent has reached objective Punish when AI does something to decrease the
chance of achieving objective (guided negative rewards)
Using both overall and guided (negative) rewards, speeds up the learning process
![Page 10: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/10.jpg)
10
RL application: Dynamic Scripting
Dynamic Scripting (DS) is an online learning technique inspired by RL
Original implementation of DS (Spronck 2004) in the Computer RolePlaying Game NeverWinter Nights ©
![Page 11: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/11.jpg)
11
Dynamic Scripting
Rulebase A
Rulebase B
Script A
Script B
Combat
generate
generate script
script
scripted
scripted
control
control
human control
human control
weight updates
team controlled by human player
team controlled by computer
A
B
![Page 12: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/12.jpg)
12
Dynamic Scripting and Requirements
Computationally Cheap - Script generation and weight updates once per encounter
Effective - Rules are manually designed Robust - Reward/penalty system Fast Learning – Experiments showed that DS
is able to adapt fast to an unchanging tactic
![Page 13: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/13.jpg)
13
NeverWinter Nights © Demo
![Page 14: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/14.jpg)
14
Dynamic Scripting in Wargus
Different rulebases for different game states State transition on constructing a building that
allows new units or new research
![Page 15: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/15.jpg)
15
Rules in Rulebases
12 Build rules 9 Research rules 4 Economy rules 25 Combat rules
AiNeed(AiBarracks)
AiResearch(AiUpgradeArmor1)
AiNeed(AiWorker)
AiForce(1, {AiSoldier, 9})AiWaitForce(1)AiAttackWithForce(1)
![Page 16: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/16.jpg)
16
Script Generation
State 1Rulebase
State nRulebase
StartRule1.1
BuildRule 1.x
Rulen.1
State 20Rulebase
Rule20.20
AttackLoop
Rule20.19
![Page 17: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/17.jpg)
17
Tactics Two `balanced’ tactics
Small Balanced Land Attack (SBLA)Large Balanced Land Attack (LBLA)
Two `rush’ tacticsSoldier Rush (SR)Knight Rush (KR)
![Page 18: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/18.jpg)
18
Dynamic Scripting Test
Dynamic player (using dynamic scripting) plays 100 consecutive games against static player
Randomisation Turning Point (RTP): First game that dynamic player outperforms static player with 90% probability according to a randomisation test (Cohen, 1995)
![Page 19: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/19.jpg)
19
Dynamic Scripting RTP Results
Tactic Tests Low High Avg. Med. >100 Won
SBLA 31 18 99 50 39 0 59.3
LBLA 21 19 79 49 47 0 60.2
SR 10 10 1.2
KR 10 10 2.3
Dynamic Scripting WORKS! It adapts efficiently against SBLA and LBLADynamic Scripting is unable to adapt to the optimized tactics SR and KR
![Page 20: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/20.jpg)
20
Improve Domain-Knowledge Reliability
Successful adaptive game AI is based invariably on domain knowledge of the game it is used in
Step 1: Use EA to evolve new tactics to deal with problematic static opponents
Step 2: Translate evolved tactics to improved domain knowledge (rulebase revision)
![Page 21: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/21.jpg)
21
Introduction to Evolutionary AlgorithmsIdea: Biological analogy on how populations of species evolve over
generations
Step 1: start with a population (each member is a candidate solution)
…
Step 2: Create the next generation by considering evolutionary operations on the population from the previous generation (genetic operators such as mutation or crossover) and a fitness function (only the more fit get to contribute to the next generation)
…
Continue the process until a certain condition is reached
![Page 22: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/22.jpg)
22
Chromosome
Start State 1 State 2 End
State marker Rule x.1 Rule x.2 Rule x.n
State m
Chromosome
State
Rule ID Parameter 1 Parameter p Rule Parameter 2
Start S C1 2 5 def B S E 8 R 15 B 4 3 S
State number x
1 3 4
Rule 1.1 Rule 1.2 Rule 3.1 Rule 3.2 Rule 3.3
State 1 State 3
AiForce(2, {AiSoldier, 5})AiForceRole(“defend”)
![Page 23: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/23.jpg)
23
Genetic Operators State Crossover Rule Replace Mutation Rule Biased Mutation Randomization
Start State 1 State 2 End Parent 1 State 6 State 8 State 12 State 13 State 16 State 19 State 20
Start State 1 State 2 End Child State 6 State 8 State 12 State 13 State 16 State 19 State 20
Start State 1 State 3 End Parent 2 State 4 State 8 State 12 State 13 State 14 State 17 State 20
![Page 24: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/24.jpg)
24
Wargus Demo
![Page 25: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/25.jpg)
25
Evolution Results
Tactic Tests Low High Avg.
SR 10 0.73 0.85 0.78
KR 10 0.71 0.84 0.75
EA was successful in rapidly discovering tactics able to defeat both rush tactics
![Page 26: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/26.jpg)
26
Analysis of Counter-Tactics
Counter `soldier rush’ with (enhanced) `soldier rush’
Against `knight’s rush’Expand territory when defendableQuickly develop strong unitsUse catapults
![Page 27: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/27.jpg)
27
Rulebase Improvement
New rulesAnti-SR: create strong soldiers and attackAnti-KR: if a `knights’ state can be reached, do so
and start building knightsDefendable expansion: build defensive force then
expand Changes
Adjustment of parameters of several combat rules
![Page 28: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/28.jpg)
28
Improved Rulebase Results
Tactic Tests Low High Avg. Med. >100 Won
SBLA 11 10 34 19 14 0 72.5
LBLA 11 10 61 24 26 0 66.4
SR 10 10 27.5
KR 10 10 10.1
18 99 50 39 59.3
19 79 49 47 60.2
1.2
2.3
Enhanced performance against ALL tactics!
![Page 29: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/29.jpg)
29
Conclusions
Dynamic scripting can be successfully implemented in (commercial) CRPG & RTS games
Offline evolutionary learning can design counter-tactics for RTS games
Evolved counter-tactics can be used to improve domain knowledge used by adaptive game AI
![Page 30: Machine Learning in Computer Games Marc Ponsen 11/29/04](https://reader035.vdocuments.us/reader035/viewer/2022062321/56649ef55503460f94c08fb3/html5/thumbnails/30.jpg)
30
References
Spronck, P., I. Sprinkhuizen-Kuyper, and E. Postma. 2004a. “Online Adaptation of Game Opponent AI with Dynamic Scripting.”
Sweetser, P. (2002). Current AI in Games: A review Manslow. “Learning and Adaptation” & “Using reinforcement
learning to Solve AI Control Problems” Laird, J.E. 2000. “Human-Level AI's Killer Application:
Computer Game AI.” Ponsen, M. Spronck. P. 2004 “Improving Adaptive AI with
Evolutionary Learning”