Evolutionary Computation and Game Development

Part 1: Background

Overview

• What is Evolutionary Computation (EC)?• EC in the Real World• EC and the Gaming Industry• What is EC? (Continued)• EC and Games• Some Ideas• Questions

What is Evolutionary Computation?

• Evolutionary Computation is the study of algorithms inspired by the natural process of evolution.o Genetic Algorithms (GA)o Evolution Strategies (ES)o Evolutionary Programming (EP)o Genetic Programming (GP)o Particle Swarm Optimization (PSO)o Ant Systems (AS)o Ant Colony Optimization (ACO)

EC in the Real World

• Schedulingo Scheduling of flightso Task scheduling

• Routingo FedExo UPSo Disney World Tour Planning

• Designo Aircraft designo Factory layout

EC in the Gaming Industry

• IEEE CIG• IEEE Transactions on Computational Intelligence and AI in Games• Started in 2005

• NERO with rtNEAT• Using EC to create actual AI opponents• Dynamic Content Creation• Play Testing

NERO and rtNEAT

• NERO – Neuro-Evolving Robotic Operatives

• NEAT – Neuro-Evolution of Augmenting Topologies

• The NERO game was a sandbox where players evolved AI bots– Supervised learning

• Evolved Neural Network controllers with a Genetic Algorithm

EC for AI Opponents

• RTS games have recently become a test-bed for evolving AI opponents

• There is also interest in evolving support functions, units, and features in RTS

• Research and competition have also been held and are still being held for AI bots in FPS games

• However, most research still center around classic board games

Dynamic Content Creation

• Key questions– How can a game keep the player engaged?– How can a game keep players interested?– How can a game challenge a player?– How can a game satisfy a player?

• Two possible approaches – Adaptive difficulty or gameplay– Adaptive game content

Play Testing

• Use EC to find potential cheats or exploits

• Use EC to test feature balancing

• Use EC to possibly find better reproduction steps for isolating bugs/crashes

• One research done using Fifa ’99– EA Burnaby

What is EC? (Continued)

• Evolutionary Computation is the study of algorithms inspired by the natural process of evolution.

• So, what does that mean?• Here's the High Level Pseudo-code

1.Randomly Generate a Population (Generation)2.Evaluate Initial Population (Evaluation)3.Select Parents (Selection)4.Perform Crossover (Crossover)5.Apply Mutation (Mutation)6.Evaluate Offspring (Evaluation)7.Replace Parents (Replacement)8.Goto 3.

Characteristics of an Evolutionary Algorithm (EA)• "Any-time" Algorithms

o The longer you run them, the better the solution (ideally)• Stochastic in Nature

o There is alot of randomness involvedo This is the primary property

• Meta-heuristicso "a heuristic method for solving a very general class

of computational problems by combining user-given black-box procedures in a hopefully efficient way"

• Flexibility• Black Box operator that can be swapped out

Black Box Components

• Problem Encodingo How is the problem encoded/representedo Genotype and Phenotype

• Operatorso Selectiono Crossovero Mutationo Replacemento Evaluation

• Candidate Solution

Problem Encoding

• Genotype - the representation of the problem that is being manipulated by the EA

• Phenotype - the physical representation of the problem that the genotype maps to

• Encoding is a key componento is important no matter what EA you are usingo improper encoding will cause bad resultso too complicated an encoding runs the risk of bogging

down the algorithm• There are guidelines, known methods, and encoding

requirements based on the EA you want to use• However, there will always be some black magic involved

Example: Knapsack Problem

Sword 10

Shield 20

Helmet 4

Chain Mail 14

Gauntlet 6

Running Shoes 2

Healing Potion 1

Magic Wand 3

Healing Potion 1

Knapsack Problem

• Large number of items, limited space

• Simplified form of a inventory/resource management problem– Inventory management in RPG’s– Resource allocation in RTS’s

Phenotype vs. Genotype

Sword 10

Shield 20

Helmet 4

Chain Mail 14

Gauntlet 6

Running Shoes 2

Healing Potion 1

Magic Wand 3

Healing Potion 1

1

1

0

1

0

1

1

0

1

Some Known Encodings

• Bit Arrays

• Integer/Floating Point Arrays

• A mix of the above two

• Finite State Machines– First used in Evolutionary Programming

• Expression Tree– Mainly used in Genetic Programming

Evaluation Function

• Just as important if not more important than how the problem is encoded

• The quality of the evaluation function directly influences the resultso EA's have no eyeso The evaluation function is an EA's guide dog

• There are also known good methods of formulating an evaluation function

• To get optimal results, the evaluation function must be customized for the problem

• There is a whole subfield of EC that doesn't use an evaluation functiono Interactive Evolutionary Computation

Knapsack Problem (Cont.)

• How would we evaluate a candidate solution?– How do we judge which item combination is better

than any other?– Do we just want to fill the sack as much as possible,

or is there a reason to prefer certain items over others?

Knapsack Problem (cont.)

Sword 10

Shield 20

Helmet 4

Chain Mail 14

Gauntlet 6

Running Shoes 2

Healing Potion 1

Magic Wand 3

Healing Potion 1

100

70

10

54

35

27

49

5

49

Sw

ord

Shi

eld

Hel

met

Cha

in M

ail

Gau

ntle

t

Run

ning

Sho

es

Hea

ling

Pot

ion

Mag

ic W

and

Hea

ling

Pot

ion

Tot

al S

pace

Tot

al V

alue

Space 10 20 4 14 6 2 1 3 1 61Value 100 70 10 54 35 27 49 5 49 399

Candidate 1 1 0 0 0 1 1 1 0 1 20 260Candidate 2 1 1 0 1 0 1 1 0 0 47 300Candidate 3 0 0 1 0 1 1 0 1 1 16 126Candidate 4 0 1 0 1 1 1 1 1 1 47 289Candidate 5 1 1 1 1 0 0 0 0 1 49 283Candidate 6 1 0 0 0 0 0 0 1 1 14 154

Selection

• Natural Selection• How do we select parents?• Which parents should be paired up for crossover?• Common selection methods

o Random Selectiono Rank-Based Selectiono Roulette Wheel Selectiono Tournament Selectiono No Selection

P1

P2

P3P4

P5

P6

Random Selection

P1, 10

P2, 15

P3, 20

P4, 25

P5, 12

P6, 18

Roulette Wheel Selection

???

P1 P4 P3

???

P6 P2 P3

Tournament Selection

Crossover

• How do we create offspring(s) with the parents?• Completely dependent on problem representation

and how many parents are involved• Arrays of discrete values

o Single Point or Multi-point Crossovero Uniform Crossover

• Array of real valueso Discrete Crossovero Interpolation

• Trees or Finite State Machineso Pruning and Grafting

Mutation

• Also representation dependent• Any form of random perturbation

o random bit flipso random value replacementso random insertions or deletions

Replacement

• This is where Natural Selection kicks in a second time

• How will the offsprings replace the parents?• Who will survive?• Two common approaches

o Offsprings replace parents completelyo Offsprings and parents compete for survival

Operators

• Evolutionary Algorithms are meta-heuristics

• Operators are black boxes

• Doesn’t matter how they are implemented

• They just have to perform some specific task– The algorithm doesn’t care how task is accomplished

Additional Notes

• Crossover and mutation are sometimes considered optionalo Some have found better results with only one of the two

• The truth is that there is alot of voodoo magic involvedo Experience will tell you how best to encode a problemo But each problem type tends to have its own tricko There are known best practices for certain problem types

Traveling Salesman Knapsack Problem

o More often than not, your problem will fall into something totally new

Precautionary Warning

• EC is a tool– Used for exploratory purposes– Used when a pretty good solution is needed in a short

period of time– Used when there is no known solution

• especially when there is no clear definition of what a solution is

• It will never perform better than a specialized algorithm

EC in Games

• Why is EC rarely used in real games?• Complexity (Space and Time)

– Which portion of the game do we want to evolve?– How would we encode the problem and where would we store the

population?– Where will we find the clock cycles needed for the evolution?– What is a good evaluation function?– How would the producers tune this and what are behaviors they

should expect?

• Uniform Experience– How do we control the overall user experience?– Can we even guarantee a predictable user experience?– Will predictability sacrifice algorithm performance?

Interactive Evolutionary Computation

• Replaces the evaluation function with some form of human response– Great for when the fitness cannot to

quantitatively evaluated and qualitative measures are fuzzy at best

• The response can be a quantitative value or just a qualitative classification

• NERO is an example of an interactive EC

Adaptive Gaming

• Gameplay adapting to user actiono Dynamic scaling of difficulty (already been done)

• The game adapts to user actions– Dynamic content generation– “No one really knows what they really want”– “They know what they want when they see it”

Questions?