cs 480: game ai

CS 480: GAME AI MACHINE LEARNING IN GAMES

5/31/2012 Santiago Ontañón santi@cs.drexel.edu https://www.cs.drexel.edu/~santi/teaching/2012/CS480/intro.html

Reminders • Check BBVista site for the course regularly • Also: https://www.cs.drexel.edu/~santi/teaching/2012/CS480/intro.html

• Project 4 due June 7th •  Final exam, next week: June 5th

•  Tactic & Strategy •  Board Games •  Learning

Outline • Student Presentations:

•  “Game AI as Storytelling” •  “Computational Approaches to Story-telling and Creativity” •  “Procedural Level Design for Platform Games”

• Very Brief Introduction to Machine Learning • Machine Learning in Games • Course Summary

Outline • Student Presentations:

•  “Game AI as Storytelling” •  “Computational Approaches to Story-telling and Creativity” •  “Procedural Level Design for Platform Games”

• Very Brief Introduction to Machine Learning • Machine Learning in Games • Course Summary

Machine Learning • Branch of AI that studies how to:

•  Infer general knowledge from data: supervised learning •  Infer behavior from data: learning from demonstration •  Find hidden structure in data: unsupervised learning •  Infer behavior from trial an error (data): reinforcement learning

• Underlying principle is inductive inference: •  E.g. after seeing 100 times that objects fall down, we can infer by

induction the general law of gravity. •  We cannot deduce gravity from observation, we can only induce it.

Supervised Learning • Given labeled (supervised) training data:

(x1,y1) (x2,y2) (x3,y3) … (xn,yn)

•  Find the general function that maps x to y: •  f(x) = y

Examples

Supervised Learning Example •  Learning when to cross the street:

•  X is defined as a list of features: (“cars coming?”, “pedestrian traffic light color”, “time”)

•  Y defined as: cross, stay •  Examples:

•  x1 = (no, green, 3pm) y1 = cross •  x2 = (no, red, 8am) y2 = don’t cross •  x3 = (yes, red, 5pm) y3 = don’t cross •  x4 = (yes, green, 9pm) y4 = don’t cross •  x5 = (no, green, 10am) y5 = cross

•  Outcome of machine learning: •  f(x) = “cross if no cars coming and light color green”

Supervised Learning • Examples can be:

•  Fixed-length vector of attributes: •  Numerical •  Categorical

•  Structured representations (graphs, trees, logical clauses)

• Machine learning works better with: •  Few representative attributes •  Lots of data

Simplest Method: Nearest Neighbor • Examples are fixed vectors of real numbers •  To predict f(x):

•  f(x) = yi where (xi,yi) is the example with xi most similar to x (nearest neighbor)

Simplest Method: Nearest Neighbor • Examples are fixed vectors of real numbers •  To predict f(x):

•  f(x) = yi where (xi,yi) is the example with xi most similar to x (nearest neighbor)

x

Simplest Method: Nearest Neighbor • Examples are fixed vectors of real numbers •  To predict f(x):

•  f(x) = yi where (xi,yi) is the example with xi most similar to x (nearest neighbor)

x Nearest neighbor f(x) = gray

Machine Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3, C4.5) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting (AdaBoost) • Kernel Methods (SVM) • Neural Networks (MLP) •  Inductive Logic Programming •  etc.

Supervised Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3, C4.5) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting (AdaBoost) • Kernel Methods (SVM) • Neural Networks (MLP) •  Inductive Logic Programming •  etc.

Which one is the best?

Supervised Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3, C4.5) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting (AdaBoost) • Kernel Methods (SVM) • Neural Networks (MLP) •  Inductive Logic Programming •  etc.

Which one is the best?

NONE! Each one has its own biases

Outline • Student Presentations:

•  “Game AI as Storytelling” •  “Computational Approaches to Story-telling and Creativity” •  “Procedural Level Design for Platform Games”

• Very Brief Introduction to Machine Learning • Machine Learning in Games • Course Summary

Why Learning in Games?

Benefit for Games:

•  Adaptive, believable Game AI •  Find good player matches •  Realistic player movement •  Reduce development cost (e.g.

fine-tuning parameters)

Benefit for AI:

•  Games are great test beds (perfect instrumentation and measurements)

•  Reduced cost •  Extremely challenging domains •  Great way to motivate

students!

Uses of Learning in Games •  Driving Games:

•  Learn good driving behavior from observing humans •  Fine-tune parameters of physics simulation, or of car handling

parameters

•  RTS Games: •  Automatically balance the game

•  RPG/FPS Games: •  Believable movements

•  Others: •  Specific game genres possible only with learning (training games) •  Automatically adapt to player strategies •  Learning player models

Example Usages: Black & White

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Represented as hand-crafted perceptrons (single

layer neural networks)

Given the current situation, they activate more or less, triggering certain desires.

Example: hunger

The structure of the

perceptrons is hardcoded, but the parameters are

learned

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Represented as learned decision trees, one per

desire.

They capture towards which object the creature should

express each desire.

Example: hunger The creature will learn a

decision tree from player’s feedback of what can be

eaten

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Beliefs are just lists of properties of the objects in the game, used for learning

opinions.

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Which desire will be attempted, and towards

which object: e.g. “destroy town” or “eat human”

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Desire, and list of objects that will be used. E.g.

“Destroy town by throwing a stone”

Learning in Black & White

Desires

Opinions

Beliefs

Intention: Abstract

Plan

Specific Plan

Action List Belief-Desire-Intention Architecture

Specific list of actions to execute the plan

Learning in Black & White • BDI model to simulate character personality

•  Learning is deployed at specific players to adapt the BDI model to player preferences

• Quite advanced, for commercial game standards!

Example Usages: Forza Motorsport

Example Usages: Forza Motorsport •  Forza Motorsport lets you create “Drivatars”

• Drivatars use machine learning to “learn” to drive like the player.

• Supervised machine learning: •  Input: the track just ahead of the car •  Output: car trajectory

Track Segments

a4 a2 a3 a1 Segments

•  Define a set of “track segments”. All the tracks are made up of these •  For each segment, define the different ways in which it can be traversed

Learning from the Player • When the player plays:

1.  For each segment the player traverses, Drivatar remembers the trajectory used (out of the ones it has stored), speed, and other parameters.

2.  Drivatar only remembers the last data seen, and forgets old data (to adapt to newer driving styles of the player)

3.  When playing the game, it selects the trajectories learned form the player for each track segment, and combines them into a minimal curvature line, which is the path taken by the AI

Minimal Curvature Line

Learning in Drivatars: Key Ideas •  Learning to drive is very complex! Too Many parameters!

• Solution: simplify the problem!

• Simplifications: •  Divide the track in segments •  Identify a finite set of possible trajectories for each segment (this

defines a discrete and finite vocabulary for the machine learning method)

•  The goal of AI in games is to make the game fun, not to create advanced AI! (at least from a game company perspective J)

Other Games Using Learning • Colin McRae Rally 2.0: Neural networks to learn how to

drive a rally car

Other Games Using Learning • Command & Conquer Renegade: learning new paths by

observing the player (e.g. player breaks a window and creates a new path)

Other Games Using Learning • Re-Volt: genetic algorithm tunes the parameters of the

steering behaviors for cars, to optimize lap time

Ways to Fake “Learning” in Games • Preprogram different levels of difficulty and change

between them at run time

• Rule-based system: have a large pool of rules, and add them bit by bit to the AI

• Change parameters during game play: •  Increase the accuracy of shooting •  Decrease reaction time of enemies •  Etc.

Outline • Student Presentations:

•  “Game AI as Storytelling” •  “Computational Approaches to Story-telling and Creativity” •  “Procedural Level Design for Platform Games”

• Very Brief Introduction to Machine Learning • Machine Learning in Games • Course Summary

Game AI •  Intelligent behavior for characters in games

•  The goal is not to make characters smart, but to make the game believable, engaging and fun

•  There is a lot of design in game AI

• Commercial games use a collection of very simple, but highly customizable AI techniques (FSMs, Behavior Trees, Influence Maps, etc.)

Game AI

AI

World Interface

(perception)

Strategy

Decision Making

Movement

Game AI •  Perception: interfacing the AI with the game

•  Movement: jumping, firing, steering behaviors

•  Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite

•  Decision Making: FSMs, Decision Trees, Behavior Trees

•  Tactics/Strategy: Rule-Based systems, waypoints, influence maps

•  Board Games: Minimax, alpha-beta, Monte-Carlo search

•  Learning in games: supervised learning, reinforcement learning

Game AI •  Perception: interfacing the AI with the game

•  Movement: jumping, firing, steering behaviors

•  Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite

•  Decision Making: FSMs, Decision Trees, Behavior Trees

•  Tactics/Strategy: Rule-Based systems, waypoints, influence maps

•  Board Games: Minimax, alpha-beta, Monte-Carlo search

•  Learning in games: supervised learning, reinforcement learning

Projects 1, 2, 3, 4

Game AI •  Perception: interfacing the AI with the game

•  Movement: jumping, firing, steering behaviors

•  Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite

•  Decision Making: FSMs, Decision Trees, Behavior Trees

•  Tactics/Strategy: Rule-Based systems, waypoints, influence maps

•  Board Games: Minimax, alpha-beta, Monte-Carlo search

•  Learning in games: supervised learning, reinforcement learning

Midterm

Game AI •  Perception: interfacing the AI with the game

•  Movement: jumping, firing, steering behaviors

•  Pathfinding: quantization, navmeshes, A*, TBA*, LRTA*, D*-Lite

•  Decision Making: FSMs, Decision Trees, Behavior Trees

•  Tactics/Strategy: Rule-Based systems, waypoints, influence maps

•  Board Games: Minimax, alpha-beta, Monte-Carlo search

•  Learning in games: supervised learning, reinforcement learning

Final

Project 4 • Project 4 (and last): Rule-based Strategy for RTS Game (S3)

•  Idea: •  Create a perception layer that creates a simple knowledge base (logical

terms) •  Create a simple unification algorithm with variable bindings •  Define a set of actions the rule-based system can execute •  Define a small set of rules (do not overdo it! J) •  RETE is optional (extra credit) •  See how well it plays and how easy is it to make the AI play well!

•  Anyone wants to do a different project 4? Any ideas?

Next Week •  Final Exam • Project 4 presentation