integrating learning in interactive gaming simulators

1Integrating Learning in Interactive Gaming Simulators

Integrating Learning in Interactive Gaming Simulators

David W. Aha1 & Matthew Molineaux2

1Intelligent Decision Aids GroupNavy Center for Applied Research in AI

Naval Research Laboratory; Washington, DC2ITT Industries; AES Division; Alexandria, VA

[email protected]

AAAI’04 Workshop on Challenges in Game AI25 July 2004


Outline

1. Motivation: Learning in cognitive systems2. Objectives:

– Support empirical studies w/ simulators– Encourage studies that address industry

& military M&S concerns3. Design: TIELT functionality & components4. Example: Knowledge base content5. Status: Implementation, collaborations6. Summary

1. Motivation: Learning in cognitive systems2. Objectives:

– Support empirical studies w/ simulators– Encourage studies that address industry

& military M&S concerns3. Design: TIELT functionality & components4. Example: Knowledge base content5. Status: Implementation, collaborations6. Summary

Thanks to our sponsor:


Rough Anatomy of a Cognitive Agent

External Environment External Environment

Communication(language,gesture,image)

Prediction,planning

Deliberative Processes

Reflective Processes

Reactive Processes

Perception Action

STM

Sensors Effectors

Other reasoning

LTM

Concepts

SentencesCog

nit

ive

Ag

en

t

Affect

Attention

Learning

Learning

(Brachman, 2003)


Problem

Status of Cognitive Learning

Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks

– It’s costly to integrate & evaluate embedded learning techniques

Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks

– It’s costly to integrate & evaluate embedded learning techniques

Complication

The ML research community has been focusing on:¬Rapid: Knowledge poor algorithms

¬Enduring: Learning over a short time period

¬Embedded: Stand-alone evaluations

The ML research community has been focusing on:¬Rapid: Knowledge poor algorithms

¬Enduring: Learning over a short time period

¬Embedded: Stand-alone evaluations


Wanted: A New Interface (thanks to W. Cohen, others)

Supervised Learning ML

SystemDatabase Interface

(standard format)

(e.g., UCI Repository)

ReasoningSystem


SystemDatabase Interface

(standard format)

(e.g., UCI Repository)

ReasoningSystem


Systemj

DatabaseiInterface

(standard format)

(e.g., UCI Repository of ML Databases)

ReasoningSystemk

Cognitive Learning Reasoning Modules

Wor

ld(S

imu

late

d/R

ea

l)

Sensors ML Module

Interface(standard API)

ML Module

ML Module(e.g., TIELT)Effectors

Cognitive Learning Reasoning Modules

Wor

ld(S

imu

late

d/R

ea

l)

Sensors ML Module


ML Module

ML Module(e.g., TIELT)Effectors

Cognitive Learning Reasoning Systemk

Wor

ldi

(Sim

ula

ted/

Re

al)

Sensors ML Module


ML Module

ML Modulej

(e.g., TIELT)Effectors

Testbed for Integrating and Evaluating Learning TechniquesTestbed for Integrating and Evaluating Learning Techniques


Objectives & Domain

Objective

Facilitate the evaluation of learning techniques in CogSys1. Develop & transition TIELT to help reduce integration costs (time, $)2. Support DARPA Challenge Problems on Cognitive Learning3. Demonstrate research utility prior to approaching industry/military

Facilitate the evaluation of learning techniques in CogSys1. Develop & transition TIELT to help reduce integration costs (time, $)2. Support DARPA Challenge Problems on Cognitive Learning3. Demonstrate research utility prior to approaching industry/military

Domain: Why interactive gaming simulators?

1. Available implementations (cheap to acquire & run)2. Challenging problems for CogSys/ML research 3. Significant interest (military, industry, academia, funding, public)

1. Available implementations (cheap to acquire & run)2. Challenging problems for CogSys/ML research 3. Significant interest (military, industry, academia, funding, public)


TIELT Specification

1. Simplifies integration & evaluation!• Learning-embedded reasoning systems & gaming simulators• Inputs: 5 descriptions (simulator I/O, game model, learning &

performance tasks, reasoning system I/O, & evaluation strategy)• Free

2. Learning foci: Many• Task (e.g., learn how to execute, or advise on, a task)• Player (e.g., learn/predict a human player’s strategies)• Game (e.g., learn/refine its objects, their relations, & behaviors)

3. Learning methods: Many types• Supervised/unsupervised, immediate/delayed feedback, analytic,

active/passive, online/offline, direct/indirect, automated/interactive• Learning results should be available for inspection

4. Gaming simulators: Those with challenging learning tasks5. Reuse: Provide access to libraries of simulators & reasoning systems

• Abstracts interface definitions from game & task models

1. Simplifies integration & evaluation!• Learning-embedded reasoning systems & gaming simulators• Inputs: 5 descriptions (simulator I/O, game model, learning &

performance tasks, reasoning system I/O, & evaluation strategy)• Free

2. Learning foci: Many• Task (e.g., learn how to execute, or advise on, a task)• Player (e.g., learn/predict a human player’s strategies)• Game (e.g., learn/refine its objects, their relations, & behaviors)

3. Learning methods: Many types• Supervised/unsupervised, immediate/delayed feedback, analytic,

active/passive, online/offline, direct/indirect, automated/interactive• Learning results should be available for inspection

4. Gaming simulators: Those with challenging learning tasks5. Reuse: Provide access to libraries of simulators & reasoning systems

• Abstracts interface definitions from game & task models


Distinguishing TIELT

System Focus $ Game Engine(s)

Prominent Feature

Reasoning Activity

DirectIA (MASA)

AI SDK FPS, RTS, etc.

Behavior authoring Sense-act, …

SimBionic (SHAI)

AI SDK FPS, etc. Behavior authoring Sense-act, …

FEAR AI SDK Quake 2, etc. Behavior authoring Sense-act, …

RoboCup Research Testbed

RoboCup Soccer game play Sense-act, coaching, etc.

GameBots Research Testbed

UT (FPS) UT game play Sense-act

ORTS Research Testbed

RTS games Hack-free MM RTS Sense-act, strategy

TIELT Research Testbed

Several genres

Experimentation for Learning Systems

Sense-act, advice processing, prediction, model updating, etc.

1. Provides an interface for message-passing interfaces2. Supports composable system-level interfaces


TIELT’sInternal

CommunicationModules

TIELT’s KBEditors

TIELT’s KBEditors

Selected/Developed Knowledge BasesSelected/Developed Knowledge Bases

GameModel

Description

Task Descriptions

GameInterface

Description

ReasoningInterface

Description

EvaluationMethodology Description

GamePlayer(s)

GameEngineLibrary

GameEngineLibrary

Stratagus

Full Spectrum Command

TIELT’s User Interface TIELT’s User Interface

PredictionInterface

EvaluationInterface

CoordinationInterface

AdviceInterface

TIELTUser

TIELTUser

SelectedGameEngine

SelectedGameEngine

ReasoningSystemLibrary


ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

SelectedReasoning

System

SelectedReasoning

System

Learned Knowledge

(inspectable)

TIELT: Integration Architecture

Knowledge Base Libraries


GIDGID

GID

RIDRID

RID

GMDGMD

GMD

TDsTDs

TDs

EMDEMD

EMD

...


TIELT’s Knowledge Bases

GameModel

Description

Task Descriptions

GameInterface

Description

ReasoningInterface

Description


Defines communication processes with the game engine

Defines communication processes with the learning system

Defines interpretation of the game• e.g., initial state, classes, operators, behaviors (rules)• Behaviors could be used to provide constraints on learning

Defines the selected learning and performance tasks• Selected from the game model description

Defines the empirical evaluation to conduct


TIELT’sInternal


TIELT’s KBEditors

TIELT’s KBEditors


GameModel

Description

Task Descriptions

GameInterface

Description

ReasoningInterface

Description


GamePlayer(s)

GameEngineLibrary

GameEngineLibrary

Stratagus



PredictionInterface

EvaluationInterface


AdviceInterface

TIELTUser

TIELTUser

SelectedGame

Engine

SelectedGame

Engine



ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

SelectedReasoning

System

SelectedReasoning

System

Learned Knowledge

(inspectable)

Example: Controlling a Game Character



GIDGID

GID

RIDRID

RID

GMDGMD

GMD

TDsTDs

TDs

EMDEMD

EMD

Raw StateProcessed

State

DecisionAction


TIELT’sInternal


TIELT’s KBEditors

TIELT’s KBEditors


GameModel

Description

Task Descriptions

GameInterface

Description

ReasoningInterface

Description


GamePlayer(s)

GameEngineLibrary

GameEngineLibrary

Stratagus



PredictionInterface

EvaluationInterface


AdviceInterface

TIELTUser

TIELTUser

SelectedGame

Engine

SelectedGame

Engine



ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

ReasoningSystem

ReasoningSystem

Learning Module

. . .

Learning Module

SelectedReasoning

System

SelectedReasoning

System

Learned Knowledge

(inspectable)

Example: Updating a Game Model



GIDGID

GID

RIDRID

RID

GMDGMD

GMD

TDsTDs

TDs

EMDEMD

EMD

Raw StateProcessed

State

Edit

Edit


SelectedGame

Engine

EvaluationEditor

Game InterfaceEditor

Percepts

User

ReasoningInterface Editor

Game ModelEditor

TaskEditor

GameModel

Description

TaskDescriptions

Pe

rf.

Ta

sk

EvaluationInterface

Evaluator

Action / Control

Translator(Mapper)

Learning OutputsActions

Translated Model (Subset)

Learning Task

GameInterface

Description

ReasoningInterface

Description

LearningTranslator(Mapper)

CurrentState

ModelUpdater

Database

EvaluationMethodologyDescription

StoredState

AdviceInterface Database

EngineState

Controller

SelectedReasoning

System

TIELT’s Internal Communication Modules


Sensing the Game State(City placement example)

TIELT

Game Interface

Editor

Sensors

User

Game ModelEditor

GameModel

Description

Up

da

tes

GameInterface

Description

ActionTranslator

Actions

GameEngine

GameEngine

CurrentState

1

2

4 3

4

In Game Engine, the game begins; a colony pod is created and placed.

1

The Game Engine sends a “See” sensor message identifying the pod’s location.

This message template provides updates (instructions) to the Current State, telling it that there is a pod at the location See describes.

4

2

The Model Updater receives the sensor message and finds the corresponding message template in the Game Interface Description.

3

ControllerModel

Updater

3

The Model Updater notifies the Controller that the See action event has occurred.

5

5


Getting Decisions from the Learning System(City placement example)

TIELT

SelectedReasoning

System

SelectedReasoning

System

LearningModule #1

LearningModule #n

. . .

User

Learning InterfaceEditor

AgentEditor

TaskDescriptions

LearningTranslator

Translated Model (Subset)

ReasoningInterface

Description

ActionTranslator

LearningOutputs

The Controller notifies the Learning Translator that it has received a See message.

The Learning Translator finds a city location task, which is triggered by the See message. It queries the controller for the learning mode, then creates a TestInput message to send to the reasoning system with information on the pod’s location and the map from the Current State.

The Reasoning System transmits output to the Action Translator.

The Learning Translator transmits the TestInput message to the Reasoning System.

1

2 23

4

Controller

CurrentState

1

4

2

3


TIELT

Game InterfaceEditor

User

ActionTranslator

Actions

GameEngine

GameEngine

1

2

4.a

The Action Translator receives a TestOutput message from the Reasoning System.

The Action Translator finds the TestOutput message template, determines it is associated with the city location task, and builds a MovePod operator (defined by the Current State) with the parameters of TestOutput.

The Game Engine receives Move and updates the game to move the pod toward its destination, or

The Action Translator determines that the Move Action from the Game Interface Description is triggered by the MovePod Operator and binds Move using information from MovePod.

ReasoningInterface Editor

3

GameInterface

Description

ReasoningInterface

Description

AdviceInterface

The Advice Interface receives Move and displays advice to a human player on what to do next, or makes a Prediction.

4.b, c1

4.a

2

3

Acting in the Game World(City placement example)

4.b

CurrentState

2

PredictionInterface

4.c

3


Initial Work

Status (July 2004)

• TIELT v1 + documentation due 9/04– Message protocols

• Current: Console I/O, TCP/IP• Future: Library calls, HLA interface, RMI (possibly)

– Message content: Configurable• Instantiated templates tell it how to communicate with other modules

– Initialization messages: Start, Stop, Load Scenario, Set Speed– Game Model representations (w/ Lehigh University)

• Simple programs• TMK process models• PDDL (language used in planning competitions)

• Stratagus/Wargus module (Lehigh University)• Initial publicity (BRIMS’04, here)• Workshops being planned: ICCBR’05 (plus competition), ICML’05, ...?

• TIELT v1 + documentation due 9/04– Message protocols

• Current: Console I/O, TCP/IP• Future: Library calls, HLA interface, RMI (possibly)

– Message content: Configurable• Instantiated templates tell it how to communicate with other modules

– Initialization messages: Start, Stop, Load Scenario, Set Speed– Game Model representations (w/ Lehigh University)

• Simple programs• TMK process models• PDDL (language used in planning competitions)

• Stratagus/Wargus module (Lehigh University)• Initial publicity (BRIMS’04, here)• Workshops being planned: ICCBR’05 (plus competition), ICML’05, ...?

Example


TIELT Collaboration Projects (2004-05)

Organization Game Interface and Model

Reasoning Interface

Tasks and Evaluation Methodology

Mad Doc Software Empire Earth 2 (RTS)

Troika Games Temple of Elemental Evil (RPG)

ISLE SimCity (~RTS) ICARUS ICARUS w/ FreeCiv, design

Lehigh U. Stratagus/Wargus (RTS), and HTN/TMK designs

Case-based planner (CBP)

Wargus/CBP

NWU FreeCiv (discrete strategy), and qualitative game representations

U. Michigan SOAR SOAR w/ 2 games(e.g., FSW, ToEE), design

U. Minnesota-Duluth RoboCup (team sports) Advice-taking components

Advice processing

USC/ICT Full Spectrum Command(RTS)

SOAR with FSC

UT Arlington Urban Terror (FPS) DCA (lite version)

UT Austin Neuroevolution e.g., Neuroevolution/EE2


TIELT’sInternal


TIELT’s KBEditors

TIELT’s KBEditors


GameModel

Description

Task Descriptions

GameInterface

Description

ReasoningInterface

Description



PredictionInterface

EvaluationInterface


AdviceInterface

TIELT User

TIELTUser

TIELT Collaborations (2004-05)

Knowledge Base LibrariesKnowledge Base Libraries

Game LibraryGame Library

Mad Doc

EE2 ToEE

Troika

FreeCiv

NWU ISLE

Platform LibraryPlatform Library

Stratagus

Lehigh U.

UT Arl.

FSC/R

USC/ICT

UrbanTerror

U. Minn-D.

RoboCup

Reasoning System LibraryReasoning System Library

Learning Modules

Soar: U.MichICARUS: ISLE

DCA: UT Arlington

Neuroevolution: UT Austin

Others: Many

LU, USC Mich/ISLEU. Mich.Many Many

U.Minn-D. USC/ICTU.Mich.


Summary: Questions? Concerns?

TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system

• Goals: – Simplify running learning expts with cognitive systems– Support DARPA challenge problems in learning

• Designed to work with many types of simulators & reasoning systems

TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system

• Goals: – Simplify running learning expts with cognitive systems– Support DARPA challenge problems in learning

• Designed to work with many types of simulators & reasoning systems

Status: • v1 scheduled for completion in 9/04

– Please see Matt Molineaux’s demo• 11 additional organizations about to start 1-year collaborations

– Enhances probability that TIELT will achieve its goals

Status: • v1 scheduled for completion in 9/04

– Please see Matt Molineaux’s demo• 11 additional organizations about to start 1-year collaborations

– Enhances probability that TIELT will achieve its goals

integrating learning in interactive gaming simulators

Documents