Modelling CGFs for tactical air-to-air combat training

Motivation-based behaviour and Machine Learning in a common architecture

Jan Joris Roessingh, Ph.D., Maj. Roel Rijken, M.Sc.National Aerospace Laboratory (NLR), Royal Netherlands Air Force

smartbandits@nlr.nl

Contents

Part 1(Intelligent CGFs)

Smart Bandits Requirements State-of-the-art in CGFs Architecture Cognitive models

Part 2 (Machine Learning)

Experiment RL (example) Pros & Cons ML Towards hybrid models (example) Conclusions Exploratory Team under IST panel

Project Goals “Smart Bandits”

Development of intelligent Computer Generated Forces

for tactical mission training of fighter pilots in the opponent role ‘humanlike’ behaviour

– capable of tactical reasoning intelligent decision making team work ..

– Constrained by Situation Awareness Memory capacity ..

Intelligent CGFs should be suitable for use in simulations at MoD

Application of intelligent CGFs in Embedded Training?

Current Scope

Tactical mission training Tactics are techniques for

using aircraft and weapons in a combined fashion with the purpose to gain advantage over / defeat the enemy

Air-to-Air

Beyond Visual Range (>10 NM)

Offensive and Defensive Counter Air (picture)

1v1, 2v2, 4v4 engagements

Requirements

Operational CGFs in the opponent role

– should be autonomous– should exhibit credible behaviour– should contribute to training value of simulation

Functional weapon system functions human functions more specific functions per mission phase

– planning & briefing– targeting– executing the game plan– self-defence

Research facility: NLR’s “Fighter 4-Ship”

One station of the Fighter 4-Ship (Four networked F-16 simulations)

F-16 executes OCA mission (Offensive Counter Air)Su-27 executes DCA mission (Defensive Counter Air)

FLOT

F-16

Su-27

State-Of-the-Art in CGFs

Scenario-management packages behaviour of CGFs is “scripted” pre-defined CGFs lack appropriate weapon and human

models limited possibilities for the use of AI

Agent Qualities Non-responsive behaviour Stimulus-Response (S-R) behaviour Delayed Response (DR) behaviour motivation-based behaviour

• combines S-R and DR behaviour + ‘motivational states’

TAC-AIR SOAR cognitive architecture deals with observations, decisions

and coordination Order of magnitude: 10.000 tactical decision rules

Machine Learning techniques

Agent Development Approach

Multi Agent - 2v2

Smart Adversary Behaviour

Reinforcement Learning

Neural Networks

Single Agent - 1v1Situation

Awareness

Theoryof Mind

Evolutionarytechniques

Multi Agent - 4v4

Decision Making

Machine Learning Techniques

Cognitive (BDI)

Models

Tactical Scenarios

(scripted)

Architecture

• Agent-models are functionally separated from simulation environment

• Human-like behaviour can be linked to CGFs

• Different agent models can run on different machines

Simulator – CGF package

Cognitive Models

(Team) Situation Awareness

Naturalistic Decision Making

Theory of Mind

Example : Situation Awareness

Definition Mica Endsley (1988) three levels of SA:

– the perception of the environment, – the comprehension and integration of

information, and – the projection of information into future events.

Translation to “BDI” framework Perceive: Observations/ Simple beliefs Understand: Complex beliefs Anticipate: Future beliefs

Human constraints belief formation constrained by workload

Cognitive model for situation awareness: overview

from Hoogendoorn, van Lambalgen & Treur, 2011

Example belief network for SA model

from Hoogendoorn, van Lambalgen & Treur, 2011

Reinforcement Learning Experiment

Pros and Cons Machine Learning

Pros Save development time (less knowledge elicitation

required) Adaptation to environment and opponent Complex behaviour in complex domains New tactics and evaluation of human tactics

Cons Learning speed Effectiveness (unpredictable behaviour) Computation time and memory requirements Adaptation to game randomness Increase development time (tweaking)

Hybrid models (Dynamic Scripting, Spronck et al., 2005)

Reinforcement learning

Scripts

Conclusions

Cognitive modelling one of the fundamental techniques for motivation-based behaviour CGFs

Machine Learning is powerful tool to: enhance and complement cognitive models reduce knowledge elicitation efforts

Smart Bandits: combination of models, utilizing advantages of different approaches

Technical Activity Proposal (TAP)Machine Learning Techniques for Battlefield Agents

Exploratory Team under the IST panel

Some topics to be covered: Current applications of ML Potential applications in Defence (all Forces) Potential barriers for application Most appropriate ML techniques Systems engineering aspects of ML

3 meetings in 2012, 1st in Amsterdam, early 2012

Leading to a Research Task Group

TAP available!

Let us know whether you are interested to participate!