04 september 2015 uk official
TRANSCRIPT
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
© Crown copyright 2015. Published with the permission of the Defence Science and Technology Laboratory on behalf of the Controller of HMSO
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Introduction
Purpose: to make you think about what underlies the rules
and tables in a wargame. In particular:
• What you need to know about the data.
• Where that data might come from, with examples of three
different approaches used by Dstl.
• Making peace with the fact you will never have 100% of
the data you want.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What this brief will cover
• A user’s perspective on input data
• Three methods for generating that data:
– Trials & Experimentation
– Performance & System Modelling
– Historical Analysis
• Checking your data
• Q&A
Caveat: Most of this presentation focuses on objective
data. This is only a part of the data conundrum that
wargamers face. A topic for later discussion.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Data in wargames
• Most wargames use numbers and rules.
• Different wargames use different levels of data aggregation to
produce these numbers (e.g. entity vs Coy).
• Unless you understand where all the numbers and rules come
from all wargames include black boxes.
• Lots of detail means lots of potential small black boxes.
• Often you want black boxes. E.g. players, emergent phenomenon.
• Make sure you’re comfortable with your black boxes.
Light Gun Battery
Manoeuvre Move
Artillery Power ISR Range
Medium
3 / 6 1 (2) 0
Artillery Range 1
2d6 Roll 1:5 1:4 1:3 1:2 1:1 2:1 3:1 4:1 5:1 6:1 7:1 8:1
2Attacker
WithdrawsEngaged
Defender
Withdraws
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
DefeatedExchange
Defender
Defeated
Defender
Defeated
Defender
Defeated
3Attacker
WithdrawsContact Engaged Exchange Exchange
Defender
Withdraws
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
4Attacker
Withdraws
Attacker
WithdrawsContact Engaged
Defender
Withdraws
Defender
WithdrawsExchange
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
Defender
Defeated
5Attacker
Withdraws
Attacker
Withdraws
Attacker
WithdrawsEngaged Contact Exchange
Defender
Withdraws
Defender
WithdrawsContact
Defender
Defeated
Defender
Defeated
Defender
Defeated
6Attacker
Defeated
Attacker
Withdraws
Attacker
Withdraws
Attacker
WithdrawsEngaged Contact
Defender
Withdraws
Defender
Withdraws
Defender
Withdraws
Defender
Withdraws
Defender
Defeated
Defender
Defeated
7Attacker
Defeated
Attacker
Withdraws
Attacker
Withdraws
Attacker
WithdrawsEngaged Engaged
Defender
Withdraws
Defender
Withdraws
Defender
Withdraws
Defender
Withdraws
Defender
Withdaws
Defender
Defeated
8Attacker
Defeated
Attacker
Defeated
Attacker
Withdraws
Attacker
Withdraws
Attacker
WithdrawsEngaged Contact
Defender
Withdraws
Defender
Withdraws
Defender
Withdraws
Defender
Withdaws
Defender
Defeated
9Attacker
Defeated
Attacker
Defeated
Attacker
Defeated
Attacker
Defeated
Attacker
Withdraws
Attacker
WithdrawsContact Contact
Defender
Withdraws
Defender
Withdraws
Defender
Withdaws
Defender
Withdraws
10Attacker
Defeated
Attacker
Defeated
Attacker
Defeated
Attacker
Defeated
Attacker
Withdraws
Attacker
WithdrawsEngaged Contact Contact Contact
Defender
Withdaws
Defender
Withdraws
11Attacker
Defeated
Attacker
Defeated
Attacker
DefeatedContact
Attacker
Defeated
Attacker
Withdraws
Attacker
WithdrawsEngaged
Defender
Defeated
Defender
DefeatedContact
Defender
Withdraws
12Attacker
Defeated
Attacker
Defeated
Attacker
Defeated
Defender
Withdraws
Attacker
Defeated
Attacker
Defeated
Attacker
DefeatedExchange Engaged
Defender
Defeated
Defender
DefeatedContact
– Trust
– Judgement
– Good logging
– Ability to review & discuss
– Validation & Verification
– Acceptable risk
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Users perspective
What I want to know about the data in my wargame:
• Where did it come from?
• Is it a good representation of what I want to investigate?
• Is it credible or is it counterintuitive? If so why?
• What are the uncertainties, boundaries & caveats?
– How will these impact the wargame?
– What caveats do I need to put on my outputs?
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Data sources Simple
Morale
1 2 3 4 5
1d6 1:1 2:1
Close Combat
Prob. Of Kill
Historical Analysis
Trials & Experimentation
Judgement
Systems & Performance
Modelling
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Morale
1 2 3 4 5
1d6 1:1 2:1
Close Combat
Prob. Of Kill
Systems & Performance
Modelling
Historical Analysis
Trials & Experimentation
Judgement (incl. soft effects & emergent
phenomenon)
Data sources In Practice
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Trials & Experimentation
From the Fields to the Tables
Mark Pickering
© Crown copyright 2015. Published with the permission of the Defence Science and Technology Laboratory on behalf of the Controller of HMSO
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Contents
• What real world data do we need?
• How do we collect data?
• What causes variation?
• From data to game.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What real world data do we need?
• Probability of hit, or “P(hit)”
• Achievable rates of fire
• Lethality
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Collecting data
• Modelling the ballistics – does not represent ‘a muddy field’.
• Trials – better real world data, but expensive.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What can be modelled?
• Lots of highly detailed aspects, such as:
– Weather
• Pressure
• Humidity
• Wind
– Manufacturing variation
• Propellant quality
• Density variation in round
• Barrel defects
– Etc.
• But some things can’t be easily modelled, such as…..
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
How do we measure P(hit)?
• With difficulty
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
P(hit)
• Miss distance
• What range?
• Limited data
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
P(hit) - Hit grid
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
From data to game
• Example Game Mechanism
– To hit roll = Probability of hit or P(hit)
– Armour save roll = Probability of not penetrating the armour
– To wound roll = Probability of ‘kill’ or P(kill)
Hit Armour Save Wound
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Aim Point
To hit roll
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
To hit roll
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Probable hit
To hit roll
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Armour save roll
Hit area
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Hit area
Armour View image from: War Thunder – www.warthunder.com
Armour save roll
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Lethality View image from: War Thunder – www.warthunder.com
Hit area
To wound roll
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Systems & Performance Modelling
Generating the rules
Dan Ledwick
© Crown copyright 2015. Published with the permission of the Defence Science and Technology Laboratory on behalf of the Controller of HMSO
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Systems and performance
modelling • I.e. what happens when X shoots Y?
• Information from trials and experimentation.
– P(hit).
– Level of damage per hit.
• Simulates outcomes of specific events.
• Reports at required level.
– E.g. individual bullet effects aggregated to Bde level.
• Faster and cheaper than full trials.
• Highly repeatable.
• Can represent potential future equipment.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Systems and performance modelling
• Focus on Vulnerability/Lethality
• Overview
• Example
• Data uses
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Vulnerability/Lethality overview
• Determine the result of a weapon attacking a
target
– Weapon performance
– Armour performance
– Damage to target components
– Resulting effect on target functionality
• Usually involves running a computer
simulation
– Large number of engagements simulated
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Vulnerability/Lethality example
Effectors
Initially: Brick
From interaction: Glass shards
Example: brick thrown at glasshouse.
Assessment Process
1. Trigger: Effector generator – brick.
2. Propagate brick.
3. Interaction with target component.
- Trigger generation of glass shards.
4. Component response – damage algorithms.
5. Propagate brick and glass shards.
6 and 7. Interaction/Component response.
- Repeat as necessary
1 2
3
4
5
6
7
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Vulnerability/Lethality data uses • Identifying vulnerable areas of vehicles
to prioritise protection improvements
• Characterising weapon performance against a target set
• Higher level wargames and models
– Computer simulations, Manual
• Training simulations
– DFWES, AWES
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Wargaming Generation
Trials and
Experimentation
System and
Performance
modelling
Tabletop Computer Algorithms
Judgement Wargame
Generation
Historical
Analysis
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Historical Analysis for Wargaming
Stevie Ho
© Crown copyright 2015. Published with the permission of the Defence Science and Technology Laboratory on behalf of the Controller of HMSO
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Data and wargaming
• Historical Analysis (HA) provides real-world data grounded
in reality.
– Increases buy-in when you can say: “this actually happened
before.”
• Testing the theoretical vs the actual.
• Can provide data on things trials and experiments cannot
or will not.
– Particularly in the operational and strategic spaces.
• However, you cannot create new historical data, you have
to work with what you get.
– Exact real world cases are rare, so historical analogy is often
a requirement.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Definition
• The use of mathematical, statistical and other forms of
analysis to understand historical engagements,
operations, campaigns and conflicts for the purpose of
providing impartial analysis and sensitive decision support
to policy makers.
• Critical to sensible and fully informed policy making.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Origins
• 1980s – Falklands War.
Field trials vs. Falklands War
• Combat Psychology Example: soldiers are much braver if they
know they’re in no real danger.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What data does HA use?
• Anything that will tell us what we want to know or allow us
to infer an acceptable estimate thereof.
• Primary data sources
– War diaries
– Post Op Reports
– Operational Data Sources
• Secondary data sources
– Official Histories
– Reference Books
– Academic Studies
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What methods does HA use?
• Quantitative analysis
– Regression Analysis, correlation etc.
– Stats packages – R, Minitab, SPSS etc.
– Excel formulae, charts, graphs etc.
• Qualitative analysis
– Historical Research, Framework Analysis etc.
– Literature reviews
– Case studies
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Large-N Studies
Quantitative
Statistical analysis Generalised results
Potential over-abstraction
Single Case Studies
Qualitative
In-depth analysis Contextual detail
Not representative
Comparative Analysis
Qualitative or Quantitative
5 – 30 cases Pattern Matching
Selection Bias
Increasing Depth
Increasing Abstraction
The Real World
Spectrum of historical analysis
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Large-N Studies
Quantitative
Statistical analysis Generalised results
Potential over-abstraction
Single Case Studies
Qualitative
In-depth analysis Contextual detail
Not representative
Comparative Analysis
Qualitative or Quantitative
5 – 30 cases Pattern Matching
Selection Bias
Increasing Depth
Increasing Abstraction
The Real World
This is similar to wargaming.
Different types of wargame will be
appropriate depending on your question and
depending on your data availability.
More data and material allows you to
produce a greater number of different kinds
of wargame, but one size does not fit all.
Spectrum of historical analysis
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
What does HA offer? • Multidisciplinary approach an advantage.
– Data from tactical up to grand strategic level.
– Understanding of interaction of qualitative and quantitative
factors.
• HA can highlight the important factors and back or
disprove perceived wisdom.
– Particularly important when the HA goes against commonly
held beliefs or perceptions.
• These factors can be fed into wargames or wargames can
be designed to highlight the importance of these factors.
• Can be blended with trials and experimentation data.
– A mixture of from HA, trials, experimentation and judgement
can help robustness, especially in the future space.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 5 10 15 20 25
Period
Pro
po
rtio
n o
f fat
alat
ies
attr
ibu
ted
to IE
Ds
Army
Marines
Iraq – US Army / USMC IED Fatalities Introduction of USMC MOJAVE VIPER Training
Example study: HA of value of training
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Example study: HA of value of training
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 5 10 15 20 25
Period
Pro
po
rtio
n o
f fat
alat
ies
attr
ibu
ted
to IE
Ds
Army
Marines
Iraq – US Army / USMC IED Fatalities Introduction of USMC MOJAVE VIPER Training
However, at this time the USMC also
introduced a new protective vehicle.
Which one was more important?
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
0
0.001
0.002
0.003
0.004
0.005
0.006
0.007
0.008
Aug-41 Feb-42 Sep-42 Apr-43 Oct-43 May-44 Nov-44
Date
% t
ota
l ja
pa
ne
se
to
nn
ag
e s
un
k/p
atr
ol
da
y
<4 patrols
>4 patrols
USN Submarines - Pacific Average proportion of Japanese fleet sunk each patrol day, each month by experienced and inexperienced captains.
Paired t-test p value < 0.001.
Example study: HA of value of training
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Checking Your Data
A User’s Peace of Mind
James King
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Checking your tables
• Comparison to historical operations.
• Is it a reasonable (and credible) representation
of the situation you’re trying to wargame?
– Judgement of people who have conducted similar
operations.
– Military/SME judgement.
• Look over as many cases as possible, not just
one example. Beware single data points.
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Data provenance
Best Case Worst Case
Very Common Very Rare Common
Availability
Lo
ts o
f R
ea
l W
orld
Cases
Exte
nsiv
e T
ria
ls
Tria
ls
Exp
erim
en
tatio
n
Mo
de
llin
g
His
torica
l Analo
gy
Exp
ert
Op
inio
n
(with
ou
t e
vid
en
ce
)
‘Exp
ert
’ Op
inio
n
(little
rele
va
nt
exp
erie
nce)
Judgement
Exp
ert
Op
inio
n
(with
evid
en
ce)
Provenance
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Availability
Lo
ts o
f R
ea
l W
orld
Cases
Exte
nsiv
e T
ria
ls
Tria
ls
Exp
erim
en
tatio
n
Mo
de
llin
g
His
torica
l Analo
gy
Exp
ert
Op
inio
n
(with
ou
t e
vid
en
ce
)
‘Exp
ert
’ Op
inio
n
(little
rele
va
nt
exp
erie
nce)
Judgement
Exp
ert
Op
inio
n
(with
evid
en
ce)
Provenance
Data provenance
Best Case Worst Case
Very Common Very Rare Common
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
And finally…
• What you need to know about the data.
– Think about what underlies the rules and tables in your wargame.
– Log your assumptions wherever possible.
– When designing a wargame make sure you can get the data you will need.
• Where that data might come from, with examples of three different
approaches used by Dstl.
– Trials & Experimentation, Systems & Performance Modelling, Historical Analysis.
– There are multiple sources of data, each with their own strengths and
weaknesses. Judgement ties them all together to make a wargame.
– Wargames and data sources are not isolated, but can test and inform each other.
• Making peace with the fact you will never have 100% of the data you want.
– Be comfortable with uncertainties and black boxes. People are black boxes!
– Be sure you are happy with the data you have. Understand the relationship
between the impact the data will have, and the certainty you need.
– Beware single data points!
© Crown copyright 2015 Dstl
04 September 2015
UK OFFICIAL
Questions?