1

CHALLENGES WITH QUANTIFYING THE QUALITATIVE

In collaboration with:

Elizabeth Whitaker, Erica Briscoe, Ethan Trewhitt, Georgia TechKevin Murphy, Frank Ritter, John Horgan, Penn State

Caroline Kennedy-Pipe, Univ. of Hull

Presented to:ONR Workshop on Human Interactions in Irregular Warfare as a Complex

System Atlanta, GA

13-14 April 2011

Presented by:Dr. Lora Weiss

Georgia Tech Research InstituteLora.Weiss@gtri.gatech.edu

2

LINKING US-UK EXPERTISE(for understanding IED perpetration in Iraq)

SMEs

Doctrine

Literature

Scenarios

Knowledge Engineering

Influence ModelsSystem Dynamics

ModelsAgent-based Models

Models

Modeling

Evaluation

SMEs

Model Considerations

• Incomplete Data• Data Provenance• Data Uncertainty• Data Perishability

Objective• Provide a methodology to scientifically capture, evaluate, and predict large-scale behaviors of potential

IED developers before they have successfully deployed devices

• Elicit information from UK subject matter experts, who have had different experiences on their homeland

• Develop analytic tools to conduct quantitative and qualitative analysis of potential interdiction points

3

Knowledge Engineering Instrument

Individual

Individual

Individual

Individual

Individual

Individual

“Western”

“Non-Western”

Common Motivations ActivitiesCommon

Goals Results

Religion

Activities ResultsMoney

Power

Goal

Goal

Goal

4

Common End-state vs. Individual Motivations

Example Interview Results

• Management and planning within IED “teams” are different than in Western civilization• Participants are not necessarily focused on an end-state. Instead individual motivations

(that may differ) are manipulated toward the individual’s end goals.

5

Interview Results - 2

• Crucial differences:– IRA was aware they were being watched and operated in a manner to “fool” their

pursuers– Iraqi insurgency less of a “game-like” attitude and is more concentrated on purely

technical aspects

IRA

Counter-IED Monitoring

Iraqi Insurgency

Counter-IED Monitoring

Game-likeplanning

Little attempt toInfluence monitoring

Activities change by being monitored vs. concentrating on technical execution

6

Interview Results - 3

Normal Behavior

Current Behavior

Missing Normal

BehaviorUnusual

Anomalous Behavior

Comparison to spot differences

Record and Share

Stories

Useful information often lost because no explicit sharing of stories when units transfer

• This information is usually subtle and not directly transcribable, e.g., noticing what is not normal about an environment (social or physical)

• Military personnel notice things that are different and have a hard time putting their fingers on exactly what that is

7

Interview Results - 4

RecruitingPersonnel

Religious Motivation

Monetary Motivation

Power Motivation

Motivation varies among lower level participants

• For lower level participants (beneath management), motivation is most often monetary or peer involvement

• Experts are conflicted as to whether religion is actually a motivator or just used as ‘clean’ explanation

8

Mind Map for Preliminary Knowledge Structuring

From Knowledge Engineering to Modeling

9

• Methodology for evaluating of complex systems over time • Represent causal relationships and feedback• Stocks and flows represent the movement of items, materials, people, or abstract

concepts • Easy experimentation with changes in structure, inputs, conditions

System Dynamics

10

Central features related to IED perpetration in Iraq

Simplified Core Model

11

Materials and Supplies

• Stock – Materials & Supplies - represents the inventory of generalized materials and supplies of insurgent groups in the area

• Input Flow - Gathering - represents actions that cause the accumulation of materials and supplies

• Output Flow - Consumption - represents the use of these materials and supplies in the construction of IEDs

12

IED Process

• Four stages of IED Process: Constructed, Inventory, Emplaced, Detonated

• Flows between stocks represent transitions from one stage to another

• The Disrupted IED stock and its related flows, Early, Middle and Late Disruption represent the destruction of IEDs by counter-IED efforts.

13

Personnel

• Represent the transition of a sympathizer into active participation within a terrorist group

• Radicalization represents transition of a person from within the general population to the Grey Population.

• A previously neutral person taking a position of sympathy for insurgent beliefs

• Deradicalization is the reverse of this, when a person loses sympathy for the insurgency

• As a person becomes an active participant in the IED process, this is represented as Recruitment

• Death and Disengagement indicate that an active insurgent has left the group in one way or another.

14

Integrated Model

15

Incorporation of Submodels

Personnel

16

Two Radicalization Submodels

17

Based on method of Bartolomei, J., Casebeer, W., & Thomas, T. (2004)

Derived from SME input

Representing Culture Influences

• Complex socio-cultural computational models include both quantitative and qualitative data– Qualitative• Interviews with perpetrators• Opinions of SMEs• Broad social, psychological theories

– Quantitative• Demographics• Economic Factors• Surveys

18

Want to start understanding the interactions of all these influences What-If Analyses

Potential Events

Environmental Context

Scenarios

Models Evaluate

Impact of Change

What-if Analysis

Enable analysts to - Experiment with different sets of parameters, variables, and relationships- Explore results of

- Events within our control (military actions, policies, diplomatic decisions) - Events not within our control (weather, crop production, actions of others)

19

Modeling Approaches Across The Sciences

20

Figure from: G. Zacharias, J. MacMillan, and S. Van Hemel (eds), Behavioral Modeling and Simulation, National Research Council, 2008.

Qualitative Socio-Cultural Data

Psychological Theories

Cultural Descriptions

Political Attitudes,Influences

External Influences on Behaviors

Observed Behavior

PoliciesOpinions, Experiences of SMEs

Interviews with Individuals or Groups

Being ModeledSocial Theories

21

Quantitative Socio-Cultural Data

Survey Data

Census Data

Demographics

Economic FactorsEnvironmental Measurements

Psychometric Measurements

Polls

22

Representing Qualitative Data in a Computational Model

Representation ExamplesLandmark Values Left, Right, StraightLikert Values Strongly Disagree, Disagree, Neither Agree

nor Disagree, Agree, Strongly Agree“On a scale of 1 to 10 …”

Fuzzy Values Low, Medium, HighRelationships and Flows of Items, Attitudes, Information

Maturity Employment Stability

Decisions, Rules, CasesRelational Expressions, Equations

23

• Accepted measurement scales may not exist• Modelers may need to create fuzzy or landmark values for abstract concepts

What Kind of Data Does Your Model Need?

• What types of socio-cultural data does your model need?• What would you do if you never got it?• What are realistic substitutes and

workarounds?

24

Realistic Substitutes and Workarounds

Indirect ways to get at the information, perhaps with a

little more uncertainty

Other types of data that might be available to

substitute

Creation of data through laboratory experiments, and

synthetic data created by software generators

Perhaps a SME’s opinion included in the model would provide useful information if the data remains unavailable

Can we estimate the validity of this kind of surrogate data for use in a particular model?

25

Dealing with Uncertainty in Socio-Cultural Data

• Where does uncertainty occur?– Uncertainty in descriptions of attitudes, cultures, behaviors– Uncertainty in measurements (physical measurements or

survey instruments)– Uncertainty in descriptions of historical situations– Uncertainty inherent in human behavior

• Variations in human choices given the same culture and situation

• What approaches exist for dealing with uncertainty?– Probabilistic approaches, random variables– Representations of likelihood (other than strict probability)– Techniques for combining certainty values

26

Data Provenance

• Provenance: Where did the data come from, who collected it, how was it collected, under what circumstances, and what was the context?– The model user should understand the provenance of data in order

to determine how appropriate it is for a particular use.– Data from an authoritative source is not automatically more useful

than data from an unreliable source.– Data known with a high degree of certainty may not be the data that

leads to recognition of unexpected behaviors.– Once a piece of information has been confirmed and ‘hits the news’,

it may no longer provide information that can be acted upon.– In contrast, rumors about conspiracies, although potentially false,

are sources of information that may allow intervention to prevent catastrophic events.

27

Data Perishability and Missing Data

• Perishability: How long will this data be valid?– The importance of knowing when to remove data from a

model and recognizing that behaviors change and adapt

• Model representations in this space need to be those that can be made robust against missing features.

28

Summary

Develop separate federated

models

Different aspects of the domain

Different views (micro, meso,

macro)

Different time scales

Build iteratively

Allows insertion of new domain knowledge

Provides ability to change as your

knowledge of the model changes

Allows for correction and

results in better models

Build to allow for easy

extension

New domain knowledge

Changes in the system being

modeled

Changes in the intended use of

the model

Make limitations

explicit

Models are built with simplifying

assumptions

Based on the view or interpretation

of a modeler

Based on data or knowledge with

some level of uncertainty

Adopt Best Practices for Integrating Qualitative Data into Quantitative Models