norman lee johnson chief scientist referentia system inc. honolulu hawaii [email protected]

Norman Lee JohnsonChief Scientist

Referentia System Inc.Honolulu Hawaii

[email protected]:// CollectiveScience.com

Complexity: A Diverse Description

ONR Workshop on Human Interactions in Irregular Warfare as a Complex System Apr 2011

Diversity

Star Wars Novel fusion device Novel diesel engine Hydrogen fuel program P&G – Diapers Large scale

Epidemiology – Flu modeling

Biological threat reduction

Bio-Risk assessment Cyber security

Future of the internet

Self-organizing collectives Diversity and collective

intelligence– Finance applications

Effects of rapid change– Finance applications

Group identity dynamics– Coexistence applications

Leadership models

Bio-cyber analogies

My Background

Counterinsurgency in Iraq:Theory and Practice, 2007

David Kilcullen

Available online at: http://smallwarsjournal.com/documents/kilcullencoinbrief26sep07.ppt

http://smallwarsjournal.com/documents/kilcullencoinbrief26sep07.ppt

© David J. Kilcullen, 2007

Caveat: the logic of field observation in Iraq

+ Everyone sees Iraq differently, depending on when they served there, what they did, and where they worked. • The environment is highly complex, ambiguous and fluid• It is extremely hard to know what is happening – trying too hard to find

out can get you killed…and so can not knowing• “Observer effect” and data corruption create uncertainty, and invite bias • Knowledge of Iraq is very time-specific and location-specific• Prediction in complex systems (like insurgencies) is mathematically

impossible…but we can’t help ourselves, we do it anyway

+ Hence, observations from one time/place may or may not be applicable elsewhere, even in the same campaign in the same year: we must first understand the essentials of the environment, then determine whether analogous situations exist, before attempting to apply “lessons”.

Diversity

Levels of Social complexity

slime molds

“low” social

insects

“high” social

insects

social mammals

“low” apes

“high” apes humans

Identity, diverse, decentralized, collective survival and problem solvingCollectively adaptable, self-organizing, emergent properties

IndividualSelf-awareness &

Consciousness

Collective: Memory, intelligence, deception, tools

Individual: High intelligence, deception & emotions, tool making

From a workshop on “The Evolution of Social Behavior” which covered a wide range

of social organisms

Example: All social organism when stressed are

“programmed” to copy the behavior of others in the

“organism”

Complex Human Dynamics: Three Challenges

System Analysis

• How different analysis perspectives can simplify

the “complexity” of the system attributes or

dynamics?

Behavioral-Social Models

• How abstract models can help simplify the

“complexity” of individuals components

• OR provide relationships between components?


“Getting it” is not enough

“[This] is a political as well as a military war…the ultimate goal is to regain the loyalty and cooperation of the people.” “It is abundantly clear that all political, military, economic and security (police) programs must be integrated in order to attain any kind of success”

Gen William C. Westmoreland, COMUSMACV, MACV Directive 525-4, 17 September 1965

Understanding by leaders is not enough: everyone needs to understand, and we need a framework, doctrine, a

system, processes and structures to enact this understanding.

Complex Human Dynamics: Three Challenges

System Analysis

How difference perspectives can simplify the

“complexity” of the system attributes or dynamics?

Behavioral-Social Model

How abstract models can help simplify the “complexity” of individuals components OR

provide relationships between components?

Tools for Decision Makers

How the above understandings lead to actionable knowledge for

decision makers?

Complex Human Dynamics: Three Solutions

System Analysis • Connection between local global

Behavioral-Social Model Features• Individual behavior models

Tools for decision makers

Habitual repetition: Classical conditioning theory (Pavlov), Operant conditioning theory

(Skinner) Individual optimization of decision:

Theory of reasoned action (Fishbein & Ajzen), Theory of planned behavior (Ajzen)

Socially aware: Social comparison theory (Festinger), Group comparisons

(Faucheux & Mascovici) Social imitation:

Social learning theory (Bandura), Social impact theory (Latané), Theory of normative conduct (Cialdini, Kalgren & Reno)

CONSUMAT model -- Marco Janssen & Wander Jager – Netherlands

Individual preference + Social drives + Options + Rationality = ?

What drives the changes?

Repeater Deliberator

Imitator Comparer

Satisfied Dissatisfied

Uncertain

Certain

Historicalcomparison

Increasedstress

Individual Behavior + Network = Global Dynamics

1000 Consumers with the same behavioral tendencybuying 10 products on a small-world network

Population of “Repeaters” - satisfied and certain

0

0.1

0.2

0.3

0.4

0.5

0.6

0 10 20 30 40 50 60 70 80 90 100time steps

market shares of products

Few products of equal distribution - highly stable

Repeater

Closest to “Rest state”

Population of “Imitators” - satisfied but uncertain

0

0.1

0.2

0.3

0.4

0.5

0.6

0 10 20 30 40 50 60 70 80 90 100time steps

Few products of unequal distribution - highly stable

Imitator

Transitional individual => social state

Population of “Deliberators” - dissatisfied but certain

0

0.1

0.2

0.3

0.40.5

0.6

0 10 20 30 40 50 60 70 80 90 100time steps

High volatility on all products

DeliberatorClosest to Homo Economicus

High rationality, low social

Population of “Comparers” - dissatisfied & uncertain

0

0.1

0.2

0.3

0.4

0.5

0.6

0 10 20 30 40 50 60 70 80 90 100time steps

market shares of products

Volatility over long times on few productsBut difficult to maintain - high energy state

ComparersSocial and Rational

“habitual” agent

Highly stable withsustained diversity

Homo Economicus

High volatility

Social and Rational

Longer time volatility - difficult to sustain

Socially driven

Highly stable - decreased diversity

Repeater Deliberator

Imitator Comparer


System Analysis • Connection between local global

• simple models can lead to complex global behavior

• Study of system thresholds


• Drivers & threshold transitions• Habitual behavior

• Importance of habitual behavior & individual threshold transitions

Tools for decision makers• Focus on system thresholds first

• rather than quantifying the details between threshold states

Rat Studies of Maximum Carrying Capacity

Social order system can carry 8 times the optimal capacity before going over the threshold.

NIMH psychologist John B. Calhoun, 1971

Control group - no “rules” => Your worst nightmare

One “social” rule =>Cooperative social structure

Both systems loaded to 2 1/2 times the optimal capacity.

Diversity

Simple Ant Foraging Model

Key concepts:Emergence, Productivity, Diversity, Structure

Using NetLogo

Collective informationEvaporationDiffusionAgent internal state: Current direction Have food? Three rules of action:Carry foodDrop foodSearch n “Productive men”

n “Salaried men” n Innovator n Collective structure

Nest

Food supply

Diversity

Quantified Environmental Change

Infinite source moves at afixed radius

and fixed angular velocity

Diversity

Slowly changing environment

Productivity is only slightly less than an unchanging source

Herd effect allows for quick utilization of new resource location

Innovators are important at all times by sustaining optimal performance of the collective

0.00

0.20

0.40

0.60

0.80

1.00

1.20

0 1 2 3 4 5 6 7 8Rate of environmental change (tenth of degree/time unit)

Production rate (food units/time units)

Collective

Individual

Formative

Co-Operational

Condensed

Food Production Rate

Effect of Rate of Change on System Development


System Analysis • Connection between local global• Study of system thresholds• Developmental perspective• Emergent behavior via multi-level

analysis



• Emergent problem solving

Tools for decision makers• Focus on system threshold changes first

Diversity

Structural Efficiency - Boom and Bust

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

Rate = 0.3

Structural efficiency

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0 1000 2000 3000 4000 5000

Structural efficiency

Time (time unit)

Rate = 0.0

Rate = 0.8

Lower average production crash avoidance strategy

Bust is proceeded by increased production

Greater minimums and maximum when compared to extreme rates!

Diversity

Tota

l pro

duct

ion

(uni

ts o

f foo

d)

Time (time units)

For the slowest rate of change

0

500

1000

1500

2000

2500

0 500 1000 1500 2000 2500 3000

transient structure

sustained structure

Combination of Sustained Structure and Change

How does the retention of structure change the collective response?

Suggests that fixed evolutionary adaptations lead to inefficiencies in the presence of even small rates of change

What would be the effect of a faster worker?

What would be the effect of mass communication?

Prigogine’s Laws of stasis, change and evolution:

Original observations for chemical systems1. Equilibrium states are an attractor for

non-equilibrium states. 2. System near equilibrium cannot evolve

spontaneously to generate spatial−temporal (dissipative) structures

3. As the system is driven far from equilibrium, it may become unstable and generate spatial−temporal structure from nonlinear kinetic processes associated with flows of matter and energy.

4. The possibility of new structures is determined by the system size.

5. Bifurcation introduces “history” into the model (trajectories are replaced by processes). Every description of a system which has bifurcations will imply both deterministic and probabilistic elements.

Generalized observations 1. Perturbed systems will return to

their normal state. 2. The mechanisms for permanent

change are not accessible to systems near equilibrium.

3. Bang on a system hard enough, existing structures can be replaced by new structures.

4. The evolution of new structures are limited by system size.

5. Multiple outcomes change certainty into probabilities, and require a fundamentally different approach.


System Analysis • Connection between local global• Study of system thresholds• Developmental perspective• Emergent behavior via multi-level

analysis• Optimization vs. robustness



• Emergent problem solving• Individual and collective structure

Tools for decision makers• Focus on system threshold changes first

Structure in a system increases over time for decentralized, self-organizing collectives (nature, societies, technologies)

Structure(e.g., the rules

required to “run” the system)

Time

Structure declines because the number of new rules are limited by past rules.

Structure increases first by components developing structure

Structure increases rapidly as components build structure together

Diversity

The Structure of Structures

Description Determinesevolutionarypath directly

Retainedonce

expressed

Alwaysexpressed

Origins:Random,

Direct,Emergent

Experiential or transient featuresLearning – Ant position

R

Shallow surface featuresColoring – Collective solution

X R

Deep surface structures (frozen ``accidents’’)Specific DNA coding

X X D,E,R

Deep system structures (frozen organization)Digital coding, nucleus formation

X X X D,E

Features reflecting fundamental lawsHydrogen bonding

X X X D

Structures direct the evolution of the system by creating and limiting potential optionsTheir definition depends on the time constant of exogenous/endogenous change.

Options around Structure also change

Structure(the rules


Time

Little initial structure means few Options

Options are greatest when structure connects the components

These ideas are captured by researchers studying “infodynamics”

Options are the free choices both created and limited by the structure (example: the rules of chess create an “environment” where many options are possible - while also limiting what choices are available)

Options

Options are reduced as more structure restricts options

Difference between Options and Diversity

Structure

Time

Diversity and Options are low

Diversity and Options are high


• Diversity is the the unique variety in the system• Options are when the diversity has multiple expressions of

differences, often expressed as multiple connectivity in the network

Options

Diversity may be very high but Options are low

Adaptability and Robustness of System

Structure

Time

Diversity

Both adaptable and robust


• Robustness is a system-level ability to sustain performance in the presence of change

• Adaptability is a component level ability to accommodate change

Options

Not robust or adaptable, but existing components can be rearranged for new features

Robustness achieved by component replacement

Diversity

Collective Response to Environmental Change

Unimpeded development

Innovators are essential

Collective actions lead to inefficiencies

Potential system-wide

failure

Condensed (optimization of

collective)

Co-Operational (synergism from

individuals)

Formative (creation of

individual features)

Featureless

Stable“no change”

Change slower than collective

response

Change faster than collective

response

Change faster than individual

response

Rate of Environmental Change

Stag

es in

Dev

elop

men

t

Diversity

Why Care about Structure-Options?

Studies of thresholds in structure:

– Prigogine’s Laws of Stasis, Change and Evolution– Joseph Schumpeter’s Creative Destruction – Foster and Kaplan "Creative Destruction: Why

Companies that are Built to Last Underperform the Market - And how to Successfully Transform Them”, 2001

– John Padgett life’s work on innovation in the Florentine (and world) finance system

Dynamic “structural” thresholds do the same


System Analysis • Connection between local global• Study of system thresholds• Developmental perspective• Multi-level analysis & emergence• Optimization vs. robustness• Interplay of structure and options



• Emergent problem solving• Individual and collective structure

Tools for decision makers• Focus on system threshold changes first• Capture structure and options:

• What has to be removed before change can occur

• Diversity is not the same as options!


“15. Do not try to do too much with your own hands. Better the Arabs do it tolerably than that you do it perfectly. It is their war, and you are to help them, not to win it for them. Actually, also, under the very odd conditions of Arabia, your practical work will not be as good as, perhaps, you think it is.”

T.E. Lawrence, “Twenty-Seven Articles”, The Arab Bulletin, 20 August 1917

Remember article 15

Diversity

Expert Performance in Finance

Why can’t financial experts outperform the S&P 500 “collective” – good + bad – consistently?

• Professional money managers fail to beat the S&P 500 at an average rate of 70% per year.

• 90% trail the S&P over a 10-year period.

• Over decades are only a few – Soros, Miller, ….

“These are the people who have more knowledge and more training than the vast majority of investors. And yet, neither the superior knowledge nor the superior experience helps them in the long run.”

Bill Mann, TMFOtter

The ant colony (and individuals) finds the shortest path

Nest

Food

Nest

Food How does it work?

Ants Solving “HARD” problems

Start

EndIn “Learning” the maze, individuals create a diversity of experience.

A Model for Solving Hard Problems

How can groups > solve hard problems,> without coordination,> without cooperation, > without selection?

The Maze has many solutions > non-optimal and optimal.

Individuals > Solve a maze> Independently> Same capability

When individuals solve the maze again, they eliminate “extra” loops

But because a global perspective is missing, they cannot shorten their path. This is were diversity helps.

How collectives find the Shortest path

Paths of three ants Collective path

Unlike in natural selection, no one individual is the fittest!

Ensemble (Averaged) Behavior

.

Individuals in Collective Decision

Nor

mal

ized

num

ber o

f ste

ps

0 5 10 15 20

0.9

1.0

1.1

1.2

1.3

0.8

Average Individual

Using novice information, with two different collections

Usingestablishedinformation

Performance correlates with high unique diversity

Diversity

Expert Performance & Complexity

Where Experts Have Value

Simple ComplexDomain Complexity

Valu

e of

Exp

erts

Michael Mauboussin - Legg Mason Capital Management

Valu

e of

Col

lect

ives

Complexity Barrier

From a workshop on Complex Science for the Physician’s

Alliance

Effect of Complexity in Stable Systems

Structure(the rules


time

System goes to optimization via“expert” route

“Complexity Barrier” requires

Collective Solutions

X

System goes to optimization via

“collective” route

Collective Error = Average Individual error

minusPrediction Diversity

“The Difference:How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies”

Diversity

Collective Performance

Where Experts Have Value

Simple ComplexDomain

Valu

e of

Exp

erts

Michael Mauboussin - Legg Mason Capital Management

Valu

e of

Col

lect

ives

Collective Error = Average Individual error

minusPrediction Diversity

Diversity

Options in infrastructure, societal structure, economies, etc.

Collectives in complex environments

begin

end• • • • • • • •

In complex domains: • People beginning points differ• Their final goals may differ• But local paths can overlay and find synergy


Why counterinsurgency is population-centric

+ This is not about being “nice” to the population, it is a hard-headed recognition of certain basic facts, to wit:• The enemy needs the people to act in certain ways (sympathy,

acquiescence, silence, provocation) -- without this insurgents wither• The enemy is fluid; the population is fixed – therefore controlling the

population is do-able, destroying the enemy is not• Being fluid, the enemy can control his loss rate and can never be

eradicated by purely enemy-centric means (e.g. Vietnam VC losses) • In any given area, there are multiple threat groups but only one local

population – the enemy may not be identifiable but the population is.

Terrain-centric and enemy-centric actions are still vital and crucial to success. Enemy and Terrain still matter, but Population is the key.





• Emergent problem solving• Individual and collective structure• Conditions for synergy/conflict• Group identity models

Tools for decision makers• Focus on system threshold changes first• Capture structure and options• New social consensus tools

Diversity - source of conflict or synergy?

Diversity can lead to synergy when collectives have:• Common goals

• Common group identity

• Common worldview (agreement on options), but with different preferences or goals

Otherwise, diversity can lead to competition and conflict

Mor

e re

stric

tive


Governance, development, democracy are not universal goods

“There is no such thing as impartial governance or humanitarian assistance. In this environment, every time you help someone, you hurt someone else.”

General Rupert Smith, Commander UNPROFOR 1995

The enemy will perceive actions by political staff, NGOs, economic and development staffs, PRTs and government officials as a direct challenge to

grass-roots control over the population, and will react with violence

Diversity

Why Care about Group Identity?

Social organisms have a strong drive to form group identity:

“... experiments show that competition is not necessary for group identification and even the most minimal group assignment can affect behavior. ‘Groups’ form by nothing more than random assignment of subjects to labels, such as even or odd.”

Group Identity can be the dominant factor of behavior: “Subjects are more likely to give rewards to those with the same label

than to those with other labels, even when choices are anonymous and have no impact on their own payoffs. Subjects also have higher opinions of members of their own group.”

Akerlof, G. A. and R. E. Kranton (2000). “Economics and Identity.” Quarterly Journal of Economics 115(3): 715-753.

Identity: Assertions and Definitions

Identity

Group Identity == Mechanism of Group Immunity Common definition: if someone does something to a person in your

identity group, it is the same as if they did it to you. Working definition: Identity is the individual behavioral bond/process that

creates a “group self” that has all of properties of an individual self. Assert: Group identity in higher social organisms can be an abstraction

that detaches from the origin of the identity group.

Despite the long-standing recognition of the importance of identity in social systems, most studies of identity are observations of identity's influence on individual and group behavior, rather than understanding the processes by which identity forms and modifies behaviors.

Questions from an Identity Perspective

Identity

What are the characteristics of identity groups? What are their dynamics? Formation, stability, coalescence,

expression, polarization, influence, dissolutionHow does group identity affect the acceptance, adaptation, spread and exploitation of an

idea? Does the rapid spread of an infectious idea (e.g., a fad) require a common group

identity? What are the conditions that cause identity groups to become destructive to other

groups or to factions within a group? How does a ”leaderless group” group use identity to self-organize? How does insurgent news/press of “violence on self” polarize the “self”? How does diversity affect identity formation? Performance? Stability? As diversity

decreases, will tolerance decrease? What are the conditions necessary for coexistence to emerge between identity groups? How does one really build a democratic nation from fractionalized groups?

Answers, or at least a beginning of an understanding, of questions like these will help to inform policy regarding intra and international negotiations and other actions designed to bring about the resolution of conflict. It will aid them in identifying unintended consequences of hasty actions, such as the use of extreme force. Or, how to prevent violent conflict and grow the conditions for peace.

Characteristics of Identity Groups

Identity Groups (IDGs) express a common worldview – an understanding of how the world works: what are options and what is forbidden

IDGs have a shared, unspoken knowledge that is typically unknowable outside the IDG

IDGs often have symbols of association such as dress or language differences, often unobserved by others

IDGs – when in larger and sustained groups – develop culture and civilizations

While most of us are born or develop within existing IDGs, we also form many identity groups during our lives.

Identity Groups under Stress

Stress – a heightened state of anxiety about one’s current state that might originate from outside the IDG (e.g., oppression) or from within (e.g., internal dissension) – can cause the IDG to act as a single organism (“circling up the wagons”)

Stressed Identity Groups:Are more likely to reject ideas coming from outside the IDGOppress, reduce, or prohibit expression of diverse ideas within the IDGMake irrational actions that are potentially self-destructiveCan “dehumanize” individuals and groups that represent opposition IDGsAre in a state that can lead to polarization, particularly if “outside” IDGs are

well defined, are in opposition and are creating the stress Can be strongly influenced by a leader (or idea!) that represents the IDG,

particularly potential “martyrs” that have received the brunt of the oppression or violence from the opposing IDGs.

Application - Tipping Point

Law of the FewIdentity largely determines the social networkIdentity coherence determines the success of

trendsetters to represent the the group and the sticky idea

Trendsetters often span multiple identity groups

Stickiness FactorThe “stickiness” strongly correlates with the

resonance with identity Ability to jump across multiple identity groups

determines widespread propagationAn idea that is sticky to an opposing identity group

will be aborted and demonized.





• Emergent problem solving• Individual and collective structure• Conditions for synergy/conflict• Group identity models• Descriptive vs. predictive models

Tools for decision makers• Focus on system threshold changes first• Capture structure and options• New social consensus tools• Tools are changing - data rich vs. poor• Validation requirements

•Prediction of collective behavior is generally easier at extremes of diversity or variation

Diversity and Collective Prediction

LowDiversity

HighDiversity

Locally and

Globally Predictabl

e

Globally Predictabl

eUnpredictabl

e

•How does this translate to distribution functions?

Diversity and Collective Prediction

LowDiversity

HighDiversity

Locally and

Globally Predictabl

e

Globally Predictabl

eUnpredictabl

eProblem distributions: • Discrete distributions • Multi-modal distributions • Long-tailed distributions

(e.g., power law, instead of Gaussian

statistics)

p(ø)

0 1

Analysis - Increasing levels of discovery: • Ontology or qualitative characterization • Statistical characterization• Dimensionless functionality (correlations)• Scaling - self-similarity – fractal structure• Descriptive-predictive “Laws”• Functional relationships• Static• Dynamic (governing equations of change)

• Higher moments (variation within)• Error generation - uncertainty quantification

Origin of the model or “The Theory”

Structure and options

Averages and outliers

Data generation

Discovery





• Emergent problem solving• Individual and collective structure• Conditions for synergy/conflict• Group identity models• Descriptive vs. predictive models

Tools for decision makers• Focus on system threshold changes first• Capture structure and options• New social consensus tools• Tools are changing - data rich vs. poor• Validation requirements • Cost-benefit assessment with transparency

and uncertainty quantification

Diversity

Norman L. [email protected]

http://CollectiveScience.com

Questions?

Some References on Prediction and Complexity

Shalizi, Cosma R., “METHODS AND TECHNIQUES OF COMPLEX SYSTEMS SCIENCE: AN OVERVIEW”, Chapter 1 (pp. 33-114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine (New York:Springer, 2006) http://arxiv.org/abs/nlin/0307015

Farmer, J. Doyne & John Geanakoplos, “Power laws in economics and elsewhere”, DRAFT April 4, 2005 (chapter from a preliminary draft of a book called “Beyond equilibrium and efficiency”) - Contact the authors for a copy.

Farmer, J. Doyne, “Power laws”, Santa Fe Institute Summer School June 29, 2005. Contact the author for a copy.

Holbrook, Morris B.. 2003. "Adventures in Complexity: An Essay on Dynamic Open Complex Adaptive Systems, Butterfly Effects, Self-Organizing Order, Co-evolution, the Ecological Perspective, Fitness Landscapes, Market Spaces, Emergent Beauty at the Edge of Chaos, and All That Jazz http://www.amsreview.org/articles/holbrook06-2003.pdf

White, Douglas R., “Civilizations as dynamic networks: Cities, hinterlands, populations, industries, trade and conflict”, European Conference on Complex Systems Paris, 14-18 November 2005. http://eclectic.ss.uci.edu/~drwhite/ ppt/CivilizationsasDynamicNetworksParis.ppt

For exceptional talks on Complexity in financial systems, see the Thought Leaders Forums:

• http://www.leggmason.com/thoughtleaderforum/2006/index.asp for 2003-2006

• http://www.capatcolumbia.com/CSFB%20Thought%20Leader%20Forum.htm for 2000-2003

http://arxiv.org/abs/nlin/0307015

http://complexsystems.lri.fr/

norman lee johnson chief scientist referentia system inc. honolulu hawaii [email protected]

Documents

complex systems

complex dynamical systems

real complexity theory

complex adaptive systems

complexity theories

word complexity

bias knowledge of iraq

irregular warfare