1 wastell asp-sintella department of psychology in collaboration with the centre for policing,...

1 Wastell ASP-SINTELLADEPARTMENT OF PSYCHOLOGY IN COLLABORATION WITH THE CENTRE FOR POLICING, INTELLIGENCE AND COUNTER TERRORISM

Investigating information selection and utilization in complex, ambiguous, and uncertain information environments.Colin A WASTELL PhDAssociate Professor Colin Wastell

has been involved in Psychological research for nearly 25 years. He completed his PhD in 1993. He joined the Royal Australian Naval Intelligence Reserve branch in 1997 and has served in N2, ASTJIC [1999-2001] and JOICAUST. In 2003 he was Research Leader of the Information Warfare Studies Group, in DSTO’s Information Networks Division. Since 2000 he has been interested in, and is currently conducting empirical research into, human reasoning and decision-making in high impact ambiguous complex environments.

He has been involved in research examining the utilisation of different reasoning processes in intelligence analysis and risk perception.A/Prof Wastell has presented at the Society for Judgment and Decision Making Conference in Toronto, Canada and Seattle Washington as well as The International Conference on Thinking in Venice, Italy and forthcoming in London. He is a member of the faculty of the Psychology Department and presently on secondment as the Director of Research for the Macquarie University Centre for Policing, Intelligence and Counter-Terrorism, as well as being an associate of Risk Frontiers.


Investigating information selection and utilization in complex, ambiguous, and uncertain information

environments.

Analysis Simulation Project [ASP] Colin Wastell PhD and Piers Duncan LLM MEd Director of Research, P.I.C.T. and Associate of P.I.C.T Centre, Macquarie University

Psychology Department, Macquarie University

Project Team Members

The project was funded by the Australian Government’s Department of Prime Minister and Cabinet and Macquarie University’s Centre for Policing, Intelligence and Counter Terrorism, whose support is gratefully acknowledged.

Alex Wearing PhDPsychological Science, University of Melbourne

Nicole Weeks BPsych (Hons)

Psychology Department, Macquarie University


Recent problems in Intelligence Analysis• The major conclusions of the reports into Intelligence Agencies in the West (Flood;

Butler; 911; WMD) have largely focused on management of the process and interagency cooperation.

• The reports identify problems in ;– Reliance on stereotypes– Lack of critique of assumptions– Heuristic and bias effects in information retrieval and evaluation

• Organisational problems include;– Groupthink– Alignment with the ethos and worldview of the organisation

• So why do the problems keep appearing?– Is the task inherently unpredictable?– Are the resources sufficient to meet the task?– Was there political interference?

• Richards Heuer (1999) of the CIA says the origins of the problems found repeatedly by reports into intelligence agencies most likely lies in the architecture of the human mind. Tetlock (2005) also argues that how analysts think is just as important as what they think.


Characteristics of complex, dynamic Information Environments1. Vast quantities of ‘data’

2. Time pressure is varied and often of a very high tempo [see Wastell, 2010].

3. Socio-political-organisational context – political and social expectations

4. Situations change with or without intervention Need to find a method that allows us to explore these external factors in combination

with the internal psychological factors that analysts bring to their work in a systematic and experimental manner.

Simulation• Simulations enable a mid point in fidelity to be achieved between field and laboratory

studies ie closer to real world environments but with the ability to systematically manipulate variables of interest [See Omodei & Wearing 1995]

• There has been a growing use of the method across a range of complex decision making environments and tasks.

Design requirements – the program needed to be able to incorporate;1. A number of interrelated variables2. Ambiguity & uncertainty in key variables3. Multiple decision and interpretation alternatives4. Dynamic variability, ones own actions alter the situation5. Environmental changes over time.


A Model of the Intelligence Information ProcessingA Model of the Intelligence Information Processing

Prepare a report on threat posed by movement X

Intelligence Example

* Previous reports prepared by the agency

* Intelligence Databases such a Janes®

* General knowledge databases e.g. UN reports

File notes

Draft versions of report

Report


ASP – SINTELLA* Development Stages1. Develop the computer software [based on Mouselab Web] including:

i. Information trackingii. Recording of data

2. Populate the program with specified information:i. Contained to 64 [8x8] information blocksii. Information block between 50 and 130 wordsiii. Design categories to reflect selected [relevant] information

3. Design and conduct tests of the program to establish general operating characteristics and to refine data capturing capability.

4. Evaluate program against training criteria in terms of ease of use and acceptability to operators.

5. Design and build a number of variants to test different aspects of information processing [see Wastell et al (2009) and Weeks et al, (2012)]. Two variants have been developed to date:i. ASP – SINTELLA: Country Assessmentii. ASP – SINTELLA: Murder Investigation

[*SINTELLA stands for Simulation of INTELLigence Analysis which was the first domain in which ASP programs were applied]


ASP-SINTELLA Screen: Country Assessment Variant


Country Assessment - Parette

InformationBoardCountry BriefTravel advisory


ASP-SINTELLA: Country Assessment Variant

1. Travel Advisory for fellow University Students– Participants are asked to write a 250 word Travel Advisory for

fellow students.– The information on the country is then used to ascertain relevant

information2. Visit Appraisal for Minister of Trade [or NGO]

– Participants take on the role of a junior staff member in the Federal government’s Trade Ministry.

– They are directed to prepare a mini country brief to assist the Minister with an upcoming visit.


ASP-SINTELLA Screen showing information access and use.

1994 saw the election of reform minded President Cecile Lescon.

President Cecile Lescon has ruled the country since 1994


SINTELLA Probability estimation ScreenCountry Assessment Travel Advisory task


Study 1Cognitive Predispositions’ impact on estimations of threat and

attitude toward Home country[See Wastell, Etheridge, McMahon, Lucas & Hartley, 2011]

1. Wastell et al (2011) found that cognitive predisposition [often called biases] influenced estimations and decisions in the minerals exploration industry.

2. Building on Wastell et al 2011 the focus of this experiment was to explore the possibility of the same phenomenon in information analysis in a political context.

3. Participants [paid $20] were required to create a brief [250 word] document outlining the level of threat to a government delegation visiting the fictitious country of Parette. In addition participants were required to estimate the general attitude of Parettians toward Australia

4. Participants were given 40 minutes to select and condense the information provided from the matrix into a report.

5. The main measure of theoretical interest was Kruglanski’s “Need for Closure” (Kruglanski, Webster & Klem, 1993) and as well as demographics such as gender. In addition information accessing behavior [e.g. distinct cells opened; repeat cell openings; time in cells etc] and measures of the document produced [e.g. length of report] were also recorded.


Study 1 ctdCognitive Predispositions’ impact on estimations of threat and

attitude toward Home country7. Attitude toward Australia

a) The range of attitude scores was from 20 to 100 (Mean= 55.46, sd=20.30) with higher scores indicating a more positive attitude toward Australia.

b) Regression analysis again incorporated the full complement of variables.c) The final model consisted of only one variable with an F(1,37) = 4.49, p

=0.041, R2 = 11%.d) The one variable was NFC Closedmindedness with a standardized beta

coefficient of -.33 indicating that higher scores on closedmindedness predicted more negative attitude toward Australia.

This work is currently revised and re-submitted to the journal “Intelligence and National Security”


Percentage of participants who opened each cell

0-

20%20-40% 40-60% 60-80% 80 -100%

Comparison of Cell utilization by Task Type


ASP-SINTELLA: Murder Investigation Variant1. The second variant of ASP-SINTELLA is based on the scenario of a

detective investigating a murder.

1. The Country Assessment variant of ASP-SINTELLA is focused on the selection of information with a particular task in mind that is the provision of advice to either students or Senior Political operatives. The Murder investigation variant emphasizes a decision based on the possibility of a correct answer but the information is indicative rather than conclusive.

2. Murder Investigation Taski. The task is to select the most likely suspect who is considered to have

committed the murder.ii. Write a report providing a rationale for the choice of the most likely

murder suspectiii. Participants are provided with information including witness

statements, forensic evidence and general background and contextual information


ASP-SINTELLA Screen: Murder Investigation Variant


Murder Investigation task

InformationBoardMurder investigation


Study 2Confirmation Bias in the context of a murder

investigation1. Each information cell was classified by two raters independently

as to the contents’ relevance as to the guilt or innocence of a particular suspect

2. Examined the information processing behavior of participants across two experiments. [NB Participants were paid $20 for their participation; University students across a range of disciplines not just Psychology]

3. Participants nominated a most likely suspect as the perpetrator of the murder as well as their confidence in their selection.

4. Confirmation bias is the selection and concentration by participants on those information cells that were consistent with their choice of perpetrator to the neglect of inconsistent cells


Study 2 ctdConfirmation Bias in the context of a murder investigation

5. Experiment 1 [N=50] Based on FINAL Suspect chosen.• The success rate for this sample was 42% (21/50). “Success” is defined as the

identification of the most likely perpetrator based on an ACH procedure utilizing ALL the evidence.

• The number of cells opened by participants who correctly [C] identified the suspect did not differ statistically from those who were incorrect [IC]. Means - C: 96.92; IC: 82.90, t=1.52, p=.135.

• Those who chose incorrectly opened a greater proportion of suspect-consistent cells than other-suspect cells [Correct did not]Means – CS: 53.47; OS: 36.49, t=4.11. p<.0005, d=1.04.

• The incorrect group also visited the workspace more frequently after access CS cells than OS cells [Correct did not].

Means – CS: 32.51; OS: 23.33, t=2.64 p=.014, d=0.51


Study 2 ctd

Confirmation Bias in the context of a murder investigation6. Experiment 2 [N=44] Added intermediate assessment of

suspect selection every 10 cells accessed.• The success rate for this sample was 36% (16/44)• The number of cells opened by participants who correctly [C] identified the

suspect did not differ statistically from those who were incorrect [IC]. Means - C: 86.0; IC: 81.07, t=.553, p>.05.

• Those who chose incorrectly opened a greater proportion of suspect-consistent cells than other-suspect cells [Correct did not]Means – CS: 54.40; OS: 42.23, t=2.34. p=.028, d=.52.

• The incorrect group also visited the workspace more frequently after access CS cells than OS cells [Correct did not].

Means – CS: 46.45; OS: 29.37, t=3.89 p=.001, d=0.78Interruptions: Classify as Early (<5), Middle (6<x<9), Late (>9) or Non-settlers

Success 2/12 (16%) 6/14 (43%) 5/8 (63%) 3/10 (30%)NB Same pattern of information acquisition in incorrect early settlers as Exp 1.


Study 2 ctdConfirmation Bias in the context of a murder

investigationComment

Experimental participants displayed information accessing and processing behavior consistent with our definition of Confirmation bias in that those who settled early on a suspect selected and acquired information for their chosen suspect irrespective of how much information they accessed post their choice of perpetrator. NB A fuller report of this research is currently under invited revised re-submission to the “Journal of Investigative Psychology and Offender Profiling”.


Future Developments and Applications • Technical development of the program including the building

of a stand alone version of the program with basic statistics and analytical capability. With content and structure that can be customized to suit a wide variety of contexts.

• Research with Intelligence and Law enforcement organizations to assist with quality enhancement of analytic product [To what extent do biases impact on assessments?].

• Evaluation of training programs and methods to demonstrate what, if any, value add they specifically contribute [Currently

developing and verifying test-retest versions of ASP programs].• Use ASP [stand alone version] to conduct distributed analysis within

organizations – harnessing the “wisdom of the crowd”, an extension of the Cell utilization procedure [Slide 15].

• Develop International collaborations on future developments of the ASP capability and its applications [UK and US agencies].


Thank [email protected]

Questions ?or

Comments !or

Expressions of interest in collaboration

The project was funded by the Australian Government’s Department of Prime Minister and Cabinet and Macquarie University’s Centre for Policing, Intelligence and Counter Terrorism, whose support is gratefully acknowledged


Bibliography

Butler, (2004). Review of intelligence of weapons of mass destruction. London: House of Commons. Retrieved 21 April, 2006, from http://www.butlerreview.org.uk/report/index.asp

Flood, P. (2004). Report of the inquiry into Australian Intelligence Agencies. Canberra: Commonwealth of Australia. Retrieved April 21, 2006, from http://www.dpmc.gov.au/publications/intelligence_inquiry/docs/intelligence_report.pdf

Heuer, Jr, R. (1999) Psychology of Intelligence Analysis. CIA: Center for the Study of Intelligence.

Kruglanski, A., Webster, D., & Klem, A.(1993). Motivated resistance and openness to persuasion in the presence or absence of prior information. Journal of Personality and Social Psychology, 65, 861-876.

Omodei, M. M. & Wearing, A. J. (1995). The Fire Chief microworld generating program: An illustration of a computer-simulated microworlds as an experimental paradigm for studying complex decision making behavior. Behavior Research Methods, Instruments, & Computers, 27(3), 303-316.

Tetlock, P.E. (2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.

The 9/11 Commission Report (2004). The 9/11 Commission Report. National commission on terrorist attacks upon the United States. Retrieved April 21, 2006, from http://www.9-11commission.gov/report/911Report.pdf

Wastell, C. A. (2010). Cognitive predispositions and Intelligence Analyst Reasoning. International Journal of Intelligence and CounterIntelligence, 23, 449-460.

Wastell, C. A., Etheridge, M., McMahon, M., Lucas, G., & Hartley, L. (2011). The impact of cognitive predispositions on exploration decisions in the minerals industry. Applied Cognitive Psychology. 25,469-479.

Wastell, C. A., Weeks, N., & Duncan, P. (2009). SINTELLA: Simulation of INTELLigence Analysis. Journal of Policing, Intelligence and Counter Terrorism. 4, 71-82.

Weeks, N., Wastell, C. A., Taylor, A., Wearing, A & Duncan, P. (2012). Tracing decision processes in complex, ambiguous, information-rich environments. International Journal of Psychological Studies 4(1), 158-173.

WMD (2005). Report to the President of the United States, The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction. http://www.gpoaccess.gov/wmd/index.html

1 wastell asp-sintella department of psychology in collaboration with the centre for policing,...

Documents

intelligence analysis

intelligence agencies

macquarie university

information selection

macquarie university

macquarie universitys

information retrieval

aprof wastell