perspectives on sensemaking simon attfield visual analytics summer school, 2010 1

Perspectives on Sensemaking

Simon AttfieldVisual Analytics Summer

School, 2010

Picture by Ed Sanders

http://drlillianglassbodylanguageblog.wordpress.com

http://news.bbc.co.uk/1/hi/england/london/3732169.stm

Sensemaking is...

• The integration of information and knowledge to form an interpretation (Attfield, last weekend)

Cognitive dissonance

The fox and the grapes ‘adaptive preference

formation’

'Oh, you aren't even ripe yet! I don't need any sour grapes.'

Picture credit - Milo Winter

A search for coherence

informationdata

knowledgeexpectationexperience

valuesaspirations

Cognition - How do people reason with information? How do they find meaning, reconstruct a domain, reach conclusions, and derive new puzzles and questions?

Social interaction – How do people interact around information? How do they share, convince, question, agree, cajole and debate?

Behaviour - How do people manage information? How do they adapt and shape their environments in order to achieve cognitive and social processes?

Tools - What can all of this say about the resources and functionality that can provide the leverage to do these things more naturally, invisibly and effectively?

Key questions

Expectation and language

• Battered Child Syndrome– Up to 1940’s no concept of battered child

syndrome.– Late 1940’s - a surprise - a paediatric radiologist

noticed a few cases of child injuries not adequately accounted for by parents medical ‘histories’.

– Discrepancy explained away.– More cases noticed.– In 1953 a comprehensive report identified 749

cases.– Within a few years laws introduced mandating

reporting. Weick, 1995

How do we makes sense of the world with very little information? –

schemata• Data structures for representing generic

concepts stored in memory• Introduced into psychology by Bartlett • ‘The building blocks of cognition’ (Rummelhart, 1980)

• Variables that can be bound to an instantiation – e.g. Buying• A schema is a ‘theory’ – goes beyond our perception• Comprehension is a process of hypothesis testing

http://venturebeat.com/2010/04/21/zipcar-buys-into-second-european-car-sharing-company/

The procedure is actually quite simple. First you arrange things into different groups depending on their makeup. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo any particular endeavor. That is, it is better to do too few things at once than too many. In the short run this may not seem important, but complications from doing too many can easily arise. A mistake can be expensive as well. The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here. At first the whole procedure will seem complicated. Soon, however, it will become just another facet of life. It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell.

From Bransford and Johnson, 1973

The procedure is actually quite simple. First you arrange things into different groups depending on their makeup. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo any particular endeavor. That is, it is better to do too few things at once than too many. In the short run this may not seem important, but complications from doing too many can easily arise. A mistake can be expensive as well. The manipulation of the appropriate mechanisms should be self-explanatory, and we need not dwell on it here. At first the whole procedure will seem complicated. Soon, however, it will become just another facet of life. It is difficult to foresee any end to the necessity for this task in the immediate future, but then one never can tell.

From Bransford and Johnson, 1973

The data-frame theory of sensemaking

• Frame– An explanatory structure that links elements e.g. a

story, a map, a script, a plan. – Accounts for data and guides the search for more

data.– Inferred from a few key anchors (3 or 4 at most) – Functions as a hypothesis

• Data– Information from the world. – The interpreted signals of events.

• Process proceeds through fitting data to the frame while testing and improving the frame.

Klein, Phillips, Rall, Peluso (2007)

13Klein, Phillips, Rall, Peluso (2007)

Depends on data, goals, repertoire and stance. Pattern matching.

Add details.

Fundamental surprise (Lanir, 1991).Navigator’s frame- breaker.Emotional responseExperts have more differentiated frames(Feltovich, 84)

Add details.

Explain away data.Knowledge shields.Fixation error produced by inaccurate anchor early on.

Add details.

Nurses in ICU maintaining multiple frames.Tracking up to three frames. Logical competitor set.

Add details.

Duncker’s radiation problem.

Add details.

Duncker’s radiation problem.

Explicit search.

Reasoning in sensemaking

• Experts and novices reason in the same way, except novices find it difficult to distinguish signal from noise and are more reluctant to speculate.

• Experts have a richer repertoire of frames – greater variety, more coverage, finer differentiation. More precise expectations.

• Abductive reasoning predominates in sensemaking – ‘reasoning to the best explanation’ – based on plausibility.

Klein, Phillips, Rall, Peluso (2007)

A symmetry of inferenceA) If it rains in the night the lawn will be wet in the

morning.B) It rained last night.C) The lawn is wet (said in the morning).

Deduction Induction Abduction

A symmetry of inferenceA) When someone with drugs in their pocket sees a

policeman they put their hand in their pocket. B) A man with drugs in his pocket saw a policemanC) The man put his hand in his pocket

Deduction Induction Abduction

http://www.guardian.co.uk/politics/2008/sep/03/london.justice

Will you ever look at things the same again?

http://www.guardian.co.uk/politics/2008/sep/03/london.justice

The rain example

If it rains in the night, the lawn will be wet in the morning.

If the children get up in the night and play with the hose, the lawn will be wet in the morning.

If a bush-fire fighting helicopter on a training mission in the middle of the night accidently drops its load of water over our garden, the lawn will be wet in the morning.

What rule am I thinking of?• I am thinking of a rule the describes the

relationship between three numbers.• The sequence 2-4-6 satisfies the rule.• Your job is to discover the rule by suggesting

other number sets - I will tell you whether each one satisfies the rule.

• When you think you know the rule, tell me.

Confirmation biasPeter Wason• Challenged subjects to identify a rule applying to

triplets of numbers• e.g. 2-4-6 • Subjects seemed to test only positive examples,

e.g. each number is two greater than its predecessor (11, 13, 15)

Confirmation biasKarl Popper• Marxism and psychoanalysis

explain everything? • Worked with Alfred Adler, co-

founder of psychoanalysis • Alder analysed child in terms of

theory of inferiority without seeing the child.

Picture credit: http://www.nndb.com/people/164/000087900/

Popper, 1963

Hang on... how does this relate to visual analytics?

Pirolli and Card’s model of intelligence analysis

Pirolli and Card (2005)

Attfield and Blandford (in press)

• Externalises evidence and reasoning • Structures thinking• Reveals gaps• Can enforce rigor

Attfield and Blandford’s model of legal investigations

Representing the story with chronologies

P1 I think it’s a very natural way for us to think here, we always use chronologies, our great organising basis. […] I had a team of five or six people and I allocated responsibilities to each of these people saying “Right you’re going to become the master of [issue a], I’m going to do [issue b], [issue c], [issue d]. Someone else is going to do [issue e]”

Representing the story with chronologies

A met B at <location > on <date> …

Evidence link

B flew to <location > on <date> …

B met C at <location > on <date> …

Support recursive decomposition

Line of enquiry

Theories

Questions

Info. seeking strategies

Evidence

Knowledge reps

Lines of enquiry

Investigators

Line of enquiry

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Line of enquiry

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Line of enquiry

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Theories

Questions

Evidence

Knowledge reps

Lines of enquiry

Investigators

Analysis of Competing Hypotheses

• Starts with a set of possibilities rather than a most likely alternative for which the analyst seeks confirmation (avoids satisficing)

• Identifies the few items of evidence or assumptions that have greatest diagnostic value.

• Seeks evidence to refute hypotheses. Most probable hypothesis is one with least evidence against it (generally).

Analysis of Competing Hypotheses (ACH)

1. Identify hypotheses that merit consideration

Heuer’s example...

Q: Will Iraq retaliate for US bombing of its intelligence headquarters?

H1 - Iraq will not retaliate.H2 - It will sponsor some minor terrorist actions.H3 - Iraq is planning a major terrorist attack.

• If you don’t generate the right hypothesis you wont get the right answer• Use a group of analysts

2. List significant evidence for and against

Heuer’s example...

E1. Saddam public statement not to retaliate.E2. Absence of terrorist offensive during 1991 gulf warE3. Assumption that Iraq would not want to provoke another US attackE4. Increase in frequency/length of monitored Iraqi agent radio broadcastsE5. Iraqi embassies instructed to take increased precautions.E6. Assumption that failure to retaliate would be unacceptable loss of face for Saddam

• Include assumptions• Ask what should I expect

to see or not see?• Note the absence of

evidence as well as its presence.

3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyse how they relate. Consider diagnosticity

Hypotheses

Evidence No retaliation

Minor actions

Major attack

E1. Saddam public statement not to retaliate. + + +E2. Absence of terrorist offensive during 1991 gulf war + + -E3. Assumption - Iraq would not want to provoke another US attack + + -E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts - + +E5. Iraqi embassies instructed to take increased precautions. - + +E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam

-- + +

4. Refine the matrix. Reconsider hypotheses and delete evidence and arguments with no diagnostic value.

Hypotheses

Minor actions

Major attack

E2. Absence of terrorist offensive during 1991 gulf war + + -E3. Assumption - Iraq would not want to provoke another US attack + + -E4. Increase in frequency/length of monitored Iraqi agent radio broadcasts - + +E5. Iraqi embassies instructed to take increased precautions. - + +E6. Assumption - failure to retaliate would be unacceptable loss of face for Saddam

-- + +

More hypotheses needed? Is your thinking influenced by evidence or assumptions not included?

5. Draw tentative conclusions about likelihood of hypotheses. Work down the matrix. Let them compete for favour.

Hypotheses

Minor actions

Major attack

-- + +

Hypothesis with fewest minuses is probably best, BUT You can never prove a hypothesis true.

6. Consider how sensitive your analysis is to a few items of evidence. What if they were mistaken?

Hypotheses

Minor actions

Major attack

-- + +

Question the linchpin assumptions – a common source of error. What about deception?

7. Report conclusions – discuss relative likelihood of all hypotheses.

Hypotheses

Minor actions

Major attack

-- + +

There is always the possibility of being wrong. Decision makers need to evaluate.

8. Identify future observations that may indicate events are taking a different course than expected.

Hypotheses

Minor actions

Major attack

-- + +

Analytic conclusions should always be considered as tentative.

Summary

Sensemaking– Integration of data and knowledge to form

an interpretation.– Dependant upon knowledge, experience,

values, aspirations, expectations.– Schemata and abduction can play a major

role.– The process is powerful BUT fallible.– Schematisation can make sensemaking

more systematic (e.g. ACH).

ReferencesAesop (c 500BC) The Fox and the Grapes.

Weick K. (1995) Sensemaking in Organisations. Sage, London, England.

Klein G.A., Phillips J.K., Rall E.L., Peluso D.A.: A Data-frame Theory of Sensemaking. In: Expertise Out of Context: Proc. of the Sixth International Conf. on Naturalistic Decision Making (Pensacola Beach, Florida, May 15-17, 2003). Lawrence Erlbaum Assoc. Inc, US, (2007) pp. 113-155, 2007.

Popper K. (1963) Science as Falsification: In Conjectures and Refutations London: Routledge and Keagan Paul, 1963, pp. 33-39, also at http://www.stephenjaygould.org/ctrl/popper_falsification.html

Pirolli P & Card S (2005) The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis, Proceedings of the International Conference on Intelligence Analysis (McLean, VA) https://analysis.mitre.org/proceedings/.

Attfield S. & Blandford A . (in press) Making Sense of Digital Footprints in Team-based Legal Investigations: The Acquisition of Focus. Human Computer Interaction Journal, Special Issue on Sensemaking.

Heuer R. J. (1999) The Psychology of Intelligence Analysis: Center for the Study of Intelligence, Central Intelligence Agency, available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html

Jones M.D. (1995) The Thinker’s Toolkit: 14 Powerful Techniques for Problem Solving. Three Rivers, New York.

perspectives on sensemaking simon attfield visual analytics summer school, 2010 1

integration of information

little information

cases of child injuries

people reason

different groups

europeancar procedure

sour grapes

cognitive dissonance

Documents

analytics: sensemaking, performance, prediction

conceptual misfits in email-based current-awareness...

co-opr: design and evaluation of collaborative sensemaking...

focus- a model of sensemaking

sensemaking in information systems: toward a sensemaking

knowledge mapping for open sensemaking communities

change and sensemaking: organizational identities

designing for online collaborative sensemaking

sensemaking in the future - cbs research portal ·...

collaborative sense-making during simulated intelligence...

focus: a model of sensemaking - air university · focus: a...

sensemaking on the web: how users get...

sensemaking - sage publications · pdf fileand erratic...

eli 2012 sensemaking analytics

requirements sensemaking using concept maps

sensemaking experiments

sensemaking, knowledge creation, and decision...

thinking about thinking: cognition, sensemaking,...

workshop - the sensemaking fragrance

ltb demo - b&p sensemaking