copernicus institute knowledge quality assessment rethinking uncertainty challenges for science and...

Copernicus InstituteKnowledge Quality Assessment

rethinking uncertainty

challenges for science and society

Jeroen van der Sluijs

Copernicus Institute for Sustainable Development and Innovation

Utrecht University

Copernicus Institute

Knowledge Quality Assessment

Complex environmental risksTypical characteristics (Funtowicz & Ravetz):• Decisions will need to be made before conclusive

scientific evidence is available;• Decision stakes are high: potential error costs of wrong

decisions can be huge • Values are in dispute • Knowledge base is mixture of knowledge and ignorance:

large (partly irreducible) uncertainties, knowledge gaps, and imperfect understanding;

• Assessment dominated by models, scenarios, and assumptions

• Many (hidden) value loadings in problem frames, indicators, assumptions

Coping with uncertainty is essential



Problematic definitions of uncertainty

Example 1: Walker et al. 2003

“We adopt a general definition of uncertainty as being any departure from the unachievable ideal of complete determinism”

Might make more sense to talk about unreality or uncomplexity as being any departure from or reduction of the inherent complexity of systems outside the controlled environment of the laboratory.



Definition from HarmoniCa uncertainty guidance document

• A person is uncertain if s/he lacks confidence about the specific outcomes of an event or action. Reasons for this lack of confidence might include a judgement of the information as incomplete, blurred, inaccurate or potentially false or might reflect intrinsic limits to the deterministic predictability of complex systems or of stochastic processes.Similarly, a person is certain if s/he is confident about the outcome of an event. It is possible that a person feels certain but has misjudged the situation (i.e. s/he is wrong).



• The definition above defines uncertainty as a property (state of confidence) of the decision. Alternatively uncertainty can be defined as a property (state of perfection) of the total body of knowledge or information that is available at the moment of judgement. Uncertainty is then seen as an expression of the various forms of imperfection of the available information and depends on the state-of-the-art of scientific knowledge on the problem at the moment that the decision needs to be made (assuming that the decision maker has access to the state-of-the-art knowledge).



Challenges• Increase society’s capacity to manage and

surmount uncertainties surrounding knowledge production and use in designing and implementing precautionary (or should I say: responsible) policies and sustainable development

• New epistemology that does not see uncertainty as deviation from deterministic ideal, nor as imperfect knowledge, nor as low quality

• Need for a new (multidimensional) definition of uncertainty (maybe even a new word)



Insights on uncertainty

• Omitting uncertainty management can lead to scandals, crisis and loss of trust in science and institutions

• More research tends not to reduce uncertainty– Usually reveals unforeseen complexities– Meets irreducible uncertainty (intrinsic or practically)

• High quality low uncertainty• Quality relates to fitness for function (robustness, PP)• In many complex problems unquantifiable

uncertainties dominate the quantifiable uncertainty• Shift in focus needed from reducing uncertainty towards

systematic ways to explicitly cope with uncertainty and quality -> knowledge quality assessment



Uncertainty as a “monster”

• A monster is a phenomenon that at the same moment fits into two categories that were considered to be mutually excluding

(Smits, 2002; Douglas 1966)



Cultural categories that we thought to be mutually exclusive and that now tend to get increasingly mixed up:

•knowledge – ignorance•objective – subjective•facts – values•prediction – speculation•science – policy



Responses to monsters

Different degrees of tolerance towards the abnormal:

• monster-exorcism (expulsion)• monster-adaptation (transformation)• monster-embracement (acceptance)• monster-assimilation (rethinking)



Footnote:compare to Lakatos (1976)

preserving mathematical models against apparent refutations

• Surrender (throw the model away and start all again),

• Monster barring, • Monster adjustment • Lemma incorporation.



monster-exorcism

• Uncertainty causes discomfort• Reduce uncertainties!• Strong believe in “objective science”: “the

puzzle can be solved”

Example:• “We are confident that the uncertainties

can be reduced by further research” (IPCC 1990)



But….• For each head science chops off of

the uncertainty monster, several new monster heads tend to pop up (unforeseen complexities)

• 1994 IGBP dropped objective to reduce uncertainty: ”full predictability of the earth system is almost certainly unattainable”



• "We cannot be certain that this can be achieved easily and we do know it will take time. Since a fundamentally chaotic climate system is predictable only to a certain degree, our research achievements will always remain uncertain. Exploring the significance and characteristics of this uncertainty is a fundamental challenge to the scientific community." (Bolin, 1994)

Former chairman IPCC on objective to reduce uncertainties:



Monster adaptation• Fit the uncertainty monster back in the

categories: purification• Quantify uncertainty,

subjective probability & Bayesian • Tendency to build system models based

on “objective science” and externalise the subjective parts and uncertainties into scenario’s and storylines

• Boundary work



IPCC 10 years after “we are confident that the uncertainties can be reduced…”



Monster adaptation meets its limits

• Different models fed with the same scenarios produce very different results

• “Integrated Assessment Modeling of Global Climate Change: Transparent Rational Tool for Policy Making or Opaque Screen Hiding Value-laden Assumptions?” (Steve Schneider)



Monster Embracement• Uncertainty is welcomed: an appreciated property of life

fascination about the unfathomable complexity of our living planet Gaia

room for spirituality and wonder as counterweight to the engineering worldview of “managing the biosphere”

• Plea for a humble science• Holism; Inclusive Science---------------------Or:• Uncertainty is welcomed because it fits well in other

political agenda’s• (strategic) Denial of realness of environmental risks by

emphasizing all those uncertainties• Manufacturing uncertainty



Monster Assimilation

• Rethink the categories by which the knowledge base is judged

• Create a place for monsters in the science policy interface

• Post Normal Science; Reflexive science; Complex systems research



Uncertainty has multiple dimensions

• Technical (inexactness)• Methodological (unreliability)• Societal (limited social

robustness)• Epistemological (ignorance)



Inexactness

Intrinsic uncertainty:– Variability / heterogeneity

Technical limitations:– Resolution error– Aggregation error– Unclear definitions



UnreliabilityMethodological limitationsLimited internal strength in:

– Use of proxies– Empirical basis– Methodological rigour– Validation

Bias in knowledge production– Motivational bias (interests, incentives)– Disciplinary bias– Cultural bias– Choice of (modelling) approach– Subjective judgement

Future scope



Limited social robustness

Limited external strength in:– Bias / Value ladenness– Insufficient exploration of rival

problem framings– Management of dissent– Extended peer acceptance /

stakeholder involvement– Transparency– Access & availability– Intelligibility

Strategic/selective knowledge use



IgnoranceEpistemological limitations• Limited theoretical understanding• System indeterminacy

– Open-endedness– Chaotic behavior– Intrinsic unknowability

• Active ignorance– Model fixes for reasons understood– Limited domains of applicability of functional relations– Numerical error– Surprise A

• Passive ignorance– Bugs (software error, hardware error, typos)– Model fixes for reasons not understood– Surprise B



RIVM/MNP Uncertainty Guidance

Systematic reflection on uncertainty and quality in:

• Problem framing• Involvement of stakeholders• Selection of indicators• Appraisal of knowledge base• Mapping and assessment of relevant

uncertainties• Reporting of uncertainty information



Detailed Guidance

RIVM-MNP Uncertainty Guidance

QuickscanHints &

Actions List

QuickscanQuestionnai

re

Mini-Checklist

Downloads: www.nusap.net

Downloads: www.nusap.net

Reminder listInvokes ReflectionPortal to QS

Further GuidanceAdviceHints & Implications

Advice on Quantitative + Qualitative tools for UA

Tool Catalogue

for UncertaintyAssessment



“Wisdom is to know,

that you do not know”

(Socrates)

copernicus institute knowledge quality assessment rethinking uncertainty challenges for science and...

Documents