Copernicus InstituteKnowledge Quality Assessment
rethinking uncertainty
challenges for science and society
Jeroen van der Sluijs
Copernicus Institute for Sustainable Development and Innovation
Utrecht University
Copernicus Institute
Knowledge Quality Assessment
Complex environmental risksTypical characteristics (Funtowicz & Ravetz):• Decisions will need to be made before conclusive
scientific evidence is available;• Decision stakes are high: potential error costs of wrong
decisions can be huge • Values are in dispute • Knowledge base is mixture of knowledge and ignorance:
large (partly irreducible) uncertainties, knowledge gaps, and imperfect understanding;
• Assessment dominated by models, scenarios, and assumptions
• Many (hidden) value loadings in problem frames, indicators, assumptions
Coping with uncertainty is essential
Copernicus Institute
Knowledge Quality Assessment
Problematic definitions of uncertainty
Example 1: Walker et al. 2003
“We adopt a general definition of uncertainty as being any departure from the unachievable ideal of complete determinism”
Might make more sense to talk about unreality or uncomplexity as being any departure from or reduction of the inherent complexity of systems outside the controlled environment of the laboratory.
Copernicus Institute
Knowledge Quality Assessment
Definition from HarmoniCa uncertainty guidance document
• A person is uncertain if s/he lacks confidence about the specific outcomes of an event or action. Reasons for this lack of confidence might include a judgement of the information as incomplete, blurred, inaccurate or potentially false or might reflect intrinsic limits to the deterministic predictability of complex systems or of stochastic processes.Similarly, a person is certain if s/he is confident about the outcome of an event. It is possible that a person feels certain but has misjudged the situation (i.e. s/he is wrong).
Copernicus Institute
Knowledge Quality Assessment
• The definition above defines uncertainty as a property (state of confidence) of the decision. Alternatively uncertainty can be defined as a property (state of perfection) of the total body of knowledge or information that is available at the moment of judgement. Uncertainty is then seen as an expression of the various forms of imperfection of the available information and depends on the state-of-the-art of scientific knowledge on the problem at the moment that the decision needs to be made (assuming that the decision maker has access to the state-of-the-art knowledge).
Copernicus Institute
Knowledge Quality Assessment
Challenges• Increase society’s capacity to manage and
surmount uncertainties surrounding knowledge production and use in designing and implementing precautionary (or should I say: responsible) policies and sustainable development
• New epistemology that does not see uncertainty as deviation from deterministic ideal, nor as imperfect knowledge, nor as low quality
• Need for a new (multidimensional) definition of uncertainty (maybe even a new word)
Copernicus Institute
Knowledge Quality Assessment
Insights on uncertainty
• Omitting uncertainty management can lead to scandals, crisis and loss of trust in science and institutions
• More research tends not to reduce uncertainty– Usually reveals unforeseen complexities– Meets irreducible uncertainty (intrinsic or practically)
• High quality low uncertainty• Quality relates to fitness for function (robustness, PP)• In many complex problems unquantifiable
uncertainties dominate the quantifiable uncertainty• Shift in focus needed from reducing uncertainty towards
systematic ways to explicitly cope with uncertainty and quality -> knowledge quality assessment
Copernicus Institute
Knowledge Quality Assessment
Uncertainty as a “monster”
• A monster is a phenomenon that at the same moment fits into two categories that were considered to be mutually excluding
(Smits, 2002; Douglas 1966)
Copernicus Institute
Knowledge Quality Assessment
Cultural categories that we thought to be mutually exclusive and that now tend to get increasingly mixed up:
•knowledge – ignorance•objective – subjective•facts – values•prediction – speculation•science – policy
Copernicus Institute
Knowledge Quality Assessment
Responses to monsters
Different degrees of tolerance towards the abnormal:
• monster-exorcism (expulsion)• monster-adaptation (transformation)• monster-embracement (acceptance)• monster-assimilation (rethinking)
Copernicus Institute
Knowledge Quality Assessment
Footnote:compare to Lakatos (1976)
preserving mathematical models against apparent refutations
• Surrender (throw the model away and start all again),
• Monster barring, • Monster adjustment • Lemma incorporation.
Copernicus Institute
Knowledge Quality Assessment
monster-exorcism
• Uncertainty causes discomfort• Reduce uncertainties!• Strong believe in “objective science”: “the
puzzle can be solved”
Example:• “We are confident that the uncertainties
can be reduced by further research” (IPCC 1990)
Copernicus Institute
Knowledge Quality Assessment
But….• For each head science chops off of
the uncertainty monster, several new monster heads tend to pop up (unforeseen complexities)
• 1994 IGBP dropped objective to reduce uncertainty: ”full predictability of the earth system is almost certainly unattainable”
Copernicus Institute
Knowledge Quality Assessment
• "We cannot be certain that this can be achieved easily and we do know it will take time. Since a fundamentally chaotic climate system is predictable only to a certain degree, our research achievements will always remain uncertain. Exploring the significance and characteristics of this uncertainty is a fundamental challenge to the scientific community." (Bolin, 1994)
Former chairman IPCC on objective to reduce uncertainties:
Copernicus Institute
Knowledge Quality Assessment
Monster adaptation• Fit the uncertainty monster back in the
categories: purification• Quantify uncertainty,
subjective probability & Bayesian • Tendency to build system models based
on “objective science” and externalise the subjective parts and uncertainties into scenario’s and storylines
• Boundary work
Copernicus Institute
Knowledge Quality Assessment
IPCC 10 years after “we are confident that the uncertainties can be reduced…”
Copernicus Institute
Knowledge Quality Assessment
Monster adaptation meets its limits
• Different models fed with the same scenarios produce very different results
• “Integrated Assessment Modeling of Global Climate Change: Transparent Rational Tool for Policy Making or Opaque Screen Hiding Value-laden Assumptions?” (Steve Schneider)
Copernicus Institute
Knowledge Quality Assessment
Monster Embracement• Uncertainty is welcomed: an appreciated property of life
fascination about the unfathomable complexity of our living planet Gaia
room for spirituality and wonder as counterweight to the engineering worldview of “managing the biosphere”
• Plea for a humble science• Holism; Inclusive Science---------------------Or:• Uncertainty is welcomed because it fits well in other
political agenda’s• (strategic) Denial of realness of environmental risks by
emphasizing all those uncertainties• Manufacturing uncertainty
Copernicus Institute
Knowledge Quality Assessment
Monster Assimilation
• Rethink the categories by which the knowledge base is judged
• Create a place for monsters in the science policy interface
• Post Normal Science; Reflexive science; Complex systems research
Copernicus Institute
Knowledge Quality Assessment
Uncertainty has multiple dimensions
• Technical (inexactness)• Methodological (unreliability)• Societal (limited social
robustness)• Epistemological (ignorance)
Copernicus Institute
Knowledge Quality Assessment
Inexactness
Intrinsic uncertainty:– Variability / heterogeneity
Technical limitations:– Resolution error– Aggregation error– Unclear definitions
Copernicus Institute
Knowledge Quality Assessment
UnreliabilityMethodological limitationsLimited internal strength in:
– Use of proxies– Empirical basis– Methodological rigour– Validation
Bias in knowledge production– Motivational bias (interests, incentives)– Disciplinary bias– Cultural bias– Choice of (modelling) approach– Subjective judgement
Future scope
Copernicus Institute
Knowledge Quality Assessment
Limited social robustness
Limited external strength in:– Bias / Value ladenness– Insufficient exploration of rival
problem framings– Management of dissent– Extended peer acceptance /
stakeholder involvement– Transparency– Access & availability– Intelligibility
Strategic/selective knowledge use
Copernicus Institute
Knowledge Quality Assessment
IgnoranceEpistemological limitations• Limited theoretical understanding• System indeterminacy
– Open-endedness– Chaotic behavior– Intrinsic unknowability
• Active ignorance– Model fixes for reasons understood– Limited domains of applicability of functional relations– Numerical error– Surprise A
• Passive ignorance– Bugs (software error, hardware error, typos)– Model fixes for reasons not understood– Surprise B
Copernicus Institute
Knowledge Quality Assessment
Copernicus Institute
Knowledge Quality Assessment
RIVM/MNP Uncertainty Guidance
Systematic reflection on uncertainty and quality in:
• Problem framing• Involvement of stakeholders• Selection of indicators• Appraisal of knowledge base• Mapping and assessment of relevant
uncertainties• Reporting of uncertainty information
Copernicus Institute
Knowledge Quality Assessment
Detailed Guidance
RIVM-MNP Uncertainty Guidance
QuickscanHints &
Actions List
QuickscanQuestionnai
re
Mini-Checklist
Downloads: www.nusap.net
Downloads: www.nusap.net
Reminder listInvokes ReflectionPortal to QS
Further GuidanceAdviceHints & Implications
Advice on Quantitative + Qualitative tools for UA
Tool Catalogue
for UncertaintyAssessment
Copernicus Institute
Knowledge Quality Assessment
Copernicus Institute
Knowledge Quality Assessment
“Wisdom is to know,
that you do not know”
(Socrates)