![Page 1: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/1.jpg)
Evaluation and the
Science of Complexity
Evaluating Complexity Conference
NORAD
29th -30th May 2008
![Page 2: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/2.jpg)
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
![Page 3: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/3.jpg)
The history of M&E in the international development sector is in four distinct phases 1960s to 1979: Early developments
1979-1984: Rapidly growing interest
1984 to 1988: M&E matures
1988 to the present: the crossroads
![Page 4: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/4.jpg)
Wealth of tools, techniques and approaches are now available Logical framework analysis Results-based management Needs Assessments, Impact Assessments Ex ante and ex post assessments ZOPP GANTT Social Network Analysis Appreciative Enquiry Most Significant Change Outcome Mapping Many many more!!
![Page 5: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/5.jpg)
For many organisations, evaluations are at the centre of a vicious circle...
Increased competition
Increased pressure to show results and impact
Lack of professional norms and standards
Poor learning and accountability
Growing need for high profile
fundraising and advocacy work
...causing problems for M&E
![Page 6: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/6.jpg)
Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes
![Page 7: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/7.jpg)
Reflection, learning and analysis are threatened by existing agency cultures and processes
Org. Learning
Existing culture & process
Evaluation
Accountability
![Page 8: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/8.jpg)
Evaluations, like other similar initiatives, often sit on top of existing organisational silos, inefficiencies and power imbalances, rather than resolving them
Silos
Evaluation
KM
![Page 9: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/9.jpg)
Agencies plough the same evaluation “field”, but stick to their own furrows
![Page 10: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/10.jpg)
Understanding of the effectiveness and use of evaluations is weak at best…
Where’s the data??!
![Page 11: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/11.jpg)
Evaluations tend to be based on wish lists, not strategies, and therefore are often overloaded
![Page 12: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/12.jpg)
Leadership and political buy-in to evaluation is rare and unreliable, with two common reactions
![Page 13: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/13.jpg)
...all of which means that (1) evaluation efforts resemble this iceberg...
What is planned
What actually happens
![Page 14: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/14.jpg)
And (2) evaluators spend most of their time feeling like this...
working against this...
![Page 15: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/15.jpg)
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
![Page 16: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/16.jpg)
“Exploring the science of complexity”
Primary aim was to explore the potential value of complexity science for those who work on change and reform initiatives within the aid sector
Drew on scientific and experimental literature – physiology, physics, mathematics, public sector reform, sociology, economics, organisational theory, plus case studies, reports and evaluations from the aid sector
Reviewed over 250 articles, books, reports and evaluations
10 peer reviewers, including 5 directors of leading research institutes
Published February 2008 Available to download from www.odi.org.uk
![Page 17: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/17.jpg)
A man was walking home one dark and foggy night. As he groped his way through the murk he nearly tripped over someone crawling around by a lamp post.
“What are you doing?” asked the traveler.
“I’m looking for my keys” replied the other.
“Are you sure you lost them here?” asked the traveler.
“I’m not sure at all,” came the reply, “but if I haven’t lost them near this lamp I don’t stand a chance of finding them.”
A (well-known) story…
![Page 18: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/18.jpg)
A closer inspection of the light under the lamp revealed…
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
![Page 19: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/19.jpg)
Logical frameworks / results chains articulate a clear world view and theory of change: “The light under the lamp”
The machine metaphor - universe as clockwork
The future is knowable given enough data points
Phenomena can be reduced to simple cause & effect relationships
Dissecting discrete parts would reveal how the whole system worked; science was the search for the search for the basic building blocks
The role of scientists, technologists & leaders was to predict and control - increasing levels of control (over nature, over people, over things) would improve processes, organisations, quality of life, entire human societies
![Page 20: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/20.jpg)
Key Assumptions
Assumptions about systems Ordered Reductionist - parts would reveal the whole
Assumptions about how systems change Linear relationships influence as direct result of force from one object to
another - predictable Simple cause & effect
Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”
![Page 21: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/21.jpg)
...Reality of aid is a little different...
![Page 22: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/22.jpg)
But a new light is being turned on (slowly, unevenly, using a dimmer switch)…
![Page 23: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/23.jpg)
Complexity science is a science of understanding change A loosely bound collection of ideas, principles and
influences from a number of other bodies of knowledge, including chaos theory fractal geometry cybernetics complex adaptive systems postmodernism systems thinking
Discovery of similar patterns, processes and relationships in a wide variety of phenomena related to the nature and dynamics of change
![Page 24: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/24.jpg)
From microscopic chemical reactions…
![Page 25: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/25.jpg)
...to the evolution of galaxies...
![Page 26: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/26.jpg)
Complexity scientists use a range of ideas and concepts (familiar, challenging and baffling) to make distinctions between simple, complicated and complex systems and phenomena
![Page 27: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/27.jpg)
Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical
and necessary
Sending one rocket increases assurance that next will be ok
High level of expertise in many specialized fields + coordination
Separate into parts and then coordinate
Rockets similar in critical ways
High degree of certainty of outcome
Formulae have only a limited application
Raising one child gives no assurance of success with the next
Expertise can help but is not sufficient; relationships are key
Can’t separate parts from the whole
Every child is unique
Uncertainty of outcome remains
Complicated ComplexSimple
The recipe is essential Recipes are tested to
assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
![Page 28: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/28.jpg)
The claims of complexity scientists
The complexity of real world systems is (usually) not recognised or acknowledged by scientists and policy makers
Or, that if it is not acknowledged, they don’t deal with them
Or, that if they do deal with them, they don’t do so effectively
Or, that if they do deal with them effectively, it’s because they used an specific approach / framework
...that is also available to you, dear client, at a reasonable daily rate plus a per diem
[JOKE]
![Page 29: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/29.jpg)
There have been diverse efforts to apply ideas to social, economic and political analysis and practice Arthur, Ormerod - Economics Stacey, Snowden - Organisations Jervis, Urry, Cutler - Intl relations De Mancha - History Gilchrist - Community development Education policy - Sanders and McCabe Health policy - Zimmerman Government reform - Chapman Strategic thinking – Saunders Evaluation - Williams
![Page 30: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/30.jpg)
Complexity and systems approaches have already proved useful in re-thinking aid and development issues Uphoff, 1990s Chambers, 1997 Sellamna, 1999 IDRC, Outcome Mapping, 2001 Warner, 2001 Rihani, 2002 Lansing and Miller, 2003 Inclusive Aid, 2004 ECDPM, 2004-06 Eyben, 2006 Guijt, various Davies, Network Analysis, various
![Page 31: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/31.jpg)
`
10 key concepts and implications
8 Adaptive Agents
10 Co-Evolution
4 Non-Linearity
3 Emergence from
SimpleRules
6 Phase space and attractors
5 Sensitivity to initial
conditions
1 Interconnected and interdependent
elements and dimensions
2 Feedback processes
7 Strange
attractors and the edge of
chaos
9 Self organisation
Features of systems
Dynamics of change
Behaviours and relationships
![Page 32: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/32.jpg)
Complex systems
Collection of parts, which collectively have a range of dimensions
Parts share an physical or symbolic environment / space
Action by any part can affect the wholeE.g. individuals, families, communities, cities,
markets, societies, populations, economies, nations, planets
![Page 33: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/33.jpg)
Complex systems are interconnected and interdependent to different degrees Interconnectedness may occur between any elements,
dimensions, systems and environments
This interconnectedness leads to interdependence between the elements and the dimensions of a system, and gives rise to complex behaviour
Complex systems can be tightly or loosely coupled, internally and with their environment, giving rise to different kinds of complex behaviours Tightly coupled: Global FOREX markets Loosely coupled: US University system, global construction industry
Systems can be understood via mapping techniques, followed by analysis to understand the dynamics and interactions of the system
![Page 34: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/34.jpg)
Foot and mouth disease: an example of failure caused by focusing on one part of the system and ignoring the links between sub-systems (biology, geography, economics)
Economic rationalisation of abattoirs and EU subsidies increased the interconnectedness of herds to a critical point.
Changes to foot and mouth reporting rules delayed the isolation of infectious animal
The relationship between these actions and the epidemiology of F&M was not appreciated in advance where it mattered because the livestock industry was not viewed as a interconnected, interdependent system
![Page 35: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/35.jpg)
Complexity also means that systems need to be understood at different scales
Atom
Molecule
Cell
Tissue
Organs
Organisms
Communities
![Page 36: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/36.jpg)
E.g. evaluating the effectiveness of child health programmes
A.N. NGO
Other NGOs
Private Sector
Community and
Family
Church
Civil Society
DevelopingCountry Govmts
Local partners
![Page 37: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/37.jpg)
Example: evaluating resource flows in the humanitarian system
![Page 38: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/38.jpg)
Issues of interconnectedness, interdependence and scale usually do not become apparent until a crisis... Foot and mouth disease, UK
Economics, cattle management, disease September 11th
Globalisation and terrorism Climate change
Western consumerism and Southern disasters Credit crunch
US mortgage market mis-selling and the world economy Food prices
Biofuels and food consumption Vulnerability to natural disasters
Sichuan earthquakes and dams Human trafficking
Desire, economics and rights abuses
![Page 39: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/39.jpg)
...indicating that we have biases in the way we view the world...
Three different kinds of problems have been identified, along with some common biases in dealing with them
They are:“Messes”“Problems” “Puzzles”
![Page 40: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/40.jpg)
“Messes” are issues that do not have a well defined form or structure. NB not a value statement, but a description There is often not a clear understanding of the problem faced
Messes often involve economic, technological, ethical and political issues It has been suggested that all of the really important issues in the world
start out as messes.
For example, how was rising HIV/AIDS incidence in Brazil dealt with in the 1990s? concerned money, technology, ethics, social relations, politics, gender
relations, poverty all of these dimensions of the problem had to be dealt with
simultaneously, and as a whole
![Page 41: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/41.jpg)
Many of the major problems we face are “messes”! Credit crunch
US mortgage market mis-selling and the world economy Food prices
Biofuels and Vulnerability to natural disasters
Sichuan earthquakes and dams Climate change
Western consumerism and Southern disasters September 11th
Globalisation and terrorism Human trafficking
Sexual preferences and human rights abuses Arms trade
Economics and war Etc, etc, etc
![Page 42: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/42.jpg)
“Problems” are issues that have a known or knowable form or structure
The key dimensions and variables of a problem are known and the interaction of dimensions may also be understood, even if only partially.
With problems, there is no single clear cut way of doing things there are many alternative solutions, depending on the constraints faced Expertise matters
For example, dealing with the sewage system in a particular city may rely on amount of money available, technology, political stance of leaders, climatic conditions, urban development, the road system, and so on
![Page 43: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/43.jpg)
Puzzles are well defined and well structured known problems with a specific “best” solution Solutions can be worked out and improved
Solutions are replicable - “best practices” are possible
![Page 44: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/44.jpg)
Policy and traditional science is biased towards puzzle-solving Real-world, complex, messy nature of
systems is frequently not recognised
Simple puzzle-based solutions are applied to complex messesE.g. Global War on Terror has been applied as
the single best solution to the mess of terrorism
![Page 45: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/45.jpg)
“Some of the greatest mistakes have been made when dealing with a mess, by not seeing its
dimensions in their entirety, carving off a part, and dealing with this part as if it were a problem, and then solving it as if it were a puzzle, all the while ignoring the linkages and connections to
other dimensions of the mess”
![Page 46: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/46.jpg)
Or to put it another way: dividing a cow in half does not give you two smaller cows
![Page 47: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/47.jpg)
Implications: analyse and deal with the reality of the system Multidimensionality, interdependence and interconnectedness of
poverty and humanitarian crises (and responses to them) should be recognised by those designing, managing and evaluating aid interventions
Analysis may need to be in line with historical research - not ‘did x cause y?’ but ‘what happened and why?’, building narratives about events and processes.
The task of selection and synthesis of data becomes as important as analysis
Different perspectives on what the system is need to be taken into account, especially when these perspectives differ as to the nature, interconnectedness and scale of the system
Messes, problems and puzzles need to be identified and dealt with using appropriate approaches
![Page 48: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/48.jpg)
Interconnectedness and interdependence gives rise to a range of phenomena and
behaviours
![Page 49: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/49.jpg)
`
To find out more, read the paper!
8 Adaptive Agents
10 Co-Evolution
4 Non-Linearity
3 Emergence from
SimpleRules
6 Phase space and attractors
5 Sensitivity to initial
conditions
1 Interconnected and interdependent
elements and dimensions
2 Feedback processes
7 Strange
attractors and the edge of
chaos
9 Self organisation
Features of systems
Dynamics of change
Behaviour of agents
![Page 50: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/50.jpg)
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Summary
![Page 51: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/51.jpg)
Evaluation of Complex Systems is NOT new Educational systems, social initiatives and
government interventions are complex social systems where effective evaluation is seen as a key process in measuring success
But there is an increasing recognition that for evaluation of complex social systems to be more effective, evaluations may need to take into account the theoretical understanding of complex systems
![Page 52: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/52.jpg)
Implications for evaluation are at four levels
1. Implications for evaluation methodologies and approaches
2. Implications for the focus of evaluations
3. Implications for the purpose and scope of evaluations
4. Implications for evaluations as a complex system in their own right
![Page 53: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/53.jpg)
Implications for evaluation methodologies and approaches
![Page 54: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/54.jpg)
Linear models dominate...
If you deliver the
product and / or service to the extent
Intended, then your participants
will benefit in certain ways
If these benefits to participants
are achieved, then certain changes in
communities, organisations and systems
might be expected to occur
If you accomplish your planned
activities, then hopefullyyou will deliver the amount of the product
and / or service that
you intended
Certain resources
are needed to operate
your program
If you haveaccess to
them, then you can use
them to achieve your
plannedactivities
Outcomes / Purpose
Impacts / Goal
OutputsInputs Activities
MONITORING EVALUATION
![Page 55: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/55.jpg)
Many evaluation “results chains” visualize change as linear, based on multiple cause-effect logic models
X Y
Linear, Predictable
Focused on the end result
The program (X) gets the credit!
Inputs Activities Outputs Outcomes Impact
![Page 56: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/56.jpg)
When applied to development, “the results chain” can lead to Faulty thinking Misguided data collection Misleading reporting of results Gives a false sense of reality to senior
managers and donors who are far from where change is taking place
![Page 57: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/57.jpg)
Complexity science sees change as…
Interconnected (multiple actors and factors)
Non-linear (unexpected results occur)
Incremental, cumulative, with tipping points
Beyond the control of the project / programme
Two-way (program also changes)
Continuous (not limited to the life of the project)
![Page 58: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/58.jpg)
Therefore
Develop new theories of change, adapt existing theories of change to challenge assumptions of linearity
Design evaluations to allow for interdependencies and interconnections in the system the program is trying to influence, and capture the resulting dynamics
![Page 59: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/59.jpg)
Implications for evaluation focus
![Page 60: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/60.jpg)
Focus of “traditional” evaluations
Formal project / programme / organisation
Environment is outside the organisation evolves separately until programme is implemented
Level of analysis is single or at most a few, relatively independent levels
![Page 61: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/61.jpg)
Implications of complexity Features of systems: evaluate from perspective of
multiple, nested levels of interconnected systems, study feedback between the organisation and its environment, look for emergent rather than planned change
Dynamics and nature of change: Look for non-linearity, anticipate surprises and unexpected outcomes, analyse the system dynamics over time and frame the “space for possible change”, look for changes in conditions that facilitate systemic change, and how well matched the programme is to the wider system
People, motivations and relationships: Study patterns of incentives and interactions among agents, study quality of relationships, study individuals and informal / shadow coalitions, vs. formal organisation, study co-evolution of organisation and environment
![Page 62: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/62.jpg)
Implications for evaluation purpose and scope
![Page 63: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/63.jpg)
M&E is seen as standing in contrast to creative dynamism of field work
![Page 64: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/64.jpg)
Purpose and scope of traditional evaluations vs complexity-oriented evaluations
Traditional Complexity-oriented
Measure success against predetermined goals
Develop new measures and monitoring mechanisms as goals emerge & evolve
Render definitive judgments of success or failure
Provide feedback, generate learning, support direction or affirm changes in direction
Aim to produce generalisable findings across time & space
Aim to produce context-specific understandings that inform ongoing innovation
Creates fear of failure Supports hunger for learning
![Page 65: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/65.jpg)
Implications for evaluation as a complex system
![Page 66: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/66.jpg)
Key Assumptions
Assumptions about systems Ordered Reductionist - parts would reveal the whole
Assumptions about how systems change Linear relationships influence as direct result of force from one object to
another - predictable Simple cause & effect
Assumptions about human actions Rational choice Behavior specified from top down Certainty and “knowability”
![Page 67: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/67.jpg)
Goals ResultsActivities
Monitoring
Evaluation
Evaluations traditionally seen as a rational, technical, information-generating process
...Reality is a little different...
![Page 68: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/68.jpg)
Evaluation systems are themselves complex systems, with many interconnected parts and dimensions; no two evaluations are the same
Focus of evaluation Policy / Guidelines Scope Project vs Policy Demand, goals Timing, quantity Preparation, TOR Management and team selection Methods and tools Engagement with stakeholders Dissemination of findings and utilisation Costs Quality maintenance mechanisms
![Page 69: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/69.jpg)
“...To be effective an evaluation program must match the dynamics of the system to
which it is applied....”
Eoyang and Berkas (1998)
![Page 70: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/70.jpg)
Implications of evaluations as a complex systemTraditional Complexity-oriented
Position the evaluator outside to assure independence and objectivity
Position evaluation as an internal, team function integrated into action and ongoing interpretive processes
Accountability to control and locate blame for failures
Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond strategically
Accountability focused on and directed to external authorities and funders
Accountability centred on fundamental values and commitments
Evaluator controls the evaluation and determines the design based on the evaluator’s perspective about what is important
Evaluator collaborates in the change effort to design a process that matches philosophically and organisationally
![Page 71: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/71.jpg)
All systems can be placed on a spectrum between extremes of ordered and chaotic E.g. solids and gases
In solids, atoms are locked into place In gases they tumble over one another at random
In between the two extremes, at a phase transition, a phenomenon called the ‘edge of chaos’ occurs This phenomenon describes systems behaviours where the evolution of
the system never quite locks into place and never quite dissolve into turbulence either.
In human organisations, the simplest example is of a system that is neither too centrally controlled (order) nor too bottom-up and therefore disorganised (chaos)
![Page 72: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/72.jpg)
Physiology on the Edge of Chaos
Healthy Heart – on the edge of chaos
Severe Congestive Heart Failure – orderly
Cardiac Arrhythmia, Atrial Fibrillation - chaotic
![Page 73: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/73.jpg)
Dynamics
Ad
ap
tab
ility
Ordered Disordered
ZONE of HEALTH
Point of Maximum Adaptability
Threshold Threshold
High
Low
Dynamic adaptability is the key to system health
Changelessness is a sign of death,transformation a sign of life.- Commentary on the I Ching
![Page 74: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/74.jpg)
Evaluation at the
edge of chaos?
![Page 75: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/75.jpg)
Agenda
Evaluations – some common issues Complexity science – origins and ideas Implications for evaluations Final points
![Page 76: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/76.jpg)
Complexity science gives additional weight to calls for re-thinking
The wider contexts of aid work The nature of the processes involved in aid work The dynamics of change involved in aid work The real influence of aid work The role of partner organisations and
beneficiaries in aid work The tools and techniques for planning,
monitoring, learning and evaluating aid work
![Page 77: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/77.jpg)
Given the resistance to change in the power dynamics of aid, real world applications may continue to be “innovative”, “under the radar”
and outside the mainstream of aid policy and practice...
![Page 78: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/78.jpg)
There are a number of common criticisms of complexity... Theoretical: adds nothing new
E.g. the ideas of complexity science have all been identified elsewhere But complexity brings them together
Practical: doesn’t specify what should be done E.g makes no specific recommendations as to how best to act in complex systems See next slide
Supports managerial “snake-oil” / “complexologists” / re-warmed ideas E.g. is abused and misused What isn’t?
Political: emergence and self organisation support neo-liberal stances Just because self-organisation happens, doesn’t mean it is good
Credit crisis Rwandan genocide
Complexity is more centre-ground, for example, “edge of chaos” systems are seen as most robust and resilient because are the optimally combination of control and flexibility
![Page 79: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/79.jpg)
...and different perspectives on the value of complexity science Deep paradigmatic insights
Champions
Interesting parallels and useful approaches, but not the only way to see things Pragmatists
Meaningless coincidences Critics
![Page 80: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/80.jpg)
Source of weakness is also the source of strength These ideas are not about “what you dos”, but about
“how you dos” Not “solutions for problems”, but “approaches to problems” Tools for furthering understanding, for opening up new ways of
seeing and thinking
They point to the personal, professional, institutional, political mindsets, attitudes and conditions which need to be in place to work effectively in and with complex systems
![Page 81: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/81.jpg)
Complexity concepts can support the intuition and navigation of practitioners
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
![Page 82: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/82.jpg)
Four suggestions
Develop collective intellectual openness to ask a new, potentially valuable, but challenging set of questions of our mission and their work
Develop collective intellectual and methodological restraint to accept the limitations of a new and potentially valuable set of ideas not misuse or abuse or let them become part of the ever-swinging pendulum of
aid approaches
Need to be humble and honest about the scope of what can be achieved through ‘outsider’ interventions, about the kinds of mistakes that are so often made, and about the reasons why such mistakes are repeated
Need to develop the individual, institutional and political courage to face up to the implications of complexity
![Page 83: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/83.jpg)
...We can't solve problems by using the same kind of thinking
we used when we created them..
Final points (1)
![Page 84: Evaluation and the Science of Complexity Evaluating Complexity Conference NORAD 29 th -30 th May 2008](https://reader035.vdocuments.us/reader035/viewer/2022062718/56649e955503460f94b99d63/html5/thumbnails/84.jpg)
Everybody thinks to change the
world; nobody thinks to
change themselves
Final points (2)