kate mckegg and nan wehipeihana (2010). a practitioners introduction to developmental evaluation
DESCRIPTION
Kate McKegg and Nan Wehipeihana (2010). Developmental Evaluation: A practitioner's introduction. A pre-conference workshop presented at the Australasian Evaluation Society (AES) Conference, September 2010, Wellington, New Zealand.TRANSCRIPT
Title Slide Making Evaluation Judgements
A quick overview
He Oranga Poutama Provider Hui, 12-13 October 2009 - Te Manukanuka o Hoturoa Airport Marae
Nan Wehipeihana, Kataraina Pipi and Kate McKegg
Developmental evaluation- A practitioner’s introduction
Kate McKegg and Nan Wehipeihana
Our whakapapa
• Who am I?• Where am I from?• What is my history?
Introductions• Who are you?• Where are you from?• What brings you to this workshop?• What do you hope to learn today?
1. Pair up with someone you don’t know (or don’t know very well)
2. Listen and pay attention to why each other is here today3. Tell us your partner’s name and what they hope to get
out of this session
Workshop OutlineTime Focus
8.30 – 9.00 Whanaungatanga / Introductions
9.00 – 10.00 Introduction to Developmental Evaluation (DE) – what is it?
10.00 – 10.30 Morning tea
10.30 – 12.30 DE - the why, the where and the who
12.30 – 1.30 Lunch
1.30 – 3.00 DE in practice - inquiry approaches
3.00 – 3.30 Afternoon Tea
3.30 – 4.30 Final thoughts, tips and questions
Developmental Evaluation - what is it?
Developmental evaluation supports innovation development to guide adaption to emergent
and dynamic realities in complex environments.
(Michael Quinn Patton, 2010)
Primary developmental evaluation purpose
Complex system challenges Implications
1. Ongoing development Being implemented in a complex and dynamic environment
No intention to become a fixed/standardised modelIdentifies effective principles
2. Adapting effective principles to a new context
Innovative initiativeDevelop ‘own’ version based on adaption of effective principles and knowledge
Top-down—general principles knowledge disseminatedBottom-up—sensitivity to context, experience, capabilities and prioritiesAdaptation vs Adoption
3. Exploring real-time solutions and generating innovative responses in the face of sudden and turbulent major change
Exisiting initiatives and responses no longer effective as conditions change suddenly
Planning, execution and evaluation occur simultaneously
Five purposes of developmental evaluation
Primary developmental evaluation purpose
Complex system challenges Implications
4. Pre-formative development of potentially scalable innovative
Changing and dynamic situations require innovative solutions to worsening conditionsModel needs to be developed/does not exist
Models may move into formative and summative evaluation, others remain in developmental modeInform different potential scaling options
5. Major systems change and cross scale developmental evaluation
Disrupt existing systemTaking an innovation to scaleMajor systems change and changing scale will add levels of complexity, new uncertainties and disagreements
Models change as they are taken across time, space and to larger systemsAdaptive cross scale innovations assume complex, nonlinear dynamics—agility and responsivenessAdaptation -- Replication
Five purposes of developmental evaluation
Adapted from Patton (2010)
Development and innovation in complex situations
• Intractable issues e.g., poverty, homelessness, environmental degradation, family violence, obesity
• Uncertainty about options and solutions• Political and economic turbulence • Environment characterised by unpredictability and
the unexpected emergence of issues and ideas (e.g., rise of social networking)
• Rapid adaption and change occurring across many contexts
DE is informed by complexity and systems thinking (picture sourced from Patton, 2009)
Some key complexity concepts• Adaption• Emergence• Non-linearity• Uncertainty• Interdependence
Exercise One: Handout 1: match each scenario with a complexity concept
Concept ScenarioSomething phenomenal comes out of the blue – like Susan Boyle
So, the fact that people know each other is more influential than expected
Flock of birds flying in formation
Iceland volcano
The introduction of the possum to New Zealand
Concept Brief overview Application to DE
Adaption Interacting elements respond and adapt to each other so that what emerges is a function of the relationships between different agents over time
The essence of DE. An evaluation process that adapts to the needs of programs and organizations, and to the relationships among program stakeholders and the evaluators themselves. Learning by doing.
Emergence Patterns emerge that are beyond, outside of and oblivious to any notion of shared intentionality – the whole is greater than the separate parts
Watching for things to percolate up from interactions, capturing those ideas and new relationships, and placing them in front of people as options for further development
Non-linearity Sensitivity to initial conditions; small actions can stimulate large reactions – a small increase in one variable produces a large increase in another.
Being alert for tipping points or critical incidences is a key part of the DE evaluator’s role, noting forks in the road that can lead to significant changes in programs or organisations – and this often happens in unpredictable ways.
Uncertainty Unpredictable conditions, sometimes unknowable in advance – a ‘maybe’ condition
In DE, the evaluation process cannot be fixed. It needs to have flexibility built in, such as a preparedness for program processes and outcomes to change. Measurement also needs to be flexible.
Interdependence Relationships between different elements are highly interdependent
Paying attention to the interrelationships and inter connections that create feedback loops is a key part of DE. This is important because very often in dynamical situations, we can sometimes not predict what might happen, we only really know once it’s happened.
Some complexity concepts: Handout 2: practical application
DE - a distinct niche• Two distinct niches– Pre-formative– Dynamic situations
• Support exploration and innovation before there is a program model (pre-formative)
• Support the ongoing development and adaption of a programme, or innovation in emergent and complex situations (dynamic situations)
DE - a distinct nicheDifferent to formative and summative• Differs from improvement orientated evaluation (making
a programme better)• It aims to support the ongoing real-time decisions –
what to change, expand, close out further developImportant Distinction:• Differentiating ongoing strategic thinking and periodic
strategic planning • DE as a form of thinking and acting strategically as an
innovative intervention unfolds
• Increase participation in sport and recreation – ‘by Māori’
• 2007-08 review and programme ‘re-visioning’
• Focus on participation ‘as Māori’ participation– Tino rangatiratanga (self
determination)
– 20 years of Māori development
The story of He Oranga Poutama
Case example – He Oranga Poutama and complexity
• Uncertainty around the new programme concept and direction, not sure what ‘as Māori’ participation would involve
• New innovative programme and goal• Values, principles and vision driven• No formal evaluation evidence from other efforts
about what to expect, no real data available to measure key concepts and variables
• Complex environment – contracting economy, political environment, changing organisation
DE – and creative thinking• Requires critical AND creative thinking• We have to think outside the box• Ask questions about what makes sense as we
try to connect evaluation to the program development
Draw four straight lines which go through the middle of all of the dots without taking the pencil off the paper
Creative thinking
Creative thinking:Thinking outside the box
http://www.brainstorming.co.uk/puzzles/ninedotsnj.html
Start here 1
Step 2
Step 3
Step 4
MORNING TEA
When not to use DE
• Not able or willing to commit the time to actively participate in the evaluation and to build and sustain relational trust – no room for the set and forget evaluation management approach
• High levels of certainty required – no tolerance for ambiguity
• Lack of openness to experimentation and reflection – high level of design certainty required
• No adaptive capacity – no real ability to make changes
• Unwillingness to ‘fail’ or hear ‘bad news’ – fear of failure context overrides learning focus
• Poor relationships between management and staff and evaluators – respectful and trusting relationships critical in a context of uncertainty, ambiguity and turbulence
Traditional evaluation is not well suited to all contexts
Traditional evaluation paradigms Developmental evaluation
Manageable and relatively stable situations
Complex dynamic situations, rapidly changing environments
Root cause(s) of problems being addressed is known and reasonably bounded
No known solution to issues, new issues entirely, no certain way forward
Intervention reasonably well defined Multiple pathways possible, need for innovation and exploration
Key variables expected to affect outcomes are knowable, and reasonably measurable
Unknown outcomes, vision and values driven processes
Exercise 2:Teasing out complexity
Handout 3:– Read the brief description of Whānau Ora– What if anything is complex about Whānau Ora?– Share and discuss in small groups– Make notes to feedback to the larger group
Roles and relationships of the DE evaluator
• Evaluator is part of the innovation team• Facilitator and learning coach• Brings evaluative thinking to the group• Support or share innovators’ values and vision• Support the feedback process around “what is
being developed”
Context / Organisation Evaluator – ‘The pragmatic bricoleur’
High levels of awareness of context, and changes in the wider environment
Vigilance to tracking internal and external emergence
Willingness to balance development and innovation with a commitment to testing reality
High tolerance for ambiguity, as well as the ability to facilitate values based sense making, argument and decisions
Willingness to explore, dig deeper, capacity to interpret data, make sense of feedback
Methodological agility and creativity combined with a willingness and ability to change and respond – design, framework, programme theory, methods etc
Courage to face the possibility that some things may not be going so well, adapt and keep going
Courage to take on ‘messy’ journey of ups and downs, sidetracks and the unexpected all the while retaining a tolerant and critical open mindedness and commitment to truth telling
Preparedness to co-create the future, collaborate and trust
Preparedness to develop long term relationships of trust – be ‘in for the long haul’
DE - the context and the evaluator
Skills, abilities and dispositions• The ability to build and maintain trusting relationship
– Critical in a context of uncertainty, ambiguity and turbulence
• High level of facilitation skills, multiple settings, stakeholders and contexts– To generate and ‘test’ ideas, to provide feedback, to ask the ‘tough’ questions;
highly engaging, to build evaluative capability
• Deep methodological toolkit– Methodological flexibility, eclecticism, and adaptability (Patton, p27, 2010)
• Enquiring– Observant and critically reflective
• Systematic – Attention to data and making data driven decisions
• Courageous– Flexible, adaptable, responsive ‘on the run, on the fly’
LUNCH TIME
What will happen to the piece of wood when the person lets go of it?
If the person is on earth…
If the person is under water…
If the person is in space…
The block drops to the ground
The block floats to the surface
The block will not move because there are no overall forces
DE in practice - Inquiry approaches
• Focus of DE is on what is being developed?• What’s emerging?• Given what’s emerged, what’s next?
The DE evaluator:• Inquires into developments• Tracks developments• Facilitates interpretation of developments so that
judgments can be made about the what, how, impact and consequence of developments
DE: Asking questions that matter and matching questions to context• Patton (2010) suggests that questions that matter
can be thought of as “a tool for working in complex situations” (p227)
• Matching questions to particular situations is a central challenge - situational responsiveness and adaption
6 simple rules (Patton, 2010)1. Connect questions with the ideas, language and frameworks
of the people you are working with2. Less is more. Limit the number of questions within the
enquiry framework. 7+ or - 2 is a good rule of thumb3. Keep the evaluation grounded in whatever basic
developmental inquiry framework you and those you’re working with choose to guide your work
4. Distinguish overarching inquiry questions from detailed implementation questions
5. There are (no) stupid questions6. Remember that whatever inquiry framework you are using,
the focus is on What is being developed?
DE – is values based‘[DE] sits alongside, doesn’t control or dampen the core values of
innovation’ (Wehipeihana, cited in Patton, 2010).
• Values / processes / relationships become very important
• ‘How’ outcomes / results are achieved is very important• Process matters• People and relationships matter• Where the end is unpredictable and emergent, values
and process become anchors‘there will never be enough certain knowledge to guide action’ Wendell Berry cited in Patton, 2010.
DE – inquiry approaches• There is no definitive list of developmental evaluation
inquiry approaches - neither should one be constructed• Developmental evaluation creatively adapts whatever
approaches and methods fit the complexities of the situation.
• It is responsive, appropriate, and credible to those you work with, and support opening up new understandings and guiding further developmentIt’s all about persistently asking questions and pursuing credible answers in time to be used. Questioning is the ultimate method.
Questions are the quintessential tools. (Patton, 2010)
DE – uses a range of interpretive frameworks
• But applies evaluation logic and thinking in complex situations
• Enables people to engage in data based, ongoing evaluative sense-making and reasoned argument– ensuring the criteria (values) people use to decide
what’s ‘good’ and desirable are open to scrutiny and made explicit
– ensuring the decisions made about development / the ‘forks in the road’ are documented and systematically reflected upon in a critical and open minded way
Our experience: introducing DE to the uninitiated
• Small steps, getting people grounded, on the same page really matters
• The most simple of frameworks:– What, why, when, how, where and who
• What is happening and what has changed? • Why?• Who has been affected, Who is involved?• How did this happen?• Where is the situation / programme at now?• When do key people need to know what?• How do people feel about the change? Why?• etc
Or some slightly more evaluative questions:
– What’s being developed? (WHAT?)– What sense can we make of emerging issues,
evidence, data about this development? (WHAT’S EMERGING?)
– What is the value / importance /significance of what we are doing / achieving? What does it mean to us now and in the future? (SO WHAT?)
– What does this mean for how we should now act? What are our options? And into the future? (NOW WHAT)
Applying DE questionsDE Question Applied Example (HOP)
What? What’s being developed? What does / might ‘as Māori’ look like in the sport and recreation sector?
What’s emerging?
What sense can we make of emerging issues, evidence, data about this development?
What are other’s doing in the ‘as Māori space? What are the patterns of ideas and practice emerging from the provider communities about ‘as Māori’
So what? What’s the value / significance / importance of what we’re doing and achieving?
Whose values are we applying to our judgments? And what importance do different values place on what’s emerging?
Now what? What does this mean for how we should now act? Into the future?
How will be take our new understandings of ‘as Māori’ and apply these to the next phase of development?
Some useful interpretive frameworks:
• Appreciative Inquiry• Success Case Method• Most Significant Change• Systems approaches – with an emphasis on
perspectives, boundaries and interrelationships• Outcome mapping• Action research
See Handout 5
Example – ‘as Māori’ a developmental journey
The process• As Māori not prescribed in the provider selection process• Series of facilitated discussions
– 1 and 2 day provider hui (n=4)– individual providers
• Ongoing iteration– Cycle of feedback loops
What’s emerging• Shared understanding• Co-constructed But still emergent
See Handout 6
Emergent understanding of ‘as Māori’
• It’s something that is led by tikanga• In a Māori way, underpinned by Māori ways, values and
perspectives. • It may also be in Māori places… but not necessarily. • It may also be doing Māori activities… but not necessarily.• It’s not specific just to Māori contexts e.g. Māori people
participating as Māori in the wider world context.• There’s not going to be one answer• I just want some clear definitions of what non-Māori are
thinking of it
Co-construction of ‘as Māori’
• Te reo• Kanohi ki te kanohi• Guided by kawa and tikanga• Whānau, hapu, iwi and Māori • Uses Māori institutions e.g. marae, kohanga• Whanaungatanga• Traditional Māori sports• Spirit of us• Creating a feeling of belonging and empowerment
What’s the value of ‘as Māori’
• It’s our own self determination of how we do it, in ways that we want to do it.
• To do the things we want to do, in ways that we want to do them
• Opportunity to portray our uniqueness• Effective and better engagement achieved when
cultural values and aspects core and/or included– Traditional sports are just as valuable - in terms of
engaging our communities
Example – ‘as Māori’ a developmental journey• An ‘as Māori’ continuum with five dimensions emerging:
By Māori Refers to the extent of rangatiratanga control by Māori in the governance, management and delivery of an initiative
For Māori Emphasises the importance of whānau, hapu, iwi, and other Māori collectives bought together for kaupapa Māori purposes
With te reo me ona tikanga
Points to the importance of te reo me ona tikanga as central to the survival and affirmation of what it means to live ‘as Māori’
In/on places of cultural significance to Māori
From ancestral marae, awa, maunga, marae to more contemporary places founded on kaupapa Māori principles such as kōhanga reo, kura etc.
Through Those activities that can be distinguished as Māori e.g., nga taonga taakaro
AFTERNOON TEA
Some final thoughts, tips and questionsFAQs we have encountered:• How systematic is DE?• How does DE differ from Action Research?• Isn’t DE just a bunch of stories – when do you
get to outcomes? And measurement? And judgment?– What questions do you have about DE?
What we’ve learned a long the way• It’s a journey with others, this is not for the
sole trader, nor for the feint hearted!• As an evaluation team, we model the
reflection process togetherThis is hard thinking stuff, it doesn’t come easy
• Great rewards possibleThank goodness we chose and are able to use DE, it’s the right way to go, but it is hard work
Wrap up• One thing you learned today• One thing you will do as a result of today• One thing you will follow up on
ReferencesGamble J., A Developmental Evaluation Primer, J.W. McConnell Family
Foundation, 2008.
Patton, M. Q. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, June 2010.
Patton, M.Q. Utilization-Focused Evaluation, 4th ed., Sage, 2008.
Wehipeihana, N and McKegg K, Developmental Evaluation in an Indigenous Context, Reflections on the journey to date. Presentation to American Evaluation Conference, Orlando, Florida, 2009
Westley, Frances; Zimmerman, Brenda; Patton, Michael Q. Getting to Maybe: How the World Is Changed? Random House Canada, 2006