moving from evaluation to learning peter york
DESCRIPTION
Moving from Evaluation to Learning by Peter YorkTRANSCRIPT
Program Evaluation:
Moving from Evaluation to Learning
2
Session Agenda
• Introductions
• The Need for Learning
• Why Nonprofits Aren’t Learning
• Evaluation vs. Evaluative Learning
• Nonprofit Evaluative Learning – Steps, Processes & Tools
The Need for Learning
4
Leadership = Success
Leadership is key to:
Growth
Sustainability
Lifecycle Advancement
Effectiveness
5
There’s More to Leadership…
• Advancing effective communication of mission and vision to
internal and external stakeholders
• Engaging internal and external stakeholders in planning
• Taking decisive action when faced with challenges
• Making decisions anchored in cost-effectiveness
• Demanding accountability that includes demonstrated
success with those being served/targeted
Only one in four nonprofit organizations is well led…
6
The Key to Leading Is Learning
Only one in four nonprofit organizations is an effective learner…
The following organizational learning behaviors are significantly and
singly the biggest predictor of organizational leadership:
• Creating sophisticated financial, operational, programmatic and
environmental data-gathering and learning processes
• Spending time leveraging program data for making meaning,
decisions, designs and plans, not accountability or validation
• Infusing learning into ALL planning, anchored in program success
• Taking immediate and decisive action, particularly at a human
resource level, as indicated by strategy research findings
Why Aren’t Nonprofits Learning?
8
The Investment Model Is All Wrong
• There’s no “outcome synchronicity” between the
investor and investee
• Effectiveness and accountability must be viewed
through the measurement of proximate effect
• Proximate cause-and-effect is the only way!
• Learning requires understanding the cause, not the
effect
• There’s no Research & Development for
programs/initiatives
9
What Results Do Investors, Business Leaders, & Consumers Want?
Example: In-Home Support Services to New Parents, For-Profits vs. Nonprofits
Investors
(Funders)
Business
LeadersConsumers
For-Profit Results:Just Give Me (or Show Me) the Direct
Results, Please…
Nonprofit Accountability:Just Give Me the Direct Results, But Somehow Prove to
“Them” That We Can Do Much More…
Recuperation, parent-child bonding, healthy
adjustment to family change, tools for care and
feeding, stress-reducing routines and habits, better
communication skills with providers
• Same Direct Results as the For-Profit Business
• Child Development, School Readiness, Crime Reduction
• Parental Reduction in Child Abuse & Neglect, Maternal Health
Nonprofit
Business
Leaders &
Clients
Funders
(Investors)
Why Evaluation Doesn’t Always
Lead to Learning
11
Why Evaluation Isn’t Achieving Learning
1. Makes uncontrollable community impact indicators the
metric of accountability
2. Evaluates the “whole” program/strategic effort to determine
if it worked, rather than the combinations of program
components to determine how it works
3. Aspires to a scientific research design ideal that seeks
widespread generalizability, rather than contextual
applications
4. Gathers data from the wrong source
Evaluation vs. Evaluative Learning
13
The Difference Between Evaluation and Evaluative
Learning
Evaluation: the owner determines success by overall
wins, losses and trophies.
Evaluative Learning: the coach/manager figures out how
to improve the odds of success for the next game by
analyzing the play of the previous ones.
14
ResourcesWhole
ProgramShort-Term Outcomes
Long-Term Outcomes
Community / Social Impact
EvaluationDetermining a “Whole” Program’s Long-Term Impact to Judge Social Value
15
Program Resources
Program Elements
Achievable Outcomes/
Next Actions
Studies say…
Community / Social Impact
Evaluative LearningMeasuring Program Success for “Best Practice” Improvement
16
Six Learning Practices that Facilitate Sustainable
Growth
Organizations are significantly more likely to sustain and grow if they
engage in the following evaluative learning behaviors:
NOT evaluating to decide if a whole program works; rather, figuring out
when and how it works.
Gathering data from recipients.
Determining outcomes by listening to recipient changes and next
actions.
Engaging people in making meaning from data.
Developing performance management processes, tools and systems.
Prioritizing cost-per-outcome.
17
Evaluative Learning Facilitates Sustainable Growth
41% of nonprofits
grow faster than
the rate of inflation
over a three year
period
Nonprofits where
leaders engage in
these behaviors
are 2.5 times
more likely to
grow faster than
the rate of
inflation.
7%
5%
4%
2%
Conducting All R&D Behaviors (0% of all NPs)
Conducting Many R&D
Behaviors (5%)
Conducting Some R&D Behaviors
(45%)
Conducing Very Few R&D
Behaviors (25%)
Conducting NO R&D Behaviors
(25%)
Avera
ge A
nn
ual
Gro
wth
Rate
Average Annual Growth Rate (Mean Per Year, Based on Three Consecutive Years of Data)
18
Guiding Steps for Conducting Evaluative Learning
1. Measure achievable outcomes, next actions, direct results.
2. Gather lots of data QUICKLY, by quantifying the qualitative.
3. Ask the participant/recipient.
4. Analyze data for patterns of cause-and-effect.
5. Gather a few key illustrative stories.
6. Spend time making meaning and making improvements.
19
Step One: Identify Your Achievable Outcomes
• It’s all about the path to behavior change:
– Awareness +
– Knowledge +
– Attitude +
– Motivation +
– Skills +
– Opportunity +
– Behaviors =
___________________
Habits
=
20
Evaluative Learning Processes to Try
1. Positive Deviant Experiment
2. Literature Review to Make the Case for Proximate Outcomes
3. Proximate Logic Model Creation
4. Survey Data Project
5. Qualitative Data Project
6. Creation of Ongoing R&D and Performance Management Tools
Thanks!