pittsburgh nonprofit summit - measuring change - moving from outputs to outcomes to impact
DESCRIPTION
More and more funders are only supporting evidence based models. What does this mean and how can your organization implement an evidence based model? A panel of experts has been assembled to answer any and all questions your organization has on how it can move from outputs to outcomes to impact. Brainstorm real strategies that your organization can implement and meet other organizations who are also moving towards an evidence based model.TRANSCRIPT
Measuring ChangeMoving from Outputs to Outcomes to Impact
10:00am – 11:15amFacilitator: Lisa Kuzma, Richard King Mellon Foundation
Speakers: Mike Bangser, MDRC / Junlei Li, Office of Child Development / Art Maxwell and Pam Meadowcroft, Meadowcroft
& Associates / Tim Weidemann, Rondout Consulting
Share your thoughts!
• Text “Summit” to 57682 with your feedback and thoughts on today’s event!
• Share your thoughts on Twitter by adding #GPNPSummit to your tweets!
Measuring Change – Moving From Outputs to Outcomes to Impacts
The Context
[We are] trying to put much more emphasis on evidence-based policy decisions here at OMB. Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges — expanding the approaches that work best, fine tuning the ones that get mixed results, and shutting down those that are failing.
-- Peter Orszag, former Director of the Office of Management and Budget
4
Key Assumptions
Agreement on what it means to be “effective” Reliable evidence on which programs are effective, using
the proper measures Reliable evidence on the core elements that contribute
to effectiveness Documentation and communication of these elements Understanding of how the elements can or should be
adapted to local conditions Providers have sufficient resources and capacity to
replicate evidence-based practices
5
Key Distinctions
Outputs (typically, units of service)
Gross or absolute outcomes (e.g., employment, learning, or behavioral gains for those who are served)
Net impacts (how the outcomes differ from what would have occurred without the intervention)
6
Outcomes vs. Impacts: Which program is more effective?
7
80
70
0
10
20
30
40
50
60
70
80
90
100
A
Per
cent
Pla
ced
In A
Job
40
0
10
20
30
40
50
60
70
80
90
100
BJob Program
GroupControl Group
Per
cent
Pla
ced
In A
Job
20
Positive Outcomes, But for Whom?
Q1 1994
Q1 1995
Q1 1996
Q1 1997
Q1 1998
Q1 1999
Q1 2000
Q1 2001
Q1 2002
Q1 20030
500
1000
1500
2000
2500
3000
3500
Average Quarterly Earnings
Mea
n Q
uart
erly
Ear
ning
s (in
200
3 D
olla
rs)
Key Challenges in Replication
Rigorous evidence of what works The right level of model specification Proper “fit” between the replicated program model and
the local population being served Other factors that may affect results (staffing, funding
levels, etc.) Finding useful “leading indicators” for program staff Data issues
Implications for Planning and Practice Need a realistic meeting of the minds on the roles of
policymakers, funders, researchers , and providers The goal is fidelity to the right things, at scale and at
reasonable cost Need appropriate support for replication and
organizational infrastructure (dissemination of research findings , tools and training, data systems, etc.)
Beware of potential distortions in interpreting outcomes – especially the influence of participant selection
Recognize that moving from outputs to outcomes to impacts is an evolutionary process
Thank you from the Greater Pittsburgh Nonprofit
Partnership!
Workshop Evaluation
Text “Impact” to 57682 with your answers to the following questions:
• Please rate the overall value of this workshopE – Excellent G – Good P – Poor F – Fair
• Did you learn anything that you will apply at your own organization?Y – Yes N - No
• Please text other comments and feedback.
Submit by hitting “send!” You will receive an auto-reply from the GPNP.Sample text: “Impact E Y I’ve learned great tips today.”