going beyond evaluation buzzwords
Post on 12-Jan-2015
216 Views
Preview:
DESCRIPTION
TRANSCRIPT
Going Beyond Evaluation Buzzwords
Lauren Decker, Ph.D. Edvance Research
Presented at Texas ACE Annual Conference, Austin, TX.
August 7, 2013
One word or phrase expressing how you feel about evaluation
“If you don’t know where you are going, how are you gonna’ know when you get there?”
Yogi Berra
Where are we going and how will we get there?
What’s a logic model and how does it help? Defining seemingly inter-changeable terms Time and tool to begin a logic model (or refine one)
What is actually happening and how does it help to know? Program and evaluation importance of understanding
implementation fidelity
Bringing thoughtful program lessons into evaluation plan and use What makes a great evaluation? Planning and asking the right question(s)
Logic Models and Theories of Change
Logic Model vs. Theory of Change
Logic Model = graphically depicts what your program is intended to do Key program components
Theory of Change = why your program operates as intended and how components and activities are hypothesized to move outcomes Assumptions underlying
expected change
Simple Logic Model
Inputs Outputs Outcomes
Resources Activities Participation Short
Medium
Long
What we do
Who we reach
What results we expect
What we invest
Logic Model Example
How does having a logic model help my program?
Program phase
New program
Existing program
Redesigning existing program
Logic model use
Creation & planning
Documentation & communication
Reinvention, improvement, & expansion
Use the paper provided to begin a logic model for your program
Remember to include resources, intended activities, participants, short-, mid-, and long-term outcomes
If you can also represent the theory of change (arrows)
What does our program logic model look like? 10 minutes
What should we include in our program Logic Model?
Identify: All key components of program
Resources, intended activities, participation, short-, mid-, and long-term outcomes
Components and pathways(mediators) through which program is expected to produce intended outcomes (theory)
Student outcome(s) program is designed to improve Short-, mid-, long-term outcomes Other outcomes as well if relevant
I have a Logic Model, now what?
– What is fidelity of implementation?
Fidelity of Implementation
The extent to which key components of the program are delivered as originally intended by the developer Key components = strategies, practices,
activities and behaviors that are critical to defining your program
Logic Model Example
Fidelity of Implementation the extent to which key components of
the program are delivered as originally intended by the developer Key components = strategies, practices,
activities and behaviors that are critical to defining your program
When is your program not your program?
How do I measure fidelity of my program?
Fidelity
Structural key
components
Interactional key
components
Structure Process
How does measuring fidelity help my program?
Understanding fidelity helps you know about: Participation Activity delivery Content
Example questions you can answer: Were activities implemented by program staff
according to design? Did staff receive program content as planned? What were the adaptations that were made to the
program? What were the factors that may have affected program
fidelity?
“I think you should be more explicit here in Step Two.”
Evaluations without measuring fidelity…
Visit someone else’s logic model in process
For the activities listed suggest: What type of
fidelity could be measured
Ways of measuring that type of fidelity for the activity
How to measure fidelity in your program? 10 minutes
What do we need to do to understand fidelity of our program?
Periodically measure implementation fidelity Measured separately for each key
component Thresholds specified for determining
whether key components of program were implemented with fidelity
Assess and Report whether each key component was implemented with fidelity
What makes a great evaluation?
High-quality evaluations include:
Evaluators who clearly use sound evaluation principles and practices (transparency, consideration of context, etc.) Complex programs require thoughtful, directed
evaluations Focus from the start on use of evaluation
processes and products to provide information to, and about, the program being evaluated Alignment to the program’s logic model and
theory of change Include provision for some level of
implementation fidelity
High-quality evaluations include:
Preparation and planning for variation within multi-site evaluations
Consideration of clear deadlines for completion and deliverables set by program
Upfront acknowledgement of the limitations of the proposed evaluation
Minimizing jargon to increase clarity of communication of results
Striking a balance between consultancy/ recommendation support and maintaining unbiased distance as an independent evaluator
Before meeting with an evaluator:
Complete your program logic model and theory of change Or have discussions and ideas of key
components for development assistance from evaluator
Consider what you really want to learn What outcomes do you want an evaluation to
focus on for your program?
What do you really want to know? Make sure you ask the right question!
Before meeting with an evaluator:
Complete your program logic model and theory of change
Consider what you really want to learn What data is already available or will need to
be collected to answer your right question(s)? If you believe data will need to be collected do
you want to collect them or have the evaluator conduct the data collection?
Think about whether you have identified SMART outcomes!
SMART Outcomes are:
Specific Reflect simple information that is communicable and
easily understood Measurable
Can changes by measured in reliable and valid ways? Achievable
Able to be collected and sensitive to change during the allotted time
Relevant Reflect information that is important and likely to be used
Time bound Progress can be tracked at desired frequency within
allotted time
Source: Adapted from World Bank Group.
Questions for an evaluator How does your proposed evaluation align with our
program logic model and theory of change? What data do you need from us to conduct the
evaluation? What data to you propose to collect, when, and in what format?
How intrusive will the data collection be for our program?
Will your evaluation be able to tell us if the program was in place as intended?
What we really want to know from an evaluation is _________. How will your proposed evaluation answer this question?
Questions to ask yourself about an evaluation proposed to you
Does the evaluation design reflect my logic model and theory of change?
Do the proposed outcomes follow my logic model and theory of change?
Is it feasible to collect outcome data at the time I expect to see changes (short- medium- and long-term outcomes) in those selected?
Are the selected outcomes and measures relevant for my program and my stakeholders?
Does the evaluation include SMART outcomes that will be evaluated?
Logic Model Example
Questions? Thank you.
Lauren Decker, Ph.D. ldecker@edvanceresearch.com
top related