1 logic models: a framework for program planning and evaluation ellen taylor-powell, ph.d....
TRANSCRIPT
1
Logic Models: A framework for program planning and evaluation
Ellen Taylor-Powell, Ph.D.Evaluation Specialist
University of Wisconsin-Extension-Cooperative Extension
Nutrition, Food Safety and Health ConferenceBaltimore, Maryland
March 31, 2005
2
Dates to late 1960’sCurrent accountability demands
•Public Sector - GPRA
•Non-Profit Sector
•Private Sector
•International Agencies
•Evaluation
A bit of history
3
• What gets measured gets done• If you don’t measure results, you can’t tell success
from failure• If you can’t see success, you can’t reward it• If you can’t reward success, you’re probably
rewarding failure• If you can’t see success, you can’t learn from it• If you can’t recognize failure, you can’t correct it.• If you can demonstrate results, you can win public
support.Osborne and Gaebler, 1992
Accountability era
4
Generic logic model
»
Inputs Outputs Outcomes Outcomes Outcomes
Activities
Participation
» »»»
Inputs Outputs Outcomes Outcomes OutcomesActivities
» »» »
A diagram of the theory of how a program is supposed to work
A graphic depiction of relationships between activities and results
Strategy Results
C O N T E X T
6
Staff
Money
Partners
Develop parent ed curriculum
Deliver series of interactivesessions
Parents increase knowledge of child dev
Parents better understanding their own parenting style Parents use
effective parenting practices
Improved child-parent relations
Research
INPUTS OUTPUTS OUTCOMES
Facilitate support groups
Parents gain skills in effective parenting practices
Simple logic model
Parents identify appropriate actions to take
Strong families
Targetedparentsattend
SITUATION: During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a result
7
Trainer
Funds
Equipment
Research base
Training curriculum
Situation: Funder requires grantees to include a logic model in their funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirement
Increase knowledge of logic models
Increase ability to create a useful logic model of program
Increase confidence in using logic models
Improved planning
Improved evaluation
INPUTS
Logic model of a training workshop
3 hour training
•Interactive activities
•Group work
•Practice
•Q and A
Create meaningful logic models
Use logic models in own work
OUTPUTS OUTCOMES
Grantees
Accountable here
Fulfill requirement of funder
8
INPUTS OUTPUTS OUTCOMES
Program investments
Activities Participation Short Medium
What we invest
What we do
Who we reach
What results
Long-term
Programs are not linear!
9
Language: What do you mean by…
• Goal = Impact• Impact = Long-term outcome• Objectives (participant focused) =
Outcomes• Activities = Outputs
– Outputs may signify “tangible” accomplishments as a result of activities
10
What does a logic model look like? • Graphic display of boxes
and arrows; vertical or horizontal– Relationships, linkages
• Any shape possible– Circular, dynamic– Cultural adaptations;
storyboards
• Level of detail– Simple– Complex
• Multiple models
11
So, why bother? What’s in this for you?
“This seems like a lot of work.” “Where in the world would I get all
the information to put in a logic model?
“I’m a right brain type of person – this isn’t for me.”
“Even if we created one, what would we do with it?”
12
What we are finding:
• Provides a common language• Helps us differentiate between “what
we do” and “results” --- outcomes• Increases understanding about program • Guides and helps focus work• Leads to improved planning and
management• Increases intentionality and purpose • Provides coherence across complex
tasks, diverse environments
13
• Enhances team work• Guides prioritization and allocation of
resources• Motivates staff• Helps to identify important variables to
measure; use evaluation resources wisely
• Increases resources, opportunities, recognition
• Supports replication• Often is required!
14
Testimonials
“Wow – so that is what my program is all about”“I’ve never seen our program on one page before”“I’m now able to say no to things; if it doesn’t fit
within our logic model, I can say no. “ “I can do this”“This took time and effort but it was worth it; our
team never would have gotten here otherwise.”“It helped us to think as a team – to build a team
program vs. an individual program.”
16
Research inputs
Extension inputs
Producer inputs
Policy inputs
Res’rchers
Policy is followed
Accurate research available and shared
Reductions in ammonia emissions
Producers
Multi agency partnership: Abating ammonia emissions from dairy farms
Adopt BMPs
Conduct research
Disseminate & educate
Develop & set standards
Test &feedback
INPUTS OUTPUTS OUTCOMES
Powell et al, 2005
17
Scientists
Staff
Funding
Equipm’t
Accurate research available
Reductions in ammonia emissions
Research widely communi-cated
•Publication
•Popular press
•Pres’tations
•Reports
Multi agency partnership: Research sub-logic model
Existing knwl’dge
Increased ability to determine ammonia emissions at different scales
Increased understanding of relationship between measurement and actual emissions
Increased skills in non-traditional science
Increased knowledge of sources, processes of ammonia emissions
Conduct process & operational level experim’ts
Interpret, validate results
Scale up/out results
Incorporate farmer feedback
Generate funding
Educate re. complexities, components, opportunities
Powell et al, 2005
Res’rchersPartners
18
Benefits of logic modeling to this partnership
• Provided framework to discuss and articulate joint work
• Helped facilitate conversation with focus on agreed upon goal that might not happened otherwise
• Shows contribution of each partner and how result depends upon all
• Keeps end outcome upfront and center• Provides way to communicate about the
partnership that has been presented at national conference
• Separates indicators of achievement - # papers published, # and type of experiments completed – from theory of change. Indicators of achievement are part of evaluation plan for the partnership.
19
Research:
evidence-base
Policy
Advocates
Practitioners
Policy change
Policy makers
Current and potential users
Disparate populations
Key stakeh’ders
Publics
System change
Tobacco Control: Global View
FundersChange in access
Change in support
Change in knowledge, attitudes, skills, motivation
Community programs
Chronic disease pr’grms
School programs
Enforcement
Evaluation and Surveillance
Counter-marketing
Administration & management
Individual change
Partners
Decreased smoking
Reduced exposure to ETS
Reduce mortality, morbidity
Statewide programs
Cessation programs
20
Research:
evidence-base
Policy
Advocates
Practitioner
Demon-strations of support
Reduce mortality, morbidity
• Policy makers
• Current and potential users
• Disparate populations
• Key stakeh’ders
Policies impl’mentedenforced
Tobacco Control: Statewide View - Community Program
Funders
Change in access
Change in support
Change in K,A,S,M
Promote smoke-free policy change
Prevent youth initiation, reduce use
Treat tobacco addiction
Individual change
Partners
Coalition development
• Coalition members
• Key stakeh’ders
Effective coalition functioning
Change in• KAS,• Self-efficacy,• Intent
Success-ful TC imple-mentation
Decreased smoking
Reduced exposure to ETS
21
Mgrs of public areas/events
Coalition
Time
Money
Partnersincluding youth
Research and best practices
Organize and implement SF campaign
PublicDemonstrations of support
SF:•Municipal buildings, grounds, & vehicles•Public areas & events•Worksites•Residence
Change in intent to make services/support available
Increased knowledge and skills to participate in SF public policy change
SF public policies implemented
SF public policies adhered to and enforced
SF policies drafted, improved
Elected officials
Worksite contacts
Increased commitment, support, demand for SF environments
Increased availability of cessation support and services
Increased awareness of importance of SF public policies
Residential owners, mgrs
Increased knowledge of SF benefits and options
OUTCOMES
Form committee
Develop grassroots support
Educate community
Organize earned media
Identify and work with supportive policy makers
Community activists
Media
Tobacco Control: Local view - smoke-free environments
22
Logic Model and Planning
•Applies at any level: national plan, statewide plan, individual plan of work, specific project/activity plan
•Model vs. more detailed program plan/management plan
•Focus on outcomes: “start with end in mind”
23
Logic model and evaluation
Needs/asset assessment:
What are the characteristics, needs, priorities of target population?
What are potential barriers/facilitators?
What is most appropriate?
Process evaluation:
How is program implemented? Fidelity of implementation?
Are activities delivered as intended?
Are participants being reached as intended?
What are participant reactions?
Outcome evaluation:
To what extent are desired changes occurring? For whom?
Is the program making a difference?
What seems to work? Not work?
What are unintended outcomes?
Staff
Money
Partners
Research Parents gain skills in effective parenting practices
Develop parent ed curriculum
Deliver series of interactivesessions
Parents increase knowledge of child dev
Parents better understand their own parenting style
Parents use effective parenting practices
Improved child-parent relations
Facilitate support groups
Parents identify appropriate actions to take
Strong families
Targetedparentsattend
EVALUATION: What do you (and others) want to know about this program?
To what extent are relations improved? Does this result in stronger families?
To what extent did behaviorschange? For whom? Why? What else happened?
To what extent did knowledge and skills increase? For whom? Why? What else happened?
Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – will they come again?
How many sessions were actually delivered? How effectively?# and quality of support groups?
What amount of $ and time were invested?
25
Data collection plan 1. Focus:
2. Questions 3. Indicators 4. Timing 5. Data collection
Sources Methods Sample Instruments
27
What logic model is not…
• A theory• Reality• An evaluation model or method
It is a framework for describing the relationships between investments, activities and results.
It provides a common approach for integrating planning, implementation, evaluation and reporting.
28
Cautions:
• Time consuming - paperwork• Too much focus on outcomes• Too little focus on testing the theory• Perfecting the key to the wrong lock• Attending to context only at front end• Universal vs. context specific mechanisms• Viewing logic model as reality
– “Pyrennes not the Alps”