logic models and organizational strategy and evaluation
DESCRIPTION
Logic Models and Organizational Strategy and Evaluation. By: Thomas J. Chapel, MA, MBA Chief Evaluation Officer CDC/NCCDPHP [email protected] 404-488-6467. Disclaimer…. - PowerPoint PPT PresentationTRANSCRIPT
Logic Models and Organizational Strategy and Evaluation
By:Thomas J. Chapel, MA, MBAChief Evaluation OfficerCDC/[email protected]
Disclaimer…
The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention.
3
Objectives
Why bother with logic models Added-value of each component of logic
model Best use of logic models in framing
planning and evaluation questions
4
Take-Home Mantras
It’s not about the model, it’s about “describing your program”
Models make the program logic clear, not true
Match the detail/format to the need Simpler is generally better
5
Presenting Problem
Complex programs operating in complex environments
Accountability focused on long-term/distal outcomes
Joint efforts of many partners necessary to reach distal outcomes
Fragmentation of program planning, performance measurement, and evaluation
6
Integrating Processes to Achieve Continuous Quality Improvement
Continuous Quality Improvement (CQI) cycle
Planning—What actions will best reach our goals and objectives.
Performance measurement— How are we doing?
Evaluation—Why are we doing well or poorly?
What do What do we do?we do?
Why are Why are we we
doing doing well or well or poorly?poorly?
How are How are we we
doing?doing?
How do we How do we do it?do it?
7
You Don’t Ever Need a Logic Model, BUT, You Always Need a Program Description
Don’t jump into planning or evaluation without clarity on:
The big “need” your program is to address The key target group(s) who need to take action The kinds of actions they need to take (your intended
outcomes or objectives) Activities needed to meet those outcomes “Causal” relationships between activities and outcomes
8
Logic Models and Program Description
Logic Models
Graphic depictions of the relationship between your program’s activities and its intended effects
9
Step 2: Describing the Program:Complete Logic Model
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
ContextAssumptions
10
What the program and its staff actually do
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
ContextAssumptions
Results of activities: Who/what will change?
11
Finding Activities and Outcomes
GoalsObjectivesActions/Tactics
ActivitiesST or MT Outcomes
LT Outcomes or Impacts
Process MeasuresProgress Measures
Impl. Measures
Outcome MeasuresImpact Measures
Key Performance IndicatorsSuccess Factors
Pla
nE
val
PM
12
Lead Poisoning: The Problem Lead poisoning is a widespread environmental hazard
facing young children, especially in older inner-city areas. Exposure lead has been linked to cognitive disruption and behavioral disorders, especially when exposure occurs early in life. The main sources of lead poisoning in children are paint and dust in older homes with lead-based paint. Lead poisoning effects can be ameliorated through medical interventions. But, ultimately, the source of lead in the environment must be contained or eliminated through renovation or removal of the lead-based paint by professionals. Short of that, families can reduce the bad effects on their children through intensive housekeeping practices and selected nutritional interventions…
13
Lead Poisoning: The Program… County X, with a suspected high number of lead-poisoned children,
has received money from CDC to support a Childhood Lead Poisoning Prevention Program. The county’s program aims to do outreach and identify children to screen, identify those whose initial and confirmatory screening results show elevated blood lead levels (EBLL) at actionable levels, assess the environments of the EBLL children for sources of lead, and case manage both their medical treatment and the correction of their environment. The program staff will also train families in selected housekeeping and nutritional practices that have been shown to reduce lead levels in homes and lead burden in children. While as a grantee the county can work to assure medical treatment and reduction of lead in the home environment, the grant cannot directly pay for medical treatment or for renovation/remediation of homes.
14
Listing Activities and OutcomesLead Poisoning
Activities Outreach Screening Case management Referral for medical tx Identification of kids with
elevated lead (EBLL) Environmental
assessment Referral for env clean-
up Family training
Effects/Outcomes Lead source identified Families adopt in-home
techniques Providers treats EBLL
kids Housing Authority
eliminates lead source EBLL reduced Developmental “slide”
stopped Q of L improved
15
Global Logic Model: Childhood Lead Poisoning Program
Early Outcomes Later OutcomesLater ActivitiesEarly Activities
If we do…
Outreach
Screening
ID of elevated kids
And we do…
Case mgmt of EBLL kids
Refer EBLL kids for medical treatment
Train family in in-home techniques
Assess environment of EBLL child
Refer environment for clean-up
Then….
EBLL kids get medical treatment
Family performs in-home techniques
Lead source identified
Environment gets cleaned up
Lead source removed
And then…
EBLL reduced
Develop’l slide stopped
Quality of life improves
16
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Activities Outcomes
Elaborating the Simple Logic Model
18
Filling in the Blanks….Mediators
18
19
Conduct training
Do career leadership
development
Competent, sustainable,
diverse workforce
Improved health
outcomesDo Strategic
workforce planning
Evidence Base
Inputs Activities Outcomes
Logic Model—Office of Workforce and Career Development
…a miracle occurs…
20
Implicit Logic Model
Conduct training
Do career leadership
development
Workforce is
CompetentSustainable
Diverse
Improved health
outcomesDo Strategic
workforce planning
Evidence Base
Inputs Activities Outcomes
21
Tangible products of activities
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
ContextAssumptions
22
Lead Poisoning: Sample Outputs
Pool (#) of eligible kids Pool (#) of screened kids Referrals (#) to medical treatment Pool (#) of assessed homes Referrals (#) for clean-up
23
Lead Poisoning: “Causal” Roadmap
ScreeningScreeningScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalMedicalManagementManagement
MedicalMedicalManagementManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids withID kids with
EBLLEBLL
ID kids withID kids with
EBLLEBLL
OutreachOutreach
Train
Families
Train
Families
Refer for Refer for
Medical TreatmentMedical Treatment
Refer for Refer for
Medical TreatmentMedical Treatment
Case
Management
Case
Management
Activities Outcomes
24
Lead Poisoning: “Upgraded” Outputs: More than Simple Counts
Pool (#) of screened kids (meeting likely risk profile)
Pool (#) of eligible kids (with lead level >XXd/ul)
Referrals (#) to (qualified or willing) medical treatment providers
Pool (#) of assessed (“leaded”) homes Referrals (#) for clean-up (to qualified
or willing orgs)
25
Outreach
Screening
ID of elevated kids
Do case mgmt
Refer for medical tx
Train family in in-home techniques
Assess environ’t
Refer house for clean-up
(#) of eligible kids meeting risk profile
(#) screened kids with lead < threshold
(#) referrals to qualified medical tx
(#) of families completing training
(#) of “leaded” homes
(#) referrals to qualified clean-up
EBLL kids get medical treatment
Family performs in-home techniques
Lead source identified
Environ cleaned up
Lead source removed
EBLL reduced
Develop’l slide stopped
Quality of life improves
Global Logic Model: Childhood Lead Poisoning Program
Early Outcomes Later OutcomesLater ActivitiesEarly Activities Outputs
26
Resource “platform” for the program
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
ContextAssumptions
27
Lead Poisoning: Sample Inputs
Funds Trained staff Legal authority to screen Relationships with orgs for med
treatment and environmental clean-up
28
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
Assessment ID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Outcomes Inputs
Funds
Staff
R’ships
Legal
Authority
Funds
Staff
R’ships
Legal
Authority
Activities
29
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
ContextAssumptions
Moderators: Contextual factors that will facilitate or hinder (i.e. “moderate”) ability of activities to produce our outcomes
30
Moderators—Lead Poisoning
Political—“Hazard” politicsEconomic— Health insuranceTechnological— Availability of
hand-held technology
31
Lead Poisoning: “Causal” Roadmap and Moderators
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Managementinsurance climate
“hazard politics”
technology
Activities Outcomes
32
Note!
Logic Models make the program theory clear,
not true!
33
Logic Models and Strategy and Planning
By Clarifying Sequence of Activities Outcomes, Helps With… Identifying/refining missionDeveloping consensus on key intended
outcomes—the “staked claim”Enhances quality and clarity of BSC
perspectives and objectives
34
Helpful Resources NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide:
http://www.cdc.gov/eval/whatsnew.htm Logic Model Sites
Innovation Network: http://www.innonet.org/ Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/ University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/ CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/
resources.htm#4 CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-
PGprogeval.htm
Texts Kellogg Foundation Logic Model Development Guide:
www.wkkf.org W.K. Kellogg Foundation Evaluation Resources:
http://www.wkkf.org/programming/overview.aspx?CID=281 Rogers et al. Program Theory in Evaluation. New Directions Series:
Jossey-Bass, Fall 2000 Chen, H. Theory-Driven Evaluations. Sage. 1990