evaluating policy, systems, & environmental change
DESCRIPTION
University of Colorado REACH 2012. Evaluating Policy, Systems, & Environmental Change. Angela G. Brega, PhD Program Evaluator. Overview. Overview of evaluation How do we evaluate PSE interventions? REACH project evaluation plans. Overview of Evaluation. Importance of Evaluation . - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/1.jpg)
Evaluating Policy, Systems, & Environmental ChangeAngela G. Brega, PhDProgram Evaluator
University of ColoradoREACH 2012
![Page 2: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/2.jpg)
Overview Overview of evaluation How do we evaluate PSE interventions? REACH project evaluation plans
![Page 3: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/3.jpg)
Overview of Evaluation
![Page 4: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/4.jpg)
Importance of Evaluation Assessing the impact of an intervention Has many benefits
Are you meeting your goals? Having the effect you expected? Help identify areas for program improvement Justification to support future requests for
funding Sustainability
Information to share with community and funders
![Page 5: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/5.jpg)
When do we evaluate? Evaluation should be part of every step
in the process of intervention planning Think about evaluation in planning your
program Part of your Community Action Plan
![Page 6: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/6.jpg)
How do we evaluate PSE strategies?
![Page 7: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/7.jpg)
Health Promotion vs. PSE Strategies Health Promotion Program
Change behavior one person at a time PSE Strategies
Change the environment in a way the makes healthy behavior easierHealth Promotion Program
PSE Strategies
Education regarding tooth brushing
Add fluoride to water system
Smoking cessation education
No-smoking policies for public places
Healthy nutrition classes Adding calorie information to menus
![Page 8: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/8.jpg)
Impact of a Health Promotion Program
Intervention
Behavior Change
Improved Health
Outcomes
Diabetes Education Classes
Improved Diet & Physical
Activity
Improved BMI
![Page 9: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/9.jpg)
Impact of a PSE Strategy
Intervention
PSE Change
Behavior Change
Improved Health
Outcomes
Implement a
Complete Streets
Program
vImproved Walking
and Biking Infrastruct
ure
Increased Physical Activity
Reduced Prevalence of Obesity
Distinctions from Health Promotion Model Addition of PSE Change stage Timeline of intervention activities and effect
![Page 10: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/10.jpg)
Implications for EvaluationIntervention
PSE
Change
Behavior Change
Improve
d Health
Outcomes
Outcome EvaluationShort-term Intermediat
eLong-term
Process Evaluati
on
![Page 11: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/11.jpg)
Preliminary PlanWill be finalized after Community
Action Plans are submitted and CDC guidance received.
REACH Project Evaluation Plans
![Page 12: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/12.jpg)
Evaluation ComponentsIntervention
PSE
Change
Behavior Change
Improve
d Health
Outcomes
University of ColoradoSubrecipient
We plan to use existing data sources to examine change over time in health behavior and outcomes in your communities (e.g., BRFSS, NHANES).
1. Community Action Plan• Progress, cost, and
reach2. Coalition Effectiveness
![Page 13: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/13.jpg)
1. Evaluation of Community Action Plan
Evaluation plan developed as part of your Community Action Plan
Community Action Plan Template Identify your Project Period Objectives
(PPOs) Identify your Annual/Multi-Year Objectives
(AMOs) Identify activities related to each AMO
![Page 14: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/14.jpg)
Community Action Plan Template Write PPOs and AMOs in a way that guides
evaluation
Identify exactly what you want to measure Think about data sources from the outset
AMO By September 30, 2014, increase the number of schools that have healthy food procurement policies from 0 to 10.
Direction
of Change
Unit of Measure
ment
What Will be Measured
Baseline
Target
Timeframe
Data Source
Increase
number of
schools that have healthy food procurement
policies
0
10
September 30, 2014
School
Board
![Page 15: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/15.jpg)
Evaluation of Activities Community Action Plan Template asks you to list
your milestone activities for each AMO Does not ask you to think through measurement We will!
PSE change takes time Want to capture your progress along the way
We’ll provide you with an additional table to submit with your Community Action Plan For each activity, what process measures would you
use to show the progress you’ve made Process measures = what have you done
Number of key sectors represented by coalition How many meetings have occurred with key stakeholders
![Page 16: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/16.jpg)
Example
Activity 1
Form an advisory board to develop model food procurement standards.
AMO By September 30, 2014, increase the number of schools that have healthy food procurement policies from 0 to 10.
• Advisory board formed
• Board meets regularly
• Standards developed
Process Measures
Activity 2
Work with 1 school district to promote adoption of standards.
• # meetings held with school district leadership
• # of school board meetings during which policy was discussed
• Policy approved
![Page 17: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/17.jpg)
Implementation Stages Stages of PSE
Intervention1. Formulation2. Enactment3. Implementation4. Maintenance/
Modification Activities & measures
differ by stageLeeman J, Sommers J, Vu M, Jernigan J, Payne G, Thompson D, et al. Framework for Obesity Prevention Policy Interventions. Prev Chronic Dis. 2012;9:110322.
![Page 18: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/18.jpg)
Evaluation of Cost & Reach CDC requires that subrecipients report data
on cost of their interventions and # of people reached Allows for an assessment of cost-effectiveness
Cost and reach estimates are likely to be requested as part of quarterly reports
Evaluation plan related to cost is not yet final Will provide training on estimating cost and
reach over the summer
![Page 19: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/19.jpg)
Estimating Cost Estimate the $$$ spent on activities
related to each objective Estimate cost separately for each PPO
and AMO Estimates
![Page 20: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/20.jpg)
Estimating Reach How many unique individuals have been impacted
# people impacted# of community members # AI/AN people impacted# of AI/AN community members
Documented separately for each PPO and AMO Goal to maximize reach
Target school district rather than single school Estimates
= % of community reached
= % of AI/AN community reached
![Page 21: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/21.jpg)
Coalition Effectiveness Coalition is a critical part of your work
Assessing your community Implementing your Community Action Plan
Ensure that coalitions are effective working groups Evaluate coalition effectiveness
Annual survey of coalition members (4th quarter) Coalition effectiveness (e.g., leadership, membership) Meeting effectiveness (e.g., clarity of goals, resolve conflict)
Identify areas of strength and areas for improvement Guide REACH staff in training needs and you in areas to target
for improvement We will provide you with the evaluation materials
needed More details over the summer
![Page 22: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/22.jpg)
Synopsis Demonstrate the impact of your work Don’t limit ourselves to traditional outcomes
measures Document achievements along the way
Successful development and maintenance of coalition
Completion of milestone activities Reaching objectives
Cost-effectiveness REACH team will assess intermediate/long-term
outcomes
![Page 23: Evaluating Policy, Systems, & Environmental Change](https://reader035.vdocuments.us/reader035/viewer/2022062814/5681687f550346895ddef555/html5/thumbnails/23.jpg)
Evaluation Team Angela Brega
[email protected] 303-724-1470
Venice Ng [email protected] 303-724-7839