implementing and evaluating evidence-based strategies
TRANSCRIPT
Implementing and Evaluating Evidence-based Strategies
Assessing your
Community
Implementing & Evaluating
Establishing Goals &
Objectives;Planning for Evaluation
Finding Evidence
Selecting Best Fitting EBAs
Adapting
Where do We Stand?
Session Objectives
• Describe successful program implementation• Discuss the importance of community
engagement, partnerships, and collaborations to implement programs
• Describe key tasks in planning, implementation, evaluation and maintenance
• Develop an implementation and evaluation work plan integrating RE-AIM measures that match program objectives and activities
Intervention Planning Model
Community Assessment
DevelopGoals &
Objectives
Plan the InterventionFinding, Selecting and Adapting an Evidence-based Approach
ImplementEvaluate
Outcomes
Share your successes
***Apply lessons
learned to Maintain and Sustain the intervention
Assess Readiness
Evaluate Processes
Planning Steps• Enlist community/stakeholder input
– Use existing committees– Understand current context, policy or environment– Best communication/outreach strategies
• Understand context, fit, and your readiness– Incorporate previous needs assessment data– Know your organizational culture related to health
• Draft work plan for program, policy, strategies– Research any previous interventions
for useful content/language – Receive feedback from expert panel
Planning your implementation and evaluation: Review Materials
Policy Process Guides
http://www.prevent.org/data/files/initiatives/smokefreepolicies.pdf
Assess Readiness
• Do you have:– Supportive leaders? – Engaged partners?– Resources (money, staff, supplies)?– Staff with knowledge, skills, and enthusiasm? – Accounting and administrative structures?– System for monitoring progress/evaluation?
• Refer to implementation guides and protocols for resources needed
Assess Organizational Readiness
Readiness Checklist for Body and Soul
What leads to implementation quality?
1. Community level engagement
(funding, policy, politics)
2. Provider/implementer characteristics– Need for innovation– Benefits of innovation– Self-efficacy– Skill proficiency for implementation
Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.
What leads to implementation quality?
3. Implementers’ organizational capacity– General organizational factors
(vision, norms, climate)– Specific processes
(decision-making, partnerships, communication)– Staff considerations
(leadership, champion)
4. Prevention support– Training– Technical assistance
Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.
Create a Work Plan
Ideas from your Organizational Readiness
Assessment
Your Work Plan for Implementing and Evaluating
Body and Soul
Components of a Work Plan for Implementation
• SMART Objectives• List of tasks or activities for each Objective• Person responsible• Resources needed• Target completion dates• Indicator(s)/Measures of completion or progress
Example: Work Plan for Body and Soul
Example Work Plan
Example Work Plan
Definitions of Workplan Steps: Policy & Environmental Change
• Review evidence/data on the problem and potential solutions• Gain stakeholder agreement on priority problems and
preferred approaches, • Draft policies in the form of rules/laws
Formulation
• Identify decision makers who will sponsor or promote the policy• Formally pass the policy in the form of a law, regulation, or
budget• Put environmental change in place (ex: inaugurate walking trail)• Communicate the new policy/environmental change
Enactment
• Monitor and document the use of the policy• Continue promotion, operationalization, and use of the policy• Conduct activities that support compliance to the policy• Establish a plan for enforcement
Implementation
• Maintain policy implementation• Ensure sustained effect through monitoring and enforcement• Possibly make further policy modification
Maintenance
Longest Jr. BB. (2009) Health Policymaking in the United States, 5th Ed.
Activity: Create an Implementation Work Plan
Activity: Create an Implementation Work Plan
Activity: Create an Implementation Work Plan
1. Review the handout on the Tobacco Use Prevention Policy Strategy
2. Choose at least 1-2 of the objectives already provided or write your own (for a worksite or faith setting)
3. For the objective(s) you chose, add:– One or more activities– Who would be responsible for each activity? – What resources are needed? – What is the completion date or due date?– What are your measures/indicators of completion or
progress?
Implementation Work Plan Discussion
4. Share different examples from the groups.
a) Specify if you chose a work site or faith setting.
b) Explain which objective you chose.
c) Discuss:– activities, – responsible persons, – resources, – dates, and – measures of completion or progress
EVALUATION Work planshave many of the same elements• SMART Objectives
• Activities for each Objective
• Indicators/Measures (of completion or progress)
• Person responsible
• Resources needed
• Target completion dates
• Processes for monitoring, reporting, modifying implementation and evaluation plans
Data sources
for evaluation activities
Actual completion dates
If you are adopting an existing evidence-based program, policy or other strategy...• Conduct formative evaluation on adapted materials
– Feedback from expert panel – Focus groups/discussion with target populations– Pilot testing
• If it was evaluated before, you can use prior process and outcome indicators, methods, and tools
– What outcomes were measured?
– How rigorous was the original evaluation?
– Do you have resources to do a similar evaluation?
• Monitor fidelity of implementing core elements and key process steps (process evaluation)
Evaluation – Using RE-AIM
• REACH your intended target population• EFFECTIVENESS or efficacy• ADOPTION by target staff, settings, or institutions• IMPLEMENTATION consistency, costs and
adaptations made during delivery• MAINTENANCE of intervention effects in
individuals and settings over time
http://www.re-aim.org/
Evaluation PhasesPlanning •Consult with the program stakeholders or
resources•Develop the questions •Identify sources of data •Make a timeline
Data Collection
•Pilot test•How will the questions be asked? •Who will ask them?
Data Analysis
•Who will analyze the data and how?
Reporting •Who will report and how? •Who will receive the data and when? •How will it affect the program?
Gather Credible Evidence
(not the other way around)
IndicatorsEvaluation questions
Methods and Measures
Gather Credible Evidence
• Link each indicator to a data source
• Look for existing data – If there are no data available – collect your own
• Look for existing data collection tools– If there are no existing tools – create your own
• Data collection plan– Who will collect?
– How will it be collected?
– When and How often will it be collected?
Evaluation for Programs
• Track program promotion and/or recruitment(# flyers, frequency of announcements)
• Document implementation of program core elements or key process steps (e.g., checklists, length & # of sessions, meetings)
• Collect process measures (e.g., attendance, timeline/work plan updates, etc.)
• Monitor program activities and performance (# of new screenings, timeliness of diagnosis)
Evaluation for Policies
• Document and publicize policy
• Track implementation of key process steps of the policy– Documentation of policy– Procedures for policy– Communication– Enforcement
• Collect process measures on benchmarks – Track participants in policy planning (partners)– Revisions of policies– Advocacy efforts (e.g., media outlets, events)– Adoption and enforcement
Example Evaluation Plan: Body and Soul
Example Evaluation Plan: Body and Soul
Discussion: Develop an Evaluation Plan
Discussion: Develop an Evaluation Plan
1. Refer to the SMART objective you chose in the work plan activity.
2. Discuss evaluation question(s) that measure success of that objective.
3. What are potential data sources to find this information?
4. What are some methods you can use to measure these processes or outcomes? (e.g. logs, surveys, etc.)
Intervention Planning Model
Community Assessment
DevelopGoals &
Objectives
Plan the InterventionFinding, Selecting and Adapting an Evidence-based Approach
ImplementEvaluate
Outcomes
Share your successes
***Apply lessons
learned to Maintain and Sustain the intervention
Assess Readiness
Evaluate Processes
Share Evaluation Results
• Use your evaluation findings
– to modify your program
– describe program benefits and challenges
• Reach all stakeholders inside and outside your organization
– In language and formats that everyone understands
– In time to make decisions
Maintenance…Plan for Sustainability
Sustainability is…
… the existence of structures and processes which allow a program to leverage resources to most effectively implement evidence-based policies and activities.
Luke D, Calhoun A, Herbers S. (Feb 2012). Building & Managing Sustainability in Public Health Programs. CDC Roundtable Presentation.
Maintenance Activities
Leverage resources to sustain interventions over time:• Provide referrals for further service, when needed• Seek additional funding• Maintain or secure a program champion• Make program a part of organizational services/policies• Surveillance of the impact over time
(e.g., tracking of smoking rates, BMI of children, screening)
• Demonstrate return on investment/cost savings to justify investment of other stakeholders
• Continue partnerships or build new ones to share costs and implementation delivery
Sustainability Planning Resources
https://sustaintool.org/
Schell, S.F., Luke, D.A., Schooley, M.W., Elliott, M.B., Herbers, S.H., Mueller, N.B., & Bunger, A.C. (2013). Public health program capacity for sustainability: A new framework. Implementation Science, 8(15).
Take-home Points: Importance of Implementation and Evaluation
• Successful implementation and evaluation requires detailed, advanced planning
• Involve partners from the beginning to implement with high quality
• Assess organizational readiness and build capacity • Use and adapt pre-existing guidance and tools• RE-AIM can be helpful to measure success in
meeting objectives• Use evaluation findings and lessons learned to plan
for long-term sustainability
Questions?