amy livingston. process evaluation focuses on assessing and documenting grantees’ : ◦...
TRANSCRIPT
Vermont Partnerships for Success
Year One Process Evaluation
Amy Livingston
Process evaluation focuses on assessing and documenting grantees’ :◦Partnerships and capacity building◦Assessment and planning◦Implementation of strategies (including
successes and challenges encountered, and program-level outcomes)
Process Evaluation Questions
Outcome evaluation focuses on:◦Assessing change over time in population-based outcome measures and targeted intervening variables
◦Comparing change in outcomes in grantee communities (both individually and collectively) to the rest of the state
◦Primary sources will be YRBS and YAS
Outcome Evaluation Questions
• Formative Evaluation• Provides template for implementation activity• Helps identify implementation problems/issues that
may be correctable• Documentation/replicability • Provides record of what was done
• Performance monitoring• Provides metrics for assessing progress in achieving
implementation objectives and goals• Interpretation of program outcomes• Helps explain why or why not outcomes were
achieved
Purposes of Process Evaluation
Most of our process data for PFS comes from data submitted quarterly to CGRS
Also site visits, networking calls, meetings with District Office staff
May do some qualitative evaluation (focused interviews and/or focus groups) to help assess changes in regional capacity
How are we doing so far?
All grantees have reported an overall increase in collaboration with community partners since the beginning of the grant
Collaboration and Community Partnerships
Period
10/13 thru 3/14 4/14 thru 9/14
Sector:
Number of Grantees with at
Least One Partner
Number of Grantees with At
Least One Partner
Collaborating at High or Very High
Level
Number of Grantees with at
Least One Partner
Number of Grantees with At
Least One Partner
Collaborating at High or Very High
LevelBusiness community 5 4 6 6Civic/volunteer group 3 3 5 2Clergy/religious organization 3 1 5 3Healthcare professionals 6 3 6 4Judiciary 2 1 3 0Law enforcement agency 6 5 6 5Media 6 4 6 3Parent 5 3 5 3School 6 5 5 4State, local and/or tribal governmental agency
6 4 6 5
Youth 6 3 6 3Youth serving organization 6 4 6 4Other individual or organization involved in preventing or reducing substance abuse 4 3 6 3
Colleges 5 5 5 4Young adults 5 2 5 2
Number of Grantees with Partners in Each Sector and with Partners Collaborating at a High Level
From the beginning of the grant through the report submitted in July 2014, TA received was assessed by intervention and by source (PC, other ADAP staff, etc.)
Starting with the report submitted in October, TA is now being assessed overall by topic area (SPF step) and source
Will be used by ADAP to monitor and improve quality of TA
TA Received
CGRS includes a measure of progress on each of the key activities from the work plan for each evidence-based intervention
Grantees are asked to rate progress on each key activity as not started, partially completed, in progress, or completed
In addition, there are other process measures in CGRS that collect both quantitative and qualitative information for each intervention
Progress on Implementation
Intervention:
Number of grantees
implementing
intervention
Started on at least half of the steps
Started on all of the
steps
Completed all of the
steps
Started but not able to complete 1
or more steps
Restrictions on alcohol outlet density 2 2 0 0 0
Restrictions on alcohol in public places 5 5 0 0 4
Enhancements to social host liability 1 1 0 0 0
Enhancements to open container regulations 1 1 0 0 0
Restrictions on advertising and promotion 1 1 0 0 0
Saturation/Party Patrols 6 5 4 0 3
Community mobilization 6 6 5 0 3
Media advocacy 6 6 4 0 3
e-CHECKUP for College 2 0 0 0 1
Alcohol Edu 1 1 0 0 0
Guiding Good Choices 1 1 1 0 0
Nurturing Parenting Program
1 1 1 1 0
Parenting Wisely 1 1 1 0 0
e-CHECKUP for the Community
1 1 0 0 1
Progress on Activity Implementation, by Intervention, April 2014 through September 2014
Type of policy education Number of grantees implementing
Number of policies developed
Number of policies approved
Enhancements to open container regulations
1 1 1
Enhancements to restrictions on advertising and promotion
1 0 0
Enhancements to social host liability
1 0 0
Restricting alcohol outlet density
2 1 1
Restrictions on alcohol in public places and/or at community events
5 0 0
Education on Policy Approaches
Successes:◦ Two successful town-level policy changes so far◦ Getting health language added to town or regional
plans◦ Systematic review and assessment of current
policies and ordinances◦ Development of summary document that can be
shared with towns throughout the region Challenges:
◦ Lack of stakeholder motivation◦ Lack of existing model policies◦ Takes time!
Education on Policy Approaches
All 6 grantees are working with law enforcement to enhance saturation and/or party patrols (work plans and reporting on these two interventions were combined in July)
Successes: Getting MOUs in place Establishing relationships with law enforcementChallenges: Coordination with different law enforcement agencies Low capacity to conduct additional patrols/utilize overtime
Enhanced Law Enforcement
Law enforcement activity Total number
Proactive patrols conducted 49
Minor in possession citations issued 69
Parties responded to by patrols 21
Type of Intervention
Number of Grantees Implementing
Total number of individuals reached
E-Checkup for College 2 2
Alcohol Edu 1 342
E-Checkup for the Community
2 74
Electronic Screening and Brief Intervention/Education
Successes: High completion rate of Alcohol Edu at JSC Decrease in alcohol policy violations at JSC; from 48 in fall of 2012
to 14 in fall of 2013Challenges: Getting colleges to commit to using e-Checkup
Type of program Number of grantees implementing
Total number of parents reached
Guiding Good Choices 1 46
Nurturing Parenting Program (Nurturing Fathers)
1 16
Parenting Wisely 1 11
Parenting Programs
Successes: Centralized coordination of parenting classes around the county
worked well Positive feedback from participantsChallenges: Getting parents to sign up and follow through with sessions
Activity Number of grantees
Relevant data
Retailer recognition for passing compliance checks
6 • A total of 123 businesses were recognized
Support activities for responsible beverage service training
6 • Grantees assisted with a total of 19 Merchant Education Seminars
Sticker Shock 5 • A total of 42 businesses participated
• A total of 63 youth volunteers participated
Other supporting activities to prevent underage drinking
A total of 3 new safe drop-off locations were established
Number of PFS regions using different types of outreach to educate the community about proper storage and safe disposal of unused prescription drugs.
Prescription Drug Prevention Activities
Type of outreach Number of grantees using
Brochures 4
Community events 6
Print media 4
Press release 3
PSA 2
Radio 3
Social media 6
Television 2
Website 5
Other (e.g. flyers, posters, newsletters, Rx bag inserts) 4
Successes:◦ Creative ways to get the word out e.g. through senior
centers, Meals on Wheels◦ Hungry Heart screenings and opiate summits have
provided good forum for communicating concrete things community members can do
◦ Great collaboration with law enforcement and health care providers on this issue
◦ Collaboration with health care providers and pharmacists has led to new understanding of the issue for all involved and allows for greater saturation of messages
Challenges:◦ Waiting for statewide materials◦ Engaging pharmacists, especially at chain pharmacies
Prescription Drug Prevention Activities
Successes:◦ Youth engagement efforts very strong in some
regions◦ Increased collaboration with schools◦ Community forums on opiate use and Hungry
Heart screenings provided great opportunities Challenges:
◦ Partners and collaborating agencies don’t always identify with county-wide approach
◦ How to engage with culturally diverse sub-groups
Community Mobilization
Grantees contacted the media a total of 185 times The total number of instances of earned media reported across all PFS
regions was 488 (*true number is probably closer to 76)
Number of instances of earned media by type
*In reviewing the description of types of media included in the “other” category, it was clear that some of the instances in this total would not be considered earned media but rather grantee-generated media such as ads, flyers, press releases, etc.
Media Advocacy
Type of media Number of instances
Letter to the editor 3
Newspaper story 43
Editorial/op-ed 6
TV story 16
Radio interview or story 6
Blog story/discussion 2
Other 402*
Number of times each type of message was addressed by earned media
*The description of other types of messages included educational opportunities to prevent substance abuse, and youth engagement (specifically an Above the Influence retreat)
^These items contribute to the ADAP media advocacy performance measure on the Dashboard
Media Advocacy contd.
Type of message Number of instances
Community efforts to reduce UAD 38
Enforcement efforts to reduce UAD 11
Community efforts to reduce Rx misuse^ 68
Community specific data or info on UAD 15
Community specific data or info on Rx misuse^ 29
General info or data related to UAD (not community specific) 21
General info or data related to Rx misuse (not community specific)^ 53
Other* 329
Successes:◦ Positive relationships developed with local media◦ Good coverage of Take Back Days and summits on
opiate use◦ Invitations to be interviewed for TV or papers◦ Invitations for regular media spot on TV or
newspaper column Challenges:
◦ Developing process for coordinated media outreach and tracking at regional level
◦ Ability/capacity to monitor local media
Media Advocacy
Next Steps… PIRE is preparing a year one evaluation brief
for ADAP which is primarily a summary of CGRS data collected so far.
We are also working with ADAP to develop performance measures related to the PFS, some of which may be displayed on the Dashboard
Modify CGRS when CLI goes live