governor’s prevention initiative for youth: evaluation jane a. ungemack, dr.p.h. evaluator...
TRANSCRIPT
Governor’s Prevention Initiative for Youth: Evaluation
Jane A. Ungemack, Dr.P.H.
Evaluator
University of Connecticut Health Center
Evaluation
• Systematic efforts to collect and use information:– Document program implementation– Describe target populations/participants– Inform and improve program performance– Access program effectiveness– Increase accountability– Increase understanding
Governor’s Prevention Initiative for Youth
Long-term Goal: Reduce substance use among adolescents
Intermediate Goal: Reduce risk factors and increase protective factors for
substance use in the individual, peer, family, school and community domains
Target Group: 12-17 year old youth
Evaluation Framework OverviewII. Contextual Conditions
(Economic, Cultural, Risk Conditions, ATOD use)
2. State System 4. State-Level 6. State-Level 8. State-LevelCharacteristics/ Collaborative Intermediate Systems
Dynamics Strategies/ Outcomes ChangeActivities
Training/TA Training/TA
1. SIGMobilization
3. Sub-RecipientCharacteristics
Training/TA
5a. SubRecipient
Planning forScience-based
PreventionInterventions
7a. Sub-Recipient/CommunityImmediate
Outcomes & systemchanges
9. CommunityIntermediateOutcomes:Risk and
ProtectiveFactors
10. Long-termOutcomes:Behavioral
Impacts
5b. ProgramInterventions:
Activities/Actions
7b. ProgramOutcomes: Risk and
Protective Factor
Evaluation Approach
Process evaluation:
• documents program implementation and activities
Outcome evaluation:
• assesses program effects or impacts
Capacity-Building for Evaluation
Science-based approach
Evaluation and assessment as integral parts of the program design
– Community-level
– Program-level
Evaluation Team Relationship with Grantees
• Training and technical assistance
• Instruments and administration protocols
• Consultation and collaboration
• Statewide coordination
Grantee Responsibilities
• Develop a program plan based on the logic model • Specify measurable program objectives• Cooperate and collaborate with UConn Evaluation
Team• Coordinate community survey• Collect and submit process data • Collect and submit outcome data • Commit time and effort to evaluation activities
Assessing Community-Level Outcomes: CSAP Requirements
School survey
Use of core substance use, risk and protective factor measures
Community-Level Assessment: School Survey
• Mandated
• 7th-10th grade students
• Representative sample (minimum n=500; 125/grade level)
• Year 1 and Year 3
• First administration: February-April, 2000
School Survey
• Self-administered during a classroom period
• Anonymous and confidential
• Parental consents
• Sampling, instrument and administration protocols provided by UConn Evaluation Team
School Survey
Measures– Demographic characteristics– Lifetime and current use of ATOD– Risk and protective factors– Limited community-specific items
School Survey
Grantee responsibilities:– Planning/coordination with UConn Evaluation
Team– Planning/coordination with school personnel– Instrument duplication– Data cleaning– Data entry– Analysis
Assessing Program-Level Outcomes: CSAP Requirements
• Select a minimum of three programs (for each of three domains)
• Measure program outcomes using core measures
• Include a sufficient sample size for analysis
• Collect pre- and post-test data
Program-Level Evaluation
Process evaluation:• Document program implementation and activities
Outcome evaluation: • Assess program outcomes
Program-Level Evaluation: Process Evaluation
Each program will be responsible for reporting:– Prevention strategies– Types of activities– Dosage– Number served– Participant characteristics (age, gender,
race/ethnicity, etc.).
Process Evaluation
• Minimum Data Set (MDS)
• Instruments, protocols, and training provided by the UConn Evaluation Team
Program-Level Evaluation:Outcome Evaluation
• Based on the logic model, identify measurable objectives that you will address in your program
• Program objectives should be selected from one or more of the risk/protective factors included in the RFP list of Connecticut Intermediate Outcomes
Outcome Evaluation
• UConn Evaluation Team staff will work with each grantee to finalize program-specific objectives and measures
• All grantees will be asked to participate in a pre/post-test assessment of age-eligible participants as appropriate
Outcome Evaluation
Pre- and Post-Test Assessments
• Youth participation will be voluntary
• Confidential
• Informed consents
• Standardized instrument plus optional program-specific items
• Minimum sample size = 50
Considerations for Estimating Evaluation Costs
• Personnel (.25 FTE minimum recommended)
• Computer equipment
• Photocopying
• Office supplies
• Data collection and cleaning
• Data entry