using a fidelity index to increase program attribution
TRANSCRIPT
Using a Fidelity Index to Increase Program Attribution in Impact Evaluation Studies
Presented by Donna Smith-Moncrieffe
Canadian Evaluation Society-Ontario Chapter
October 4th-5th, 2010
Presentation Outline
� National Crime Prevention Centre (NCPC) mandate
� What is a Quantifiable Fidelity Index? Why is it Important?
� How can using a Quantifiable index increase Program Attribution?
� Steps: From Fidelity Tool Development to interpretations regarding program attribution
2
� Steps: From Fidelity Tool Development to interpretations regarding program attribution
� Sample Fidelity tool and output data for two Model programs
� Towards No Drug Program (results)� Stop Now and Plan Program ® (application of treatment intensity
measures)
� Summary
NCPC Mandate/Core Activities
Mission statement:
To provide national leadership on effective and cost-efficient ways to both prevent and reduce crime by addressing known risk factors in high risk populations and places
Core activities:
3
Core activities:
� Supporting targeted interventions in local communities
� Building and sharing practical knowledge with policy makers and practitioners
NCPC Priorities:
Provide funding to the following target groups/crime issues:
� Children and youth at risk
� Crime prevention in Aboriginal communities
4
� Prevent recidivism among high-risk groups
� Priority crime issues (youth gang, drug-related crimes)
Use of Evaluation in NCPC
NCPC Time-Limited Funding
Results contribute to Identify and encourage the development of
Disseminate resultsand encourage
the province and municipalities to replicate
effective programs
5
UseMulti-site
EvaluationsTo IncreaseKnowledge
Results contribute to Treasury Board
Reports
Encourage the development of Fidelity Tools to
Increase Attribution
Identify and encourage the development of Model and Promising Programs
What is Program Fidelity ?
� Fidelity in evaluation is used to describe the extent to which the initiative/intervention corresponds to the originally intended program
� The literature uses the following terms interchangeably
6
� Adherence� Compliance� Integrity
Benefits of a Fidelity Index
A Fidelity Index:
� Lists the elements that contribute to the success of the program
� Makes the important elements of the program visible and helps evaluators “bound” the program
When expected results are unfavourable, model programs can
7
� When expected results are unfavourable, model programs can attribute the results to low fidelity levels while still being able to maintain a reputation for being an effective program
� Identifies what elements practitioners should focus on to increase compliance
� Ensures that elements in the index will have a complementary quality assurance protocol and definitions for practitioners to review
What is a Quantifiable Fidelity Index?
� A quantifiable fidelity index is a set of measureable items that determine whether the program elements have been implemented as planned.
� Each item provides a quantifiable measure using ratio or interval level data that can be later used to test for statistical significance, changes in effect sizes or clinical significance
8
� The items in the index can be checked for construct validity. We can confidently determine if we are measuring what we think we are measuring. (i.e. Using Chronbach’s alpha and other reliability tests can be achieved by using quantitative measures)
� Composite indices and aggregate scores can be inserted into multivariate analysis to determine whether the fidelity related scores contribute to participant related changes.
How Can a Measureable Fidelity Index Contribute to Better Program Attribution?
� You can confidently answer the question, ‘Did the model program contribute to change or were additional unintended program elements contributing to changes ? (i.e. Was the inconsistent implementation of cognitive behavioural sessions responsible for the lower than expected changes in anger for boys in the SNAP program?)
� Each element that is quantified can be placed into a multivariate causal model. The evaluator can:� Isolate whether the program’s adherence levels contributed to
favourable or unfavourable change in the outcomes of interest
9
favourable or unfavourable change in the outcomes of interest� Isolate what specific aspect of the program contributed to greater effects
or changes in the outcomes of interest� Identify the need to find other explanatory variables that are contributing
to program impact (i.e. R squared adjusted result < .80 informs us that there are other factors that account for the change in reduction of drugs or offending for example)
� Identify if low fidelity to the program is contributing to variation in the variables of interest (i.e. Does low participant responsiveness in 25% of the TND classrooms affect the overall goal of preventing or reducing drug use in youth?)
Use of Fidelity in Four NCPC Model Programs(Based on literature prior to NCPC Implementation)
75% of model programs used fidelity for
Process evaluation only
25% of model programs used fidelity for
Outcome/Impact Evaluations
10
Method of Attribution is limited toInferences about how
Program is related to outcomes
Method of Attribution is stronger and can identify how much
Correlation exists between the Program elements and outcomes
Key Steps in Using a Quantifiable Fidelity Tool to Increase Attribution
STEP 1: Identify all the key program elements using a comprehensive framework
STEP 2: Apply appropriate measures
STEP 3: Construct the causal model
11
STEP 3: Construct the causal model
STEP 4: Calculate and interpret the coefficients (multivariate analysis)
STEP 5: Reporting: Enhance program attribution. Determine if program fidelity levels contributed to outcomes
STEP 1: Identify all key elements using a comprehensive framework
Based on Dane & Schneider Framework (1998)
Develop the fidelity tool and include all elements in the following four areas:
1. Implementation: Was the program delivered as intended? (i.e. Were all 8 sessions delivered on a weekly basis over a 2 month period?)
2. Dosage: How much of the program was delivered (i.e. Did the youth receive the 48 hours of individual case management?)
12
the 48 hours of individual case management?)
3. Quality: Are the main components of the program delivered clearly and correctly?
4. Participant Responsiveness: Does the program stimulate interest among the participants and practitioners? (i.e. Are teachers interested in implementing the curriculum to students?)
5. Monitoring Control/Comparison fidelity- What services did the comparison/control group receive? Did the program adhere to their intention to provide minimal services or “usual care”?
STEP 2:Apply Appropriate Measures
Durlak and Dupre 2008 reviewed 59 studies (1998-2005) that had some type of fidelity tool and found the following typical measures:
� 27/59 studies (46%) used categorical measures� Ordinal scales (i.e. assigning a definition to low, moderate and
high fidelity levels)
13
high fidelity levels)
� 32/59 studies (54%) used continuous variables/interval measures� Averages reported as percentages (i.e. observations of each
session identified that 65% of the youth showed interest in the counselling sessions)
� Likert scales results were converted to continuous variables� Used actual # of hours, # of sessions, # of months especially for
dosage related measures
STEP 2:Sample Treatment Intensity Measures
(Stop Now and Plan® Program)
Sample : Quantifiable fidelity checklist (note: this is not the full checklist)v Target Group Population Met: 8/10
v 12 Cognitive Behavioural group Sessions 10/12� Sessions delivered weekly within 3 months
� Delivered by a trained facilitator� Approved manuals used (weekly documentation)
14
� Approved manuals used (weekly documentation)
v 12 Cognitive Behavioural concurrent parenting sessions 10/12
Fidelity Level: High Total Score: 28/34 or 82.4%____________________________________________________________
Fidelity LegendNo Fidelity: 0-29%Low Fidelity: 30-69%
Moderate Fidelity: 70-79%High Fidelity: 80-100%
STEP 3: Construct the causal model
(Multiple Regression Analysis sample)
� A multiple regression equation can be used to make estimates about key program outcomes (i.e. recidivism) based on given values for a number of explanatory variables (i.e. levels of drug use, # of hours participating in treatment, and type of practitioner)
15
� Ensure all statistical assumptions are met before using the model
� Construct the causal model by using the following equation :
y = β0 + β1X1 + β2X2 + …. + β kXk + ε
STEP 4 : Calculate and interpret coefficients
Interpretations
Overall, we are looking at the marginal contribution that each X
(explanatory) variable is making on the Y (outcome) variable when the
other X variables are being held constant.
Review and interpret the output and consider the following
16
Review and interpret the output and consider the following questions:
� Are the elements from the fidelity tool statistically significant?
� What elements of the fidelity tool have higher values and contribute more to change in the key outcomes of interest?
� What is the overall contribution of the program elements on the outcome?
� How much does the impact of fidelity have on program outcomes?
STEP 4: Calculation and Interpretation of Coefficients
cont’d
B e c le a r a b o u t h o w yo u g o t fro m yo ur fid e l ity to o l to d isc us sing wh e th e r a d h e re n ce le v e ls h a d a n im p a ct o n p ro gra m o u tc o me s …
17
1 1
“I th ink y ou sho u ld be m ore exp lic it h e re
in ste p tw o .”
im p a ct o n p ro gra m o u tc o me s …
Th e re a re n o m ira cle s h e re … ..
Step 4: Calculate and Interpret Coefficients (sample data only)
Y= Aggressive Behaviour
Constant 5.274
Coefficient of X1
(CBS) Cognitive Behavioural Sessions (0-12 hours)0.4487
18
Coefficient of X2
(DSH) Duration of Service Hours (total program hours)0.3334
Coefficient of X3
(FPS) Levels of Fidelity (Participant Responsiveness)0.2448
R 2 (adj.) 0.7553
Step 4: Calculate and Interpret Coefficients
� For every increment of cognitive behavioural sessions, aggressive behaviour decreases by 0.4487
� For every increment of cognitive parental group sessions, aggressive behaviour decreases by 0.3334For every increment of fidelity (responsiveness), aggressive
Y= 5.274 - 0.4487 (CBS) - 0.3334 (CPG) - 0.2448 (FPS)
R Square (adj.) =.7553
19
� For every increment of fidelity (responsiveness), aggressive behaviour declines by 0.2448
� R square (adjusted) : The treatment intensity of the cognitive behavioural session, duration of service hours and total group responsiveness contributed to 76% of the variability in Aggression levels.
� Fidelity levels related to group responsiveness are positively correlated with the outcomes measuring changes in aggression levels but contribute less than treatment related factors
Step 4 Calculation cont’dUse Multi-Level Analysis
� Include fidelity scores in the ANOVA or regression equation
� Use a multilevel analysis approach:Group fidelity measure� Participant responsiveness� Implementation measures� Quality of Implementation
20
Quality of Implementation
Individual measures� Dosage (quantity)
� Ensure all key quantifiable fidelity scores are tested for separate cohorts (i.e. various classrooms, sites or session). Correlate fidelity scores with key results (i.e. changes in knowledge, attitudes and behavioural changes)
Step 5: Reporting ResultsWhat Really Contributed to the Results?
High Levels of Fidelity to the Program
Low levels of Fidelity to the Program
Favourable ChangesMade in the
Outcomes of InterestIncreased
Confidence that the Program is Effective
(It Works!!)
�Determine whether fidelity scores or core treatment measures are contributing to results�Qualify results by explaining how fidelity may have contributed to results
21
(It Works!!) results�Be cautious about reporting on program attribution
Unfavourable Changes
Made in the Outcomes of Interest
IncreasedConfidence that the
Program is Ineffective
Explore other sources of data to explain findings:� How many elements of the fidelity index were low?�What elements of the index were low?
STEP 5: Determine if Program Outcomes are related to Fidelity(Towards No Drugs sample)
Outcome FidelityProgram-specific knowledge 0.33 (0.09)***Beliefs: health-as-a-value 0.10 (0.04)**Beliefs: pro-drug myths -0.15 (0.06)**Cigarette intentions -0.08 (0.04)**
22
Cigarette intentions -0.08 (0.04)**Marijuana intentions -0.09 (0.04)**Alcohol intentions -0.07 (0.04)*
Multi-level models * p< .10 ** p < .05 *** p < .0001, one tailed
Rohrbach, L.A., Gunning, G., Sussman, S., & Sun, P. (May; 2008).Predictors of implementation in the Project Towards No Drug Abuse dissemination trial.
Summary
� Attempt to utilize 5 categories: Use Implementation, Dosage, Quality, Participant Responsiveness and Monitoring control group fidelity categories to ensure the tool/index is comprehensive
� Utilize a participatory approach to identify the key elements
� Use a statistical approach (backward or forward selection procedures)
� Evaluators should advocate for the development and use of a
23
� Evaluators should advocate for the development and use of a quantifiable fidelity tool to be implemented in the evaluation study
� Encourage the use of interval levels of measurement to each item in the tool. This type of measure can provide specific information about incremental changes in the outcomes of interest
� Qualitative information should also be collected to explain why the results may be favourable or unfavourable
Summary
� Construct a multivariate analysis model that will incorporate elements of the fidelity tool
� Consider using multilevel models that utilize more than one equation that will take into consideration different levels of data (i.e. Towards No Drugs fidelity tool required a multilevel model for the analysis: schools are nested in communities; pupils are nested in schools etc..)
� Report how the fidelity levels related to the outcomes of interest
24
� Where fidelity levels and expected results are low, ensure that triangulation with other data is used to verify program attribution levels
� Explore other elements of the regression or other multivariate equations used to isolate what program elements may or may not have contributed to the results.
Contact Information
Donna Smith-Moncrieffe, BSc., Crim Dip, MSc.Senior Evaluation Advisor
Public Safety CanadaNational Crime Prevention Center
25
Policy Research and Evaluation DivisionE-mail: [email protected]