achieved relative intervention strength: models and methods chris s. hulleman david s. cordray...
TRANSCRIPT
![Page 1: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/1.jpg)
Achieved Relative Intervention Strength:
Models and Methods
Chris S. Hulleman
David S. Cordray
Presentation for the SREE Research ConferenceWashington, DC
March 5, 2010
![Page 2: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/2.jpg)
Overview
• Conceptual Framework– Definitions and Importance– Indexing Fidelity as Achieved Relative Strength (ARS)
• Three examples– Lab and Field Experiments– Reading First
• Practical Considerations and Challenges• Questions and discussion
![Page 3: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/3.jpg)
Definitions and ImplicationsFidelity
– The extent to which the implemented Tx (tTx) was faithful to the intended Tx (TTx)
– Measure core intervention components
Achieved Relative Strength (ARS)– The difference between implemented causal components in the Tx and C– tTx – tC
– ARS is a default index of fidelity
Implications– Infidelity reduces construct, external, and statistical conclusion validity
![Page 4: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/4.jpg)
Achieved Relative Strength = 0.15
Infidelity
“Infidelity”
0.50d
85 700.50
30d
t c
pooled
Y Yd
sd
(85)-(70) = 15
tC
t tx
cY
tY
TTx
TC
.45
.40
.35
.30
.25
.20
.15
.10
.05
.00
Treatment Strength
with fidelity
with fidelity
90 650.83
30
T C
pooled
Y Yd
sd
d
Expected Relative Strength = TTx - TC = (0.40-0.15) = 0.25
100
90
85
80
75
70
65
60
55
50
Outcome
TY
CY
![Page 5: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/5.jpg)
Indexing Fidelity as Achieved Relative Strength
Intervention Strength = Treatment – Control
Achieved Relative Strength (ARS) Index
• Standardized difference in fidelity index across Tx and C• Based on Hedges’ g (Hedges, 2007)• Corrected for clustering in the classroom (ICC’s from .01
to .08)• See Hulleman & Cordray (2009)
Tx C
T
ARS IndexS
t t
![Page 6: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/6.jpg)
Indexing Fidelity
Average– Mean levels of observed fidelity (tTx)
Absolute– Compare observed fidelity (tTx) to absolute or
maximum level of fidelity (TTx)
Binary– Yes/No treatment receipt based on fidelity scores
– Requires selection of cut-off value
![Page 7: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/7.jpg)
Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a
Motivation Intervention
Examples 1 and 2
![Page 8: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/8.jpg)
PERCEIVED UTILITY VALUE
INTEREST
PERFORMANCE
MANIPULATED RELEVANCE
Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)
The Theory of Change
Fidelity Measure:Quality of participant
responsiveness (0 to 3 scale)
![Page 9: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/9.jpg)
Achieved Relative Strength Indices
Observed Fidelity
Lab vs. Class Contrasts
Lab Class Lab - Class
Average Tx 1.73 0.74
C 0.00 0.04
g 2.52 1.32 1.20
Absolute Tx 0.58 0.25
C 0.00 0.01
g 1.72 0.80 0.92
Binary Tx 0.65 0.15
C 0.00 0.00
g 1.88 0.80 1.08
![Page 10: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/10.jpg)
Achieved Relative Strength = 1.32
Fidelity
Infidelity
Infidelity
TTx
TC
0.74 0.04ARS 1.32
0.53g
t c
pooled
X XARS g
sd
100
66
33
0
Treatment Strength
tC
t tx
cX
tX
3
2
1
0
Average ARS Index
(0.74)-(0.04) = 0.70
![Page 11: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/11.jpg)
Assessing Implementation Fidelity in a Large-Scale Policy Intervention: The
Case of Reading First
Example 3
![Page 12: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/12.jpg)
In Education, Intervention Models are Multi-faceted (from Gamse et al., 2008)
Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application
Teacher professional development in the use of materials and instructional approaches
1)Teacher use of instructional strategies and content based on five essential components of reading instruction
2) Use of assessments to diagnose student needs and measure progress
3) Classroom organization and supplemental services and materials that support five essential components
![Page 13: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/13.jpg)
From Major Components to Indicators…
Professional Development
Reading Instruction
Support for Struggling Readers
Assessment
Instructional Time
Instructional Material
Instructional Activities/Strategies
Block
Actual Time
Scheduled block?
Reported time
Major Components
Sub-components
Facets Indicators
![Page 14: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/14.jpg)
Reading First Implementation: Specifying Components and Operationalization
Components Sub-components Facets Indicators(I/F)
Reading Instruction
Instructional Time 2 2 (1)
Instructional Materials 4 12 (3)
Instructional Activities /Strategies 8 28 (3.5)
Support for Struggling Readers (SR)
Intervention Services 3 12 (4)
Supports for Struggling Readers 2 16 (8)
Supports for ELL/SPED 2 5 (2.5)
Assessment Selection/Interpretation 5 12 (2.4)
Types of Assessment 3 9 (3)
Use by Teachers 1 7 (7)
Professional development
Improved Reading Instruction 11 67 (6.1)
4 10 41 170 (4)
Adapted from Moss et al. 2008
![Page 15: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/15.jpg)
Reading First Implementation: Some ResultsComponents Sub-
componentsPerformance Levels (% of Absolute Standard)
AbsoluteStandard
ARSI
RF Non-RF
Reading Instruction
Daily (min.) 105 (117%) 87 (97%) 90 0.63
Daily in 5 components (min.)
59 50.8 -- 0.35
Daily with High Quality practice
18.13 16.2 -- 0.11
Professional Development
Hours of PD 25.8 13.7 -- 0.51
Five reading dimensions
4.3 (86%) 3.7 (74%) 5 0.31
0.38
Adapted from Gamse et al. (2008) and Moss et al. (2008)
![Page 16: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/16.jpg)
Linking Fidelity to Outcomes
![Page 17: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/17.jpg)
ARS: How Big is Big Enough?
Effect SizeStudy Fidelity
ARSOutcome
Motivation – Lab
1.88 0.83
Motivation – Field
0.80 0.33
Reading First*
0.35 0.05
*Averaged over 1st, 2nd, and 3rd grades (Gamse et al., 2008).
![Page 18: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/18.jpg)
What Do I Do With Fidelity Indices?
Start with:– Scale construction, aggregation over model
sub-components and components
Use as:– Descriptive analyses– Causal analyses (Intent-to-Treat: ITT)– Explanatory (AKA exploratory) analyses
• E.g., LATE, Instrumental variables, TOT
Except for descriptive analyses, most approaches are relative new and not fully tested
![Page 19: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/19.jpg)
In Practice….• Identify core intervention components
– e.g., via a Model of Change• Establish bench marks for TTX and TC
• Measurement– Determine indicators of core components– Derive tTx and tC
– Develop scales– Convert to ARS
• Incorporate into intervention analyses– Multi-level analyses (Justice, Mashburn, Pence, &
Wiggins, 2008)
![Page 20: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/20.jpg)
Some Challenges
Intervention models– Often unclear– Scripted vs. Unscripted
Measurement– Novel constructs– Multiple levels– Aggregation (within and across levels)
Analyses– Weighting of components– Uncertainty about psychometric properties– Functional form not always known
![Page 21: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/21.jpg)
Summary of Key Points
• Identify and measure core components• Fidelity assessment serves two roles:
– Average causal difference between conditions– Using fidelity measures to assess the effects of
variation in implementation on outcomes
• Post-experimental (re)specification of the intervention
• ARS: How much is enough?– Need more data!
![Page 22: Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,](https://reader030.vdocuments.us/reader030/viewer/2022032313/56649e675503460f94b61f21/html5/thumbnails/22.jpg)
Thank You
Questions and Discussion