getting inside the “black box” – capitalizing on natural and random variation to learn from...
TRANSCRIPT
Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact StudyPresenters:
Alan Werner, co-Principal Investigator
Laura Peck, co-Principal Investigator
Project Director:
Gretchen Locke
APPAM Conference November, 2013
Abt Associates | pg 2
Presentation Overview
What we know and don’t know about what works
The HPOG Impact Study
Strategies to “Get Inside the Black Box”
Q&A
Abt Associates | pg 3
Effects of Training: Progress to Date
It is well-established that vocational training and employment support programs for low-income individuals work
– Strong experimental impact research supports claims of effectiveness of specific approaches
– Best when program model tested in multiple sites and/or when program models tested head-to-head (e.g., HCD vs. LFA in NEWWS)
– But, new program models within Career Pathways framework-based programs, and…
Abt Associates | pg 4
Challenges in Evaluating Training
Challenge has been to get inside the “black box” of program to develop findings more useful for policy and program design, for example:
– What “parts” of a tested program are most responsible for impacts?
– What accounts for variation in impacts across multiple program realizations?
– What design and implementation strategies work better than others?
– What works best for whom?
Abt Associates | pg 5
HPOG and Its Impact Evaluation
Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers
HPOG-Impact is part of a rich research “portfolio” at ACF
Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations
Abt Associates | pg 6
Study Sample and Data Collection
Sample size– Individuals: about 10,500 overall: 7,000 T; 3,500 C
– Study sites: 38 study sites programs across 20 grantees
– Planned variation sample (TBD)• Peer support
• Emergency financial assistance
• Non-cash incentives
Data collection– At baseline (before RA), from PRS & supplement
– Quarterly wage data (NDNH)
– Follow-up surveys at 15 months post-randomization
– Implementation study site visits
– Grantee and other surveys from NIE
Abt Associates | pg 7
Research Questions
HPOG-Impact will address the following questions:
1. What impacts do the HPOG programs as a group have on the outcomes of participants and their families?
2. To what extent do those impacts vary across selected subpopulations?
3. Which locally-adopted program components influence average impacts?
4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?
Abt Associates | pg 8
Research Questions
HPOG-Impact will address the following questions:
1. What impacts do the HPOG programs as a group have on the outcomes of participants and their families?
2. To what extent do those impacts vary across selected subpopulations?
3. Which locally-adopted program components influence average impacts?
4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?
Abt Associates | pg 9
Sources of Variation
Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
Abt Associates | pg 10
Sources of Variation
Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
Abt Associates | pg 11
Intervention Components by Site and RA Sample: An Illustration
SiteP Q R S N
Type of Program Component: Naturally occurring, universal F F F F F Naturally occurring, varied – G E H – Randomized enhancement – E – – EComponents Provided for Each Random Assignment Sample:
C group sample A B C D LT1 sample A, F B, F, G C, F, E D, F, H L, FT2 sample – B, F, G, E – -- L, F, E
Abt Associates | pg 12
Sources of Variation
Extracting the best information on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
Abt Associates | pg 13
Conceptually…
When exposed to treatment…
• used program component (e.g., emergency assistance)
• achieved interim outcome (e.g., recognized credential)
If exposed to treatment, would have…• used program component
(e.g., emergency assistance)• achieved interim outcome
(e.g., recognized credential)
Abt Associates | pg 14
Practically… Step 1
Step 1: Use baseline (exogenous) characteristics to predict subgroup membership– To capitalize on the internal validity of the experimental
design, use symmetric identification of T and C subgroups
HPOG application– Use baseline variables…
• demographics, supplemental baseline Qs on efficacy, work preferences, barriers/needs
– …to predict participation in various program components:• used emergency assistance, supports, used child care
support/services, accessed majority of available supports
– …or to predict selected achieved/short-term outcomes (mediators):
• earned recognized credential, found healthcare job
Abt Associates | pg 15
ASPES: Steps 2 & 3
Step 2: Estimate impacts on predicted subgroups– Impact estimates are unbiased
Step 3: Convert estimated impacts for predicted subgroups to represent actual subgroups– Conversion rests (1) on an assumption of the homogeneity
of impacts among those predicted to be in a subgroup; and (2) on the foundation of an experimental impact estimate
Further Information
Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]
Alan Werner & Laura PeckCo-Principal InvestigatorsAbt Associates [email protected][email protected]