![Page 1: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/1.jpg)
The Need for Rigorous Evaluation of Interventions to Improve the Transition to Adulthood for Youth in State Care Mark E. Courtney School of Social Service Administration and Chapin Hall University of Chicago
![Page 2: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/2.jpg)
My Purpose Today
Present recent research on foster youths’
transitions to adulthood Describe the focus of social policy on this
population Summarize the weak evidence regarding the
effectiveness of interventions for this population
![Page 3: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/3.jpg)
How do foster youth fare during the transition?
![Page 4: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/4.jpg)
Midwest Study Design and Sample
Largest prospective study of foster youth making the transition to adulthood since the Foster Care Independence Act of 1999
Collaboration between state child welfare agencies and the research team
Foster youth in Iowa, Wisconsin and Illinois who: Were still in care at age 17 Had entered care before their 16th birthday Had been placed in care because they were abused, neglected or
dependent Not originally placed because of delinquency
Data from in-person interviews (structured and in-depth qualitative) and government program administrative data
![Page 5: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/5.jpg)
Study Design and Sample (continued)
Wave Year Number Interviewed
Response Rate
Age at interview
1 ’02 – ’03 732 96% 17 – 18
2 ‘04 603 82% 19
3 ‘06 591 81% 21
4 ‘08 602 82% 23-24
5 ’10 – ’11 596 83% 26
![Page 6: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/6.jpg)
Young Women’s Educational Attainment
![Page 7: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/7.jpg)
Young Men’s Educational Attainment
![Page 8: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/8.jpg)
Young Women’s Educational Enrollment
![Page 9: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/9.jpg)
Young Men’s Educational Enrollment
![Page 10: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/10.jpg)
Young Men’s and Young Women’s Employment
72% employed during year; mean earnings among employed = $13,989
![Page 11: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/11.jpg)
Family Formation Among Young Women
19% of women with children have a nonresident child
![Page 12: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/12.jpg)
Family Formation Among Young Men
66% of men with children have a nonresident child
![Page 13: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/13.jpg)
Young Women’s Criminal Justice System Involvement
![Page 14: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/14.jpg)
Young Men’s Criminal Justice System Involvement
![Page 15: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/15.jpg)
Summary of What We Know About Early Adult Outcomes Post Chafee
Outcomes are relatively poor across a variety of domains Trends are generally problematic:
Declining engagement in education, though some are still in school Gradually increasing but poor engagement in the workforce Many non-resident children Troubling levels of justice system involvement continuing through mid
20s Functioning in other domains (e.g., mental and behavioral health, risk
behaviors, victimization) is also poor
Outcomes vary by gender; males fare worse Despite a sobering picture overall, many young people leaving
the care of the state do well
![Page 16: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/16.jpg)
16
US Social Policy and the Transition to Adulthood for Foster Youth
![Page 17: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/17.jpg)
U.S. Demographic, Developmental, and Policy Context
The transition to adulthood in the U.S. is taking longer Markers of the transition are happening later; half of
young people between 18-24 live with a parent $38k in direct support between 18-34
Developmental psychologists describe a new period of “emerging adulthood”
Yet, U.S. policy provides little support for young adults
![Page 18: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/18.jpg)
U.S. Policy on Foster Youth in Transition
Research from 1990s continues to show poor outcomes
1999 Foster Care Independence Act $140 million per year allocated to states Funds a broad range of services Up to 30% of funds can be used for room and board Allows states to extend Medicaid to foster youth through age 21 Amendment to law allows appropriation up to $60 million per
year to fund education/training vouchers for up to $5000 per year through age 23
Creates outcome reporting requirements and devotes 1.5% of funds to rigorous evaluation of promising programs
![Page 19: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/19.jpg)
A Brave New World: The Fostering Connections to Success Act of 2008
Among its provisions, the law: Extends Title IV-E funding (including guardianship and
adoption subsidies), at state option, to age 21 Youth must be 1) completing high school or an
equivalency program; 2) enrolled in post-secondary or vocational school; 3) participating in a program or activity designed to promote, or remove barriers to, employment; 4) employed for at least 80 hours per month; or 5) incapable of doing any of these activities due to a medical condition
Existing IV-E protections remain, including ongoing court oversight of state foster care provision
![Page 20: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/20.jpg)
But…Evidence of What Works is Lacking Cochrane collaboration review of evaluation research on IL programs
(Montgomery et al, 2006) found no rigorous studies: “Further research incorporating randomized designs is both feasible and necessary”
Recent ACF-funded randomized evaluations: No impact of life skills training, tutoring/mentoring, and employment support Massachusetts Adolescent Outreach had some positive effects, but those
appear to be mediated by the program’s impact on youth remaining in care past age 18.
The bottom line: Too many programs are poorly targeted, have poorly
developed logic models, and are not intensive enough to influence outcomes for youth making the transition to adulthood from foster care.
WE NEED RIGOROUS EVALUATION RESEARCH!!!
![Page 21: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/21.jpg)
Evaluation of Youth Villages’ Transitional Living Program
John Martinez
Deputy Director, Health and Barriers to Employment Policy Area
![Page 22: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/22.jpg)
Overview of Session
Introduction to MDRC and evaluation Who is in the study Evaluation Status
2
![Page 23: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/23.jpg)
Who is conducting the evaluation?
MDRC, a non-profit, non-partisan education
and social policy research organization and intermediary
Based in New York City with a regional office in Oakland, CA
Dedicated to learning what works best to improve the lives of low-income families
Nearly 40 years of experience evaluating social policy programs
3
![Page 24: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/24.jpg)
Who is funding the evaluation?
The Edna McConnell Clark Foundation The Annie E Casey Foundation The Bill and Melinda Gates Foundation
4
![Page 25: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/25.jpg)
Components of the Evaluation (I)
Impacts: To what extent does TL improve outcomes for
youth? Which approaches are most effective for whom? On what outcomes (e.g. housing stability, educational
outcomes, labor market success, reduced crime) Over what period of time (e.g. length of follow-up) To what magnitude? For what subgroups?
Use Random Assignment Research Design Data sources: public records data, baseline forms, and surveys
5
![Page 26: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/26.jpg)
Components of the Evaluation (II)
Program implementation: What services are provided? How are they delivered? What challenges are encountered? Are the challenges related to serving youth aging out of state care? Measuring enrollment and participation rates Measuring implementation fidelity Measuring the contrast Data sources: surveys, field research, YV MIS, in-depth
interviews with youth
Costs and Benefits: What are the costs? Do benefits outweigh the costs?
6
![Page 27: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/27.jpg)
Impact Analysis Design
Evaluation uses random assignment, the most reliable way to assess what difference a program makes
Youth eligible for TL were assigned, by chance, to one of two groups: TL Program Group: receives TL services Community Services(CS) Group (Control Group): not eligible
to receive TL services for up to five years
MDRC will follow both groups for at least three years (and possibly longer, depending on funding)
7
![Page 28: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/28.jpg)
Random Assignment
Participants meet program criteria
Participants give consent
Baseline data collected
Random Assignment Control group
Receive other services in the community
Program group Enroll in program
8
![Page 29: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/29.jpg)
Why Random Assignment?
Gold standard of research as it is the most reliable way to measure impacts: Ensures motivation levels and personal characteristics of youth in
program and control groups are same on average at beginning of program
Youth in the control group illustrate what would have happened if not for the program
Any subsequent difference in outcomes can be attributed to the program with the highest confidence
Widely used in public service settings
Endorsed by OMB, DOE and other federal agencies
Fair and equitable way to determine who receives the program
9
![Page 30: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/30.jpg)
Who is in the study?
![Page 31: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/31.jpg)
What is the baseline information form?
Youth completed a baseline survey prior to random assignment
Baseline data provides a “picture” of these youth at study entry
Multiple uses Allows researchers to assess whether random assignment
worked Could be used in impact analysis models Can also be used to determine whether TL works differently
for different types of youth
Presenting data today on 1,225 study participants
![Page 32: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/32.jpg)
Did RA work?
Goal: two research groups in which the only difference is that one group was eligible to receive TL and one was not
Baseline data can help support that the two groups were equivalent
Data suggests this was the case: of 22 variables, only 2 had a SS difference between the 2 groups (and we would expect that to happen by chance)
![Page 33: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/33.jpg)
Demographics
Characteristic Full Sample (%)
Gender
Male 52
Female 48
Race
Hispanic 5
White/non-Hispanic 51
Black/non-Hispanic 38
Other/non-Hispanic 6
![Page 34: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/34.jpg)
Age at random assignment
71%
20%
9%
Age at RA
181920-24
![Page 35: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/35.jpg)
Age at 1st custody placement
1%
6% 6%
23%
64%
LT 11-56-1011-1415-18
![Page 36: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/36.jpg)
Contact with biological mother
Contact with biological father
43%
16% 8%
8%
25%
Every day
At least1X/wkAt least1X/moLT 1X/mo
Never
16%
12%
8% 11%
53%
Every day
At least1X/wkAt least1X/moLT 1X/mo
Never
Contact with biological parent
![Page 37: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/37.jpg)
Other characteristics
Characteristic Full Sample (%)
Contact with any other relative at least 1X/mo 88
Pregnant at baseline 4
Has any children 17
Enrolled in school 54
Ever repeated a grade or held back 43
Ever suspended from school 81
Ever arrested 64
![Page 38: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/38.jpg)
Current Status
Random assignment complete 1322 youth randomly assigned within two years
Most field work associated with the implementation study is complete
Survey fielding is ongoing Very high response rates (about 85 percent)
![Page 39: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/39.jpg)
What’s Next
Survey fielding will continue (survey firm should wrap up winter/spring 2014)
Implementation report slated for publication in January 2014
Impact report slated for publication in spring/summer 2015
![Page 41: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/41.jpg)
All contents ©2011 by Youth Villages, Inc. with all rights reserved
Presentation to First Focus/SPARC March 19, 2013
The promises and perils of random assignment evaluations – a provider’s perspective
![Page 42: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/42.jpg)
Discussion Topics
• Embarking on a random assignment evaluation
• Confronting the ethical issues
• Recruiting study subjects
• Monitoring program fidelity
• Controlling study costs
• Awaiting results
![Page 43: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/43.jpg)
Why do a random assignment evaluation?
• Frightening
• Time consuming
• Referral sources/ service purchasers don’t require it
• Youth Villages already has an on-going outcome evaluation process
• Provides a benchmark
• Value in having independent evaluation
• Exciting
• Opportunity for program improvement
• Push toward evidence-based practices from govt. and foundation funders
• Meets organizational goal of increasing use of evidence-based services
PRO CON
![Page 44: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/44.jpg)
Confronting the Ethical Issues of Random Assignment
Forty percent of youth in the study are denied entry into the
TL Program.
How do we justify that?
![Page 45: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/45.jpg)
Recruiting Study Subjects The expectation:
The reality:
![Page 46: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/46.jpg)
0
100
200
300
400
500
600
700
800
900
1,000
1,100
1,200
1,300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Month
Goal Actual
Original goal – 1,600 Adjusted goal – 1,300
It wasn’t as easy as we thought!
![Page 47: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/47.jpg)
Monitoring Program Fidelity
In some ways, this has been the easy one!
• Program Model Adherence Reviews
• Balanced Scorecard
Is that enough?
![Page 48: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/48.jpg)
Controlling Study Costs
The evaluation is funded by the Edna McConnell Clark Foundation and the Bill and Melinda Gates Foundation
What are the provider’s costs?
• Study coordinator (full time) • Leadership staff time • Assessors (to determine eligibility and
appropriateness for program) • Training and travel
![Page 49: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/49.jpg)
Awaiting Study Results
Planning Begins – July 2008
Study Recruitment Begins – October 2010
One Year Follow-up Begins – November 2011
Study Recruitment Ends – October 2012
One Year Follow-up Completed – January 2014
Preliminary Report on One Year Outcomes – July 2014
Six Years!
![Page 50: Finding what works helping young adults transition into adulthood](https://reader034.vdocuments.us/reader034/viewer/2022051817/54874a4cb47959190d8b53a4/html5/thumbnails/50.jpg)
Lessons learned?
• Go in with eyes wide open
• Take time to prepare
• Be ready for the unexpected