Efficacy to Implementation in the Efficacy to Implementation in the Child Welfare SystemChild Welfare System

J. Landsverk*, P. Chamberlain**, J. Reid**

(Eve Reider***, Chair)

PSMG

October 26, 2005

* Child and Adolescent Services Research Center, San Diego, CA

** Oregon Social Learning Center & Center for Research to Practice, Eugene, OR

*** National Institute on Drug Abuse, Rockville, MD

• Cascading Dissemination of a Foster Parent Intervention (R01 MH 60195)Principal Investigator: Patti Chamberlain Co-Investigators: Joe Price, John Reid, John LansverkFunding Agency: National Institute of Mental Health

• Preventing Health-Risking Behavior in Delinquent Girls (R01 DA15208-1)Principal Investigator: Patti Chamberlain Co-Investigators: John Reid, Leslie LeveFunding Agency: National Institute of Drug Abuse

• Preventing Problems for Girls in Foster Care (R01 MH54257-6)Principal Investigator: Patti Chamberlain Co-Investigators: Leslie Leve, John ReidFunding Agency: National Institute of Mental Health

• Oregon Prevention Research Center (2 P30 MH 46690)Principal Investigator: John Reid Co-Investigators: Chamberlain, DeGarmo, Eddy, Fisher, Leve, Marinez, Fetrow, Patterson, Forgatch, Bank, Shortt, CapaldiFunding Agency: National Institute of Mental Health

• Pathways Home: Reducing Risk in the Child Welfare System (1 P20 DA017592)Principal Investigator: John Reid Co-Investigators: Chamberlain, DeGarmo, Eddy, Fisher, Leve, Martinez, Fetrow, Bronz, Sprengelmeyer, and SmithFunding Agency: National Institute of Mental Health

• Patterns of Youth Mental Health Care in Public Service Systems (U01 MH55282) Principal Investigator: Richard HoughFunding Agency: National Institute of Mental Health

• Mental Health Services Across Child Welfare Agencies (R01 MH59672)Principal Investigator: John Lansverk

Funding Agency: National Institute of Mental Health

• Center for Research on Child and Adolescent Mental Health Services (P50 MH50313)Principal Investigator: John Lansverk

• Improving Care for Children in Child Welfare (R24 MH67377) Principal Investigator: John LansverkFunding Agency: National Institute of Mental Health

• Mixed-Methods Study of a Statewide EBP Implementation (R01)

Principal Investigator: Greg AaronsFunding Agency: National Institute of Mental Health

Child Welfare Service Sector – Child Welfare Service Sector – Characteristics Related to Design Characteristics Related to Design

IssuesIssues

Child Welfare as a Mission ContextChild Welfare as a Mission Context

• Effectiveness and implementation studies are service sector context specific (usual care), efficacy studies usually are not.

• Mission of Child Welfare– Child Safety– Child Permanence– Child Well-Being

– Safety & permanence services provided by CW– Well-Being services provided by other sectors (mental health,

substance abuse services)

Service Sector Comparisons on Risk level & Service Sector Comparisons on Risk level & Intervention Type, Reason for Entry & Sex RatioIntervention Type, Reason for Entry & Sex Ratio

Risk Status

Level

Intervention

Type

Sector of Care

Reasons for Entry

Sex Ratio

General Health

Physical Health Care

Even

Primary

Universal Education

Education

Even

Child Welfare

Maltreatment/Caregiver

absence

Even

Secondary

Targeted

Juvenile Justice

Legal violations

M > F

Substance Abuse

Substance abuse

M > F

S.E.D. Mental health problems

M > F

Tertiary

Clinical

Mental Health Mental health problems

M > F

Service Sector Comparison on Service Sector Comparison on Age at EntryAge at Entry

Sector of Care Infancy / Toddlerhood

Childhood Adolescence

General Health Primary Care

Education

Child Welfare

Juvenile Justice

Substance Abuse

S.E.D.

Mental Health

= Most common entry age into service sector

Context of Child Welfare Context of Child Welfare

• Yearly: 5,000,000 referred, 3,000,000 investigated (4.5% of all children), 826,000 substantiated for child abuse and neglect

• Multiple living environments for child when services delivered: (1) home of origin, (2) out-of-home care or foster care [relative/non-relative, congregate care], (3) adoptive home

• Service trajectories may involve movement from one child setting to another

Nationally Representative Child Nationally Representative Child Welfare Cohort Study (NSCAW)Welfare Cohort Study (NSCAW)

• 92 primary sampling units, 6,000+ investigated cases (substantiated or indicated), followed for 36 months

• 5 months after investigation– 1% in residential care (congregate) (5.7% after 36

months)– 10% in relative or non-relative family foster care– 24% received services in parent’s home– 65% no further services beyond investigation

Need and Opportunity for Need and Opportunity for Preventive InterventionsPreventive Interventions

• High rates of mental disorders (42% for ages 6-17 – POC San Diego study, highest rates for ADHD, ODD, CD), developmental problems , social and cognitive problems

• At any given time, 1% of children under 18 live in out-of-home care, estimated to be 3-7% of all children at some time

• Externalizing problems, mission elements of safety and permanence >>> parent mediated interventions

Child Welfare Context and Effectiveness Child Welfare Context and Effectiveness Study Design and Measurement IssuesStudy Design and Measurement Issues

• Multiple child settings – multiple environmental contexts and multiple parent informants for longitudinal designs

• Clustering at entry – multiple children removed from same family and placed in out-of-home care with confounding by maltreatment type ( more clustering for neglect)

• Clustering for case worker functions – supervisory units of 8-10 case workers

• Timing for identification, sampling and enrolling because of early exit bias

Child Welfare Context and Effectiveness Child Welfare Context and Effectiveness Study Design IssuesStudy Design Issues

• Little tradition of randomized study designs and use of standardized measurement

• Labor force: (1) case workers - BA and MSW in social work, (2) para-professional foster parent (relative & non-relative), congregate care shift workers (BA level)

• Cost analysis complicated by multiple sector costs:

– safety and permanence costs are paid by child welfare

– parent training costs paid by education sector (community college)

– ameliorative costs (mental health, substance use, developmental) paid by other service sectors

Child Welfare Context and Effectiveness Child Welfare Context and Effectiveness Study Design – Organization of ServicesStudy Design – Organization of Services

• Multiple levels: state and county agencies, programs within agency, supervisory units within programs, families within case worker, children within families – adjustment for clustering and nested designs

• Child nested within foster care and within biological family • Service programs organized by type of child setting –

importance of change of setting and it’s meaning– Entry into services– Entry into out-of-home care– Change of placement within out-of-home care– Exit from out-of-home care– Exit from services

Collaborations among Disparate Collaborations among Disparate Research TraditionsResearch Traditions

• Intervention Development and Efficacy Trials – relatively small n studies, “thick” measures with triangulation

• Services Research and Clinical Epidemiology – large N studies, “thin” measures including administrative data

• Prevention Science Methodologists

Developmental modeling and Developmental modeling and efficacy trials leading to efficacy trials leading to

effectiveness and Implementation effectiveness and Implementation trials of an intervention for foster trials of an intervention for foster

parents in the CWSparents in the CWS

Early trials of PMTEarly trials of PMT

• Crafted on basis of a great deal of clinical and systematic observation in homes

• Case studies and small randomized trials.

• Very thick measurement

Replications Across Replications Across Developmental Level and Sites Developmental Level and Sites

• OSLC: first graders through adolescents

• Wahler: children

• Webster-Stratton: preschoolers

• Eyeberg

• Forehand

Thick, Multi-Method Thick, Multi-Method MethodologyMethodology

• Naturalistic and laboratory observations

• Parent, child, and teacher reports

• Administrative data from schools and juvenile courts

• Psychiatric data

• Analogue Tasks

Word Slide of Neglect Model

.48

ChildAggression

D4

.15Supervision-

Tracking

CRSSCongruence

e6

CEESCongruence

e5

.30

PunitiveDiscipline

log AbusiveIndex

e8

CBCL T e13

Age

D3D1

Sex

SocialStatus

Social StatusHEQSSe2

MotherEducatione1

-.02

.12

Care &Environmental

Neglect

NeclectIndexe7

D2

.22*

Achieve-HEQACHe3

.02

r2 r 2

r 2

1.00

r 2

-.11MotherOccupatione4

-.17†

-.05

AnalogPhysicals -Escalation

e9

AngerSTAS - APT

e10

.81***

.44

.33

.28*

*

.70***

.73***

.36***

.03 Inconsistent

Disciplinee11

.40*

ChildScenarios e14

InterviewerRating e15

TRF T e17

.23

.71 .38*

**

.36

log IPCMother to

Childe12

.18*

-.28*

-.55*

.56*

.79

.30**

.27*.36**-.3

3*

.20*

1.29

ChildAggression

WB

D4bCBCL T e13b

r 2

ChildScenarios e14b

InterviewerRating

e15b

TRF T e17b

.75

.27**

.20*.39**

.41.13

.62.68.66**.20*

.47**.08

cmin=5356.00,df=\df,p=\p,cmindf=\cmindf,cfi=\cfi

-.15

.25*

Mediated model for lift, playground etc

r2 .06

AverageSubstance

Use

r2 .13

GrowthSubtance

Use

D1

D2

Prob.SolvingFall 5th

r2 .2

5

Prob.SolvingSpr. 5th D4

parents/child

e8

mother/child

e7

parents/child

e10

mother/child

e9

AssignedIntervention

r2 .09

Accel.Substance

Use

D3-.2

0*

Sex ofChild

InterventionX Sex

.48***

AggressPlayground

Fall 5th

Obs 1

e11

Obs 2

e12

Obs 3

e13

r2 .4

2

AgressPlayground

Spr. 5th

Obs 1

e14

Obs 2

e15

Obs 3

e16

.41

D5

.39

.52***

.76 .73

.71

.44.36 .34.37

.26*-.32*

-.16*-.15*

.23*

.24*

-.16*

.72

.60***

.48***

-.26**

.13*

-.21*

Academic Progress

Inter-vention

Control

1st Grade (p < .01)

14.0 -12.0 -10.0 -

8.0 -6.0 -4.0 -2.0 -0.0 -

2.3

10.5

Inter-vention

Control

5th Grade (n.s.)

14.0 -12.0 -10.0 -

8.0 -6.0 -4.0 -2.0 -0.0 -

12.410.7

CD AP DSM IV Criteria

Inter-vention

Control

1st Grade (p < .05)

14.0 -12.0 -10.0 -

8.0 -6.0 -4.0 -2.0 -0.0 -

5.5

12.6

Inter-vention

Control

5th Grade (n.s.)

14.0 -12.0 -10.0 -

8.0 -6.0 -4.0 -2.0 -0.0 - 3.6 2.5

Foster Parents as Agents of Foster Parents as Agents of Change Change

• Multi-dimensional treatment Foster Care

Multidimensional Treatment Foster Care (MTFC)

For youth placed in out-of-home careYouth are placed singly in intensively trained and

supervised community foster homes that are contacted daily and supported 24/7 for 6–9 mo.

Interventions are implemented using multiple methods (e.g., family and individual therapy, skill training, academic supports) in key settings

Program supervisors carry a caseload of 10, supervise foster parents, therapists, & skills trainers, and work with parole/probation officers

Youth attend public schools

MTFC Effects for BoysMTFC Effects for Boys

More time in program/fewer runaways

Less time in “locked” incarceration in follow-up

Fewer criminal offenses (½ the rate of GC boys)

Less likely to commit violent crimes 2 years later

Delinquency effects mediated by:• Supervision• Relationship with a mentoring adult• Consistent non-harsh discipline • Less association with delinquent peers

Chamberlain & Reid, 1998; Eddy & Chamberlain, 2000; Eddy, Whaley, & Chamberlain, 2004

MTFC Effects for GirlsMTFC Effects for Girls

Delinquency

Deviant Peer Association

School Attendance & Homework Time

Leve, Chamberlain & Reid (in press), J of Counseling and Clinical Psychology; Leve & Chamberlain (2005) J of Abnormal Child Psychology;

Chamberlain & Leve, in preparation

4.5

1.3 1.4

5.2

0.8 0.70

1

2

3

4

5

6

Me

an

nu

mb

er

of

arre

sts

GC MTFC

Intervention group

24-month arrest outcomes

12 months pre-treatment entry12 months post-treatment entry24 months post-treatment entry

2.7 1.5

90

57

72 75

22 200

102030405060708090

100

Mea

n n

um

ber

of

day

s in

lo

cked

se

ttin

gs

GC MTFC

Intervention group

24-month locked settings outcomes

12 months pre-treatment12 months post-treatment24 months post-treatment

129

42

Efficacy to Effectiveness and Efficacy to Effectiveness and ImplementationImplementation

Project Keep

Project KEEP

Cascading Dissemination of a Foster Parent Intervention

A collaboration between the:

• San Diego Health and Human Services Agency, • Child and Adolescent Services Research Center,• Oregon Social Learning Center, and

funded by the National Institute of Mental Health.

The Goals of KEEPThe Goals of KEEP

• To increase the parenting skills of foster and kinship parents

• To decrease the number of placement disruptions • To improve child outcomes• To increase the number of positive placement

changes (e.g. reunification, adoption)• To test the “cascade” question-can the intervention

be implemented with equal effectiveness by a second generation of interventionists?

Project KEEP aims to Project KEEP aims to accomplish these goals by--accomplish these goals by--

• Promoting the idea that foster parents can serve as key agents of change for children.

• Strengthening foster parent’s confidence and skill level so that they can successfully change their own and their child’s behaviors.

• Helping foster parents use effective parent management strategies and provide them with support to do so

• Increasing short and long term positive child outcomes in multiple domains and settings – home, school, with peers.

Support and Training for Foster Parents

Child Demographic InformationChild Demographic Information

Mean

Age Age at Baseline 8.8 (SD=2.2, n=700)Age Range 4-13 (n=700)

Gender Female 52% (n=364)Male 48% (n=336)

Ethnicity Caucasian 29% (n=206)African-American 25% (n=175)Hispanic/Latino/a 33% (n=228)Mixed/Other 13% (n=91)

Foster Parent Demographic InformationFoster Parent Demographic Information

Mean

Language Spoken English only 60% (n=404)

Spanish only 8% (n=52)

Both English and Spanish 32% (n=122)

Household Income Less than 64,999 69% (n=483)

Over 65,000 17% (n=120)

Refused/Don’t Know 14% (n=97)

Foster Parent Demographics - 2Foster Parent Demographics - 2

Mean

Employment Currently Employed

(not including foster parenting)49% (n=343)

# of hours works per week

(includes unemployed foster parents)17.1

(SD=20.8, n=696)

Education Level

High School/GED or less 41% (n=285)

Some College 46% (n=325)

Vocational or Technical Degree 1% (n=48)

Bachelor’s Degree 7% (n=48%)

Graduate Degree 5% (n=32)

Average Number of Children in HomeAverage Number of Children in HomeControl

(n=341)

Treatment

(n=359)

Both

(n=700)

Biological/Step Children .7

(1.1)

.7

(1.2)

.7

(1.2)

Adopted/Foster Children 2.4

(1.9)

2.6

(2.0)

2.5

(2.0)

Other Children .3

(1.1)

.1

(.5)

.2

(.9)

All Children 3.5

(2.0)

3.5

(1.8)

3.5

(1.9)

Rates of child problems: Parent daily Rates of child problems: Parent daily report (PDR)report (PDR)

• PDR is collected by telephone from foster/kin parents

• Each call takes 5-10 minutes• We collect 3 calls at baseline, another 3 calls 4

months later, and a final 3 calls 6 months after that (10 months after baseline)

• PDR produces data on the occurrence of child behavior problems and foster parent stress

Baseline PDR

Fitte

d L

og

Ha

za

rd o

f P

lace

me

nt

Dis

rup

tio

n

0 5 10 15 20

01

23

Greater than 5 problem behaviors per day at baseline predicts placement disruption within the next 6 months

After 5 behaviors, every additional behavior on the PDR increases the probability of disruption by 13 %

Good news: We learned that…Good news: We learned that…

• Foster/kin parents tolerate about as much child problem behaviors as non-system families do –5 behaviors

• PDR data is feasible to collect and is well tolerated by foster and kin parents

• PDR data tells you who to concentrate the intervention on given limited resources : 40/60

PDR – Total # of Problem Behavior PDR – Total # of Problem Behavior (All Children)(All Children)

Baseline Termination

Control Group

5.8

(4.0)

(n=265)

5.4*

(4.1)

(n=265)

Treatment

Group

5.9

(4.3)

(n= 299)

4.4

(3.9)

(n=299)

*P < .05

PDR – Total #of Problem BehaviorsPDR – Total #of Problem Behaviors(TC only)(TC only)

Baseline Termination

Control Group

5.4

(3.7)

(n=229)

5.1*

(3.9)

(n=229)

Treatment

Group

5.7

(4.2)

(n= 241)

4.2

(3.8)

(n=241)

*P < .05

PDR – Total #of Problem BehaviorsPDR – Total #of Problem Behaviors(Substitute Child only)(Substitute Child only)

Mean (SD)

Control Group7.4 (4.9)

(n=36)

Treatment

Group

5.1 (4.2)*

(n=64)

PDR by interventionist cohort control experimental

baseline term baseline term

developers

N=5065.7

(4.0)

5.4

(4.2)

6.0

(4.3)

4.4

(4.0)

cascade N=191

6.0

(3.9)

5.5

(4.1)

5.6

(4.2)

4.3

(3.6)

Intervention Effects on Parenting Intervention Effects on Parenting

Social Learning parenting practices

Control

BL T

Treatment

BL T

Uses rewards (1=daily, 7=never)

3.7 4.0 3.6 3.4*

% use point charts 18 22 26 51*

% who use time out 42 37 36 52*

Doesn’t warn/discuss 50 48 45 41

ConclusionsConclusions

• Preliminary data suggests KEEP group participation increase foster and kin parent skills

• Increased parent skills translate into lower rates of child problem behaviors

• Lower rates of child problem behaviors translate into fewer placement disruptions & more frequent placement transitions to family/relative care

Review of Design and Methods Review of Design and Methods Issues RaisedIssues Raised

• Use of Administrative Data to measure outcomes (CWS vs. Juvenile Court / School Records)

• Measurement of proximal or targeted processes (Parent Reports and PDR)

• Moving from Thick to Thin Assessment• Power when large community units are unit of

randomization• How do we team up efficiently with PSMG