+ experimental evidence on the early effectiveness of intervention in childhood orla doyle (ucd...
TRANSCRIPT
+
Experimental Evidence on the Early Effectiveness of Intervention in ChildhoodOrla Doyle (UCD School of Economics & UCD Geary Institute)
Conference on Irish Economic Policy1st February 2013
+Importance of Early Childhood Investment
Targeted early intervention programmes effective way of reducing socio-economic inequalities in children’s skills
Well-designed interventions can generate a return to society ranging from $1.80 to $17.07 for each dollar spent
YET primarily US-based evidence only on the effectiveness & cost-effectiveness of such interventions
Argument for early intervention in more generous welfare systems is not rooted in strong empirical evidence
+Irish Context
Prevention & Early Intervention Programme (The Atlantic Philanthropies) & Government , DCYA) initiated in 2004
Aim: to support the development, implementation and evaluation of strategies for disadvantaged children
All programmes evaluated using experimental or quasi-experimental design
Little tradition of social experimental evaluation in Ireland
Comparing the effectiveness of multiple programmes will enhanced resource allocation and policy design
+ Preparing for Life Programme
Community-led initiative: operated by Northside Partnership in highly disadvantaged area of Dublin, Ireland
Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives
Evidence of need: Children scored below the norm on cognitive & non-cognitive skills on school entry (Doyle & McNamara, 2011)
Aim: Improve levels of school readiness by assisting parents in developing skills to prepare their children for school
Evaluation Design: RCT & embedded quasi-experimental design
+Design of Preparing for Life
+PFL Evaluation
Impact Evaluation Aim: To determine if the programme/service is effective
PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths, 24mths, 3yrs, 4yrs, school entry
Implementation Evaluation Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity
to the model
PFL data collection:1. Implementation data2. Focus groups with participants3. Semi-structured interviews with programme staff
+ Recruitment
Eligibility Criteria: Cohort of pregnant women residing in PFL catchment area between Jan
2008-August 2010 (32 months) Includes nulliparous and multiparous women
Recruitment: Maternity hospital at first booking visit (b/w 12-26 weeks) Within the local community
Population-based recruitment rate, based on all live births during the recruitment phase, was 52%
+Randomisation Procedure
Individual-level randomisation Unconditional probability randomisation strategy
Design procedure that preserves the integrity of randomization ie. no opportunity for recruiter to intentionally influence assignment
Randomisation process After informed consent obtained, participant pressed key on tablet laptop to
allocate her assignment condition
Evaluation & Implementation team automatically received an email with the assigned ID number and treatment condition
Thus no opportunity to compromise the randomisation process
+ Baseline Analysis
Determine the effectiveness of the randomisation procedure
Baseline assessment conducted post randomisation, pre intervention
Randomisation worked!
PFL Baseline Assessment Proportion of Measures Not Significantly Different at Baseline
Topic PFL Low – PFL High PFL – Comparison group
Family Socio-demographics 33/33 (0%) 27/33 (82%)
Maternal Well-being 24/24 (0%) 18/24 (75%)
Maternal Health & Pregnancy 35/35 (0%) 26/35 (74%)
Parenting & Childcare 10/13 (74%) 6/13 (46%)
Social Support 17/18 (94%) 9/9 (0%)
Total NOT Statistically Different 119/123 (97%) 86/114 (75%)
+ Baseline Differences in Selected Socio-Demographics
Low Treatment – High Treatment
InstrumentN
(nLOW/ nHIGH)
MLOW
(SD)MHIGH
(SD)p Effect Size
(d)
Mother’s Age 205(101/104)
25.30(5.99)
25.46(5.85)
ns .03
First-time Mothers 205(101/104)
0.50(0.50)
0.54(0.50)
ns .09
Mother Married 205(101/104)
0.18(0.38)
0.14(0.35)
ns .09
Mothers with Junior Certificate Qualification or Lower
205(101/104)
0.40(0.49)
0.34(0.47)
ns .12
Mothers with Primary Degree 205(101/104)
0.03(0.17)
0.03(0.17)
ns .01
Mothers Unemployed 205(101/104)
0.41(0.49)
0.43(0.50)
ns .05
Fathers Unemployed 198(97/101)
0.31(0.46)
0.43(0.50)
ns .24
Residing in Social Housing 204(101/103)
0.55(0.50)
0.55(0.50)
ns .00
In Possession of a Medical Card 205(101/104)
0.66(0.47)
0.60(0.49)
ns .14
+Methodology
Estimate treatment effects at birth, 6 months, 12 months
Permutation based hypothesis testing (Heckman et al. 2010) Classical hypothesis tests unreliable when the sample size is small and the
data are not normally distributed Permutation test are distribution free, thus suitable in small samples Based on the assumption of exchangeability between treatment
conditions under the null hypothesis
Stepdown procedure (Romano & Wolf, 2005) Ignoring the multiplicity of tests may lead to the rejection of “too many”
null hypotheses Test multiple hypotheses simultaneously by controlling overall error rates
for vectors of hypotheses using the family-wise error rate (FWER) as a criterion
Less conservative & more powerful than other methods as takes account of statistical dependencies between tests
+Analysis of Maternity Records
Maternity records N(nHIGH/ nLOW)
MHIGH
(SD)MLOW
(SD)Individual Test
p
Infant Outcomes Apgar 5 180
(95/85)9.79(0.48)
9.69(0.63)
ns
Birthweight (grams) 181(95/86)
3281(664)
3294(614)
ns
Low BW (<2500gr) 181(95/86)
0.10(0.30)
0.05(0.23)
ns
High BW (>4000gr) 181(95/86)
0.09(0.29)
0.12(0.33)
ns
Labour Outcomes
Weeks at gestation 174(92/82)
39.60(2.51)
39.56(2.01)
ns
Prematurity (<37 wks) 174(92/82)
0.07(0.26)
0.06(0.24)
ns
Labour onset (Spontaneous=0, Induced=1)
178(93/85)
0.31(0.46)
0.43(0.49)
p=0.056
Caesarean section 178(93/85)
0.15(0.35)
0.25(0.44)
P=0.036
Data Collection Summer 2012 accessed maternity records for PFL participants who attended
Rotunda Hospital (85%) & NMH (15%) 283 parents gave consent to access records - available records for 272
+PFL Maternity Records – Summary
No programme impact on birth weight or gestational age, but programme altered the birth experiences of the participating families
Significantly fewer mothers in the high treatment group had a c-section compared to those in the low treatment group
Result remains when the incidence of a previous c-section and pariouness are controlled for
Programme particularly effective in preventing high treatment mothers from having a repeat c-section
Result possibly linked to Tip Sheets which specifically focus on preparing the mother for the birth and being aware of the birthing process and possible labour outcomes
+ Analysis of 6 Month Results
PFL 6 Month Results Proportion of Measures Significantly Different at 6 Months
PFL Low – PFL High Individual Tests
Multiple Hypothesis Tests
Child Development 0% (0/13) 0%
Child Health 10% (3/30) 0%Parenting 23% (5/22) 20%Home Environment & Safety 36% (8/22) 50%
Maternal Health 5% (1/20) 25%
Social Support 38% (5/13) 0%
Childcare & Service Use 7% (1/14) 0%
Household Factors & SES 0% (0/26) 0%
Total Statistically Different 14% (23/160) 12% (3/25)
Test for treatment effects across 160 measures & 25 Step-down categories
Sample: PFL High/Low treatment groups: 84/90
+
ASQ Scores & Difficult Temperament
N(nHIGH/ nLOW)
MHIGH
(SD)MLOW
(SD)Individual
Test p1
Step Down Testp2
Effect Sized
ASQ Gross Motor Score 173(83/90)
40.78(11.93)
38.50(12.99)
ns ns 0.18
ASQ Communication Score 173(83/90)
53.07(7.84)
51.78(8.49)
ns ns 0.16
*Difficult Temperament 173(83/90)
11.70(5.71)
12.21(5.50)
ns ns 0.09
ASQ Personal Social Score 172(82/90)
46.52(12.09)
45.94(13.57)
ns ns 0.05
*ASQ Social-Emotional 173(83/90)
14.76(10.68)
15.17(13.75)
ns ns 0.03
ASQ Fine Motor Score 173(83/90)
50.84(9.46)
51.39(10.17)
ns ns 0.06
ASQ Problem Solving 173(83/90)
51.87(9.39)
52.56(9.92)
ns ns 0.07
Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.
EXAMPLE OF RESULTS: CHILD DEVELOPMENT @ 6 MONTHS
+Positive Treatment Effects @ 6 Months
Child Development: none
Child Health: Appropriateness & frequency of eating; Immunizations
Parenting: Parent-child dysfunctional interactions; Baby comparison score; Parental hostile-reactive behaviour; Interaction with baby
Home Environment: HOME sub-scales – variety, childcare, toys & books, physical environment, learning materials; Electrical socket covers, Framingham safety survey
Mother’s Health: Mother less hospitalisation for complications after birth
Social support: Frequently sees grandparent; Discuss programme with partner & others; Knows children same age as baby; Meet friends frequently
Childcare & Service use: Voted in last election
Household & SES: none
+Analysis of 12 Month Results Test for treatment effects across 140 measures & 23 Step-down
categories
Sample: 12 Months: PFL High/Low treatment groups: 83/82
+Positive Treatment Effects @ 12 Months
Child Development: ASQ Fine Motor skills; BITSEA Social Competence cut-off
Child Health: Chest Infections; Immunizations; Appropriateness eating
Parenting: none
Home Environment: none
Mother’s Health: Drank alcohol
Social support: Meet friends frequently; Voted in last elections
Childcare & Service use: ~
Household & SES: none
Results consistent with the home visiting literature
+Attrition & Dosage
Attrition/disengagement Attrition may bias evaluation results if non-random
29% of high treatment & 29% of low treatment dropped out/disengaged between randomisation & 12 months
Few individual characteristics predict attrition – IQ more important
Dosage/engagement Variation in dosage, may lead of variation in effect
Collected data on number/duration of home visits
On average, high treatment group received 21 home visits of ~1 hr in duration between pregnancy & 12 months 58% of prescribed visits delivered based on bi-weekly visits
Higher maternal IQ associated with higher engagement
+ Conclusions to date
Many of the results are in hypothesized direction
Few significant effects, yet consistent with literature Significant findings on birthing experiences, parenting, home environment, child
health, social support Corresponded directly to information on the PFL Tip Sheets No significant impact on many key domains such as child development, birth weight,
breastfeeding, maternal well-being etc.
Attrition & disengagement between 6 & 12 months very low
Engagement below prescribed dosage
Participate satisfaction is relatively high (focus groups)
+ Update on PFL
Oldest PFL child is 4 years and old & youngest is 18 months
24, 36 & 48 month surveys are currently in the field
18 month results available very soon
Over 1,600 interviews conducted as part of impact evaluation
Conducted five school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011, 2012
Evaluation will continue until all children start school
Programme website: www.preparingforlife.com
Evaluation website: http://geary.ucd.ie/preparingforlife/