Opportunities and Challenges for ECEC Impact

Studies in Longitudinal Studies

ICEC Conference on Longitudinal Studies Berlin,

Germany

November 17, 2014

Steve Barnett, PhD

Guidelines for Impact Evaluation

• Specify the purpose: what do we want to learn?

• Measure what matters

• Obtain a solid baseline: where did we start?

– With programs

– With children and families

• Create longitudinal data systems to measure change

• Engage program & policy stakeholders in developing

the evaluation as they are the key to its use

4 Layers of measurement in evaluation

1º Information on Policies • Access/Enrollments

• Resources

• Standards

• Ratios

• Qualifications

• Services

• Eligibility, etc

2º Information on

Implementation

• Structural Quality: ratios, qualifications,

experience, class size, length of day,

services and referrals, etc.

• Process Quality: use of time, teacher-child

interactions, children’s experiences,

transitions, etc.

3º Information on Children’s

Development & Wellbeing

and Family

• Cognition (general, subject-matter related,

executive functions), language, socio-emotional

and physical development, approaches to

learning

• School success and progress, behavior in and

out of school, employment and earnings, health

• Family background, home and community

experiences

4º

Information

on Progress/

Impact

Evaluation

Why are all these important?

• Policies describe intentions, resources, and goals

• Implementation is the link between policies and outcomes

• Learning, development and well being measures are all important outcomes

• Outcomes depend on family, home and later schooling as well ECEC policy and program

• Analysis is required to go from measures to outcomes

• All 4 together provide:

– A stronger basis for causal conclusions about impacts

– A means to improve policy and implementation in ways that will increase impacts

What Should be Assessed?

• For policies--what varies? • For programs--what is likely to influence outcomes and cost?

• What matters to children, parents policymakers, & the public? – Important for quality of life

– Influence school and social success

– Have lasting social and economic consequences

• Outcomes likely to significantly affected by program & policy

• Social and emotional development – Prosocial (caring, conflict resolution, cooperation), anti-social

• Executive functions, self-efficacy, persistence, motivation

• Language, literacy and math

• Approaches to learning – Dispositions (likes & interests), curiosity, creativity, problem solving

• Well being – Engagement and rights

How Should Young Children be Assessed?

• Valid and reliable measures

• Comparably over time and across the population – May need “overlapping” assessments

• At reasonable cost--financial, time, and stress

– Minimize burden to children, teachers, parents

• Multiple options available – Performance or authentic assessment

– Standardized tests (and games)

– Rating scales and checklists by teachers, other professionals, & parents

– More intensive qualitative approaches

What does previous research tell us

about child outcomes to measure in

impact evaluation? • Examples include: Perry, Abecedarian, Chicago

Longitudinal Study, birth cohort studies in many

countries, EPPE, Mauritius, Jamaican Early

Stimulation & Nutrition

Analytic Framework

Early

Childhood

Education

Programs

Increase child’s motivation and positive attitudes for

learning

Increase child’s “academic” abilities

Improve executive functions

(planning, attention, short term

memory)

Family: supportive home

environments both cognitive

and social emotional (less

abuse and neglect, less

exposure to violence outside

home)

Cognitive, social,

emotional, exec.

functions, health

Cognitive:

Increased

knowledge, skills,

and achievement Social: Improved

behavior, peer

relations, teacher

relationships

Health: better mental

and physical health,

increased attendance

and engagement

Higher educational

attainment, less special

education and delayed

school progress

Increased

income Decreased risky

behavior, crime,

violence Intervention

Immediate

Outcomes

Long-term

Outcomes

Better adult

health, lower

care costs

Improve child’s social and emotional development

Increase child’s health

Chicago CPC: Long-term

Academic and Social Impacts

Temple, J. A., & Reynolds, A. J. (2007). Benefits and costs of investments in preschool education: Evidence from the Child-Parent Centers and related programs.

Economics of Education Review, 26(1), 126-144

Child Maltreatment

9.2

14.3

1.73.6

4.7

8.8

0

2

4

6

8

10

12

14

16

18

20

22

24

Perc

enta

ge

Child

abuse/neglect

2 or more CAN Out of home

placement

Preschool group

Comparison group

Adult Arrest and Incarceration

35.8

40

20.3

24.7

20.6

25.6

0

5

10

15

20

25

30

35

40

45

50

Perc

enta

ge

Arrest Guilty Charge Incarceration/Jail

Preschool group

Comparison group

Plan and

Analyze

Introduce

profesional

development

& technical

assistance

Measure and

Evaluate

Progress

Evaluation for Continuous Improvement:

New Jersey Example Develop

Standards

3.9

19.9

34.6

27.7

12.1

1.70.0 0.24.2

32.2

47.4

16.0

0

10

20

30

40

50

60

1.00-1.99 2.00-2.99 3.00-3.99 4.00-4.99 5.00-5.99 6.00-7.00

Pe

rce

nta

ge

of C

lassro

om

s

ECERS-R Score (1=minimal, 3=poor 5= good 7=excellent)

00 Total (N = 232) 08 Total (N = 407)

NJ Raised ECCE Quality Over 8 Years

NJ Effects on Achievement for 1 and 2

Years ECCE

.12

.18 .17

.14

.17

.26

.22

.37

.29

.37

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

0,40

LAL 4th LAL 5th Math 4th Math 5th Science 4 th

1 year Abbott pre-k 2 year Abbott pre-k

12% 12%

19%

17%

0,00

0,02

0,04

0,06

0,08

0,10

0,12

0,14

0,16

0,18

0,20

Retention Special edcuation

Abbott pre-K no Abbott pre-K

NJ ECCE Effects on Retention and Special

Education

Opportunities Measures of policy, programs and children over time are needed to capture

change and validly evaluate impacts

Some outcomes that matter most to children, parents, policy makers are easily measured (including long-term that “matter” more than short-term)

Many options are available to measure learning & development

Measures are available for implementation—to explain, increase confidence in results, and provide a basis for continuous improvement

Measures of family, home, and school also feasible and common

Numerous successful examples of longitudinal studies informing policy

Randomized trials and other manipulations of policy/practice can be embedded to make use of data from longitudinal studies

Comparisons across states and countries can inform policy

Evaluation better for continuous improvement than summative judgments

Challenges Tests perform well over time and across children but cover limited

domains and are sensitive to what is taught when

Language is particularly complex to assess because of multiple languages

Broadest measures of learning and development are ratings that have weaker technical adequacy than tests

Measuring all aspects of LDWB that are of high interest is expensive

Data requirements for valid policy/practice evaluations are stringent

Strengths/weaknesses suggest diverse portfolio including “real life” measures which requires multidisciplinary teams

Measuring implementation costly—teacher self-report replace observation?

Large sample sizes and collection of data from home and school expensive

Engaging policy makers in evaluation process is difficult because they face so many urgent demands on their time

State and international comparisons can be highly political