the results are in: using early childhood outcome data

The Results are In: Using Early Childhood Outcome Data

Kathy HebbelerEarly Childhood Outcomes Center at SRI International

August, 2011

What we will cover

• Quick review of the reporting requirement and state approaches

• Share the national data• Describe how the national data were computed• Discuss the quality of the national data• Discuss the meaning of the numbers• Framework and self-assessment (maybe…)

2Early Childhood Outcomes Center


OSEP Reporting Requirements: Child Outcomes

• Positive social emotional skills (including positive social relationships)

• Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])

• Use of appropriate behaviors to meet their needs


OSEP Reporting Categories

Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to move nearer

to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged

peers but did not reach itd. Improved functioning to reach a level comparable to

same-aged peerse. Maintained functioning at a level comparable to same-

aged peers3 outcomes x 5 “measures” = 15 numbers

Illustration of 5 Possible Paths

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peersAchieved functioning comparable to age peersMoved nearer functioning comparable to age peersMade progress; no change in trajectoryDid not make progress

The Summary Statements

1. Of those children who entered the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned 3 years of age or exited the program.

2. The percent of children who were functioning within age expectations in each outcome by the time they turned 3 years of age or exited the program.


State Approaches to Outcomes Data

Approach Part C (56

states/jurisdictions)

Preschool(59

states/jurisdictions)

COS 7 pt. scale 41/56 (73%) 37/59 (63%)

One tool statewide 7/56 (13%) 9/59 (15%)

Publishers’ online analysis

3/56 (5%) 6/59 (10%)

Other 5/56 (9%) 7/59 (12%)

Early Childhood Outcomes Center 8

Note: Based on 29 States with highest quality data

Social relationships Knowledge and skills Action to meet needs0%

5%

10%

15%

20%

25%

30%

35%

40%

Estimated Data for Part C, 2009-10




10%

20%

30%

40%

50%

60%

70%

80%

Estimated Summary Statement Data for Part C, 2009-10



5%

10%

15%

20%

25%

30%

35%

40%

Estimated National Data for Early Childhood Special Education, 2009-2010





20%

40%

60%

80%

100%

Estimated National Summary Statements for Early Childhood Special Education, 2009-2010

Criteria for States with Quality Data

1. Low percentage of missing data

2. No odd patterns in “a” or “e” categories

3. Did not use questionable data collection methods


Calculating Missing Data for Part C

Proxy for missing data =Number with data for C3/

Exiting Data (618)


• Do not expect this number to be 100%

• ..but we don’t expect it to be 10% either

Part C: Percent of Exiters included in Outcomes Data

08-09<10% = 10*10- 20% = 420- 30% = 830- 40% = 1140- 50% = 850- 60% = 860- 70% = 470- 80% = 2

>80% = 1

09-10<10% = 5*

10- 20% = 420- 30% = 630- 40% = 840- 50% = 550- 60% = 1160- 70% = 970- 80% = 1

>80% = 0

*3 states are sampling for Part C. Excluded states with <28%.

Calculating Missing Data for 619

Proxy for missing data =Number with data for B7/

Child count

15

• Do not expect this number to be 100%

• ..but we don’t expect it to be 10% either

Percent of Child Count included in Outcomes Data for ECSE

08-09<10= 11*

10- 20%= 1520- 30%= 1230- 40%= 1240-50% =1>50% = 2

09-10<10= 6*

10- 20%= 1120- 30%= 1230- 40%= 1640-50% =4>50%= 0

*4 States are sampling for 619Excluded states with <12% of child count

Problem with Missing Data

• We don’t know how well the data the state has represent the entire state.

• If the data are representative, the percentages for the a to e Progress Categories and the Summary Statements won’t change as data are added on more children.



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 490%

2%

4%

6%

8%

10%

12%

14%

16%

Percent Reported in "a“ for Knowledge and Skills for ECSE by State

Exclusion criteria: a<10% in any outcomes e<65% in any outcomes

Can we trust these data?


Pattern checking for validity

• Checking across years–How do the 2009-10 compare to the

data for 2008-09?• Checking across methods

–How do the data for all states compare to states with highest quality data?


All states 19 best All states 29 bestFFY 08-09 FFY 09-10

0

10

20

30

40

50

60

70

80

64.570.2

6471

SS1: % who IncreasedGrowth Rates

Part C, Outcome A: Social Relationships


0

10

20

30

40

50

60

70

80

6761.3

64.7 61.8

SS2: % who Exited at Age Expectations

Part C, Outcome B: Knowledge and Skills


0

10

20

30

40

50

60

70

8070.4

76.6

68.174.2

SS1: % who Increased Growth Rates


0

10

20

30

40

50

60

70

80

5953.9 55.8 53.8


Part C, Outcome C: Meets Needs


0

10

20

30

40

50

60

70

8069.7

76

68.1

75.6



0

10

20

30

40

50

60

70

80

64.660.5 61.5 59.7


Part B Preschool: Social Relationships


0

10

20

30

40

50

60

70

80

90

77.282.7 79.7 82.6



0

10

20

30

40

50

60

70

80

90

61.7 58.762.1

58.7


Part B Preschool: Knowledge and Skills


0

10

20

30

40

50

60

70

80

90

76.282.7

78.381.5



0

10

20

30

40

50

60

70

80

90

55.851.2

55.5 52.3


Part B Preschool: Meets Needs


0

10

20

30

40

50

60

70

80

90

75.381.6

78.181.9



0

10

20

30

40

50

60

70

80

90

67.8 67.2 66.7 66.7


Part C

What to these data tell us?

• Nationally, a high proportion of children who receive Part C and ECSE services are showing greater than expected progress

• Nationally, many (over half) are exiting the program functioning like same age peers in at least one of the outcomes.


Should your state’s data look like the national data?

• Probably not• More important that each state continue to

focus on the quality of its own data– Getting outcomes data on all children who exit– Working with programs whose data look

unusual to address possible data quality issues


Supporting States in Building a Child Outcomes Measurement System

Framework and Self-Assessment

• FRAMEWORK– Set of components and quality indicators– Provides the structure for the self-assessment

• SELF-ASSESSMENT – Scale that provides criteria for levels of

implementation within each quality indicator– Rating assigned based on level of

implementation within each indicator 32Early Childhood Outcomes Center

Purpose of the Self Assessment

• Assist states in setting priorities for improving their COMS

• Provide information to assist states in advocating for resources for systems development

33

• Provide guidance to states on what constitutes a high quality child outcomes measurement system.

Early Childhood Outcomes Center

Additional information

For information on improving data quality and using data for program improvement

www.the-eco-center.org


Measuring and Improving Child and Family Outcomes ConferenceSeptember 18-21, 2011

New Orleans, LA

http://www.the-eco-center.org/

the results are in: using early childhood outcome data

Documents

outcomes dataapproach

outcomes data08

improved functioning

highest quality datacriteria

c3exiting data

maintained functioning

early language communication

level comparable