illinois october, 2008 early childhood outcomes: using data for program improvement kathy hebbeler...

IllinoisIllinoisOctober, 2008October, 2008

Early Childhood Outcomes: Early Childhood Outcomes: Using Data for Program Using Data for Program

Improvement Improvement

Kathy HebbelerKathy HebbelerECO at SRI InternationalECO at SRI International

Robin Rooney and Christina KasprzakRobin Rooney and Christina KasprzakECO at Frank Porter Graham InstituteECO at Frank Porter Graham Institute

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2

Objectives Objectives

Understand the purposes of the child outcomes data collection

Be familiar with key considerations related to accurately completing the Child Outcomes Summary Form

Be able to evaluate child outcomes data for accuracy as part of supervision

Understand the use of child outcomes data for program improvement, including for instruction, resource allocation, and professional development


Why are we doing this?


Keeping our eye on the prize:

High quality services for children and

families that will lead to good outcomes.


Goal of early childhood Goal of early childhood special educationspecial education

“…To enable young children to be active and

successful participants during the early

childhood years and in the future in a variety

of settings – in their homes with their

families, in child care, in preschool or school

programs, and in the community.”

(from Early Childhood Outcomes Center,http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf)


High Quality Data on OutcomesHigh Quality Data on Outcomes

Data are a piece of a system that helps to achieve overarching goals for children and families

Data yield Evidence that allows you to make an

inference that should lead to specific actions to improve the system.

Prof’l Development•Preservice•Inservice

System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


The Vision: Using Data as a Tool for The Vision: Using Data as a Tool for Program ImprovementProgram Improvement

Illinois will have quality data available on an ongoing basis about multiple components of the system Outcomes for children and families Programs and services provided Personnel (types, qualifications, etc.) Etc.


Driving Force for Data on Child Goals Driving Force for Data on Child Goals Comes from the Federal LevelComes from the Federal Level

Government Performance and Government Performance and Results Act (GPRA)Results Act (GPRA)

Program Assessment Rating Tool Program Assessment Rating Tool (PART)(PART)

Individuals with Disabilities Individuals with Disabilities Education Act (IDEA)Education Act (IDEA)


Requires goals and indicators be

established for IDEA Indicators and data collection further

along for school age population than

for EC Previously, for early childhood data

had been collected on:

• Number of children served

• Settings

Government Performance and Results Government Performance and Results Act (GPRA) passed in 1993Act (GPRA) passed in 1993


130 programs examined in 2002; 50%

programs had no performance data

Programs looking at inputs, not results

Part C and Section 619

No long-term child outcome goals or data

Need to develop a strategy to collect

annual performance data in a timely

manner

PART evaluation results (2002)PART evaluation results (2002)


Federal Funding for Preschool Federal Funding for Preschool Special EducationSpecial Education

Total U.S. Illinois

2004 $387,699,000 $17,943,956

2005 $381,385,691 $17,812,936

2006 $380,751,030 $17,650,453


SEC. 616. <<NOTE: 20 USC 1416.>> MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.-…..…..``(2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- ``(A) improving educational results and functional outcomes for all children with disabilities;

Individuals with Disabilities Education ActIndividuals with Disabilities Education Act


How Office of Special Education How Office of Special Education (OSEP) responded(OSEP) responded

Required states to submit outcome data in their Annual Performance Report (APR)

Funded the Early Childhood Outcomes (ECO) Center to do research, make recommendations, and assist states


Where are we now:

Federal reporting requirements


OSEP Reporting Requirements: OSEP Reporting Requirements: the Outcomes the Outcomes

Positive social emotional skills (including positive social relationships)

Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])

Use of appropriate behaviors to meet their needs


OSEP Reporting CategoriesOSEP Reporting Categories

Percentage of children who:

a. Did not improve functioning

b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

3 outcomes x 5 “measures” = 15 numbers


Reporting ScheduleReporting Schedule

Due February 2008 Data in reporting

categories at exit for all children who have been in the program for at least 6 months

Must be reported for the year beginning July 1, 2006

Repeat with next year’s data in 2009, etc.

2010 States must set targets Summary statements

for targets recommended; not yet official*

States must report data to public by school districts for these target numbers

*See http://www.fpg.unc.edu/~ECO/meetings.cfm for more information on recommended targets

http://www.fpg.unc.edu/~ECO/meetings.cfm


Where are we now:

State decisions and activities


To respond to federal

reporting requirements

To have data for program

improvement and

to respond to federal reporting

requirements

Purpose

WHY? (State Version)


To provide data to the

state

To have data for program

improvement and

to provide data to the state

Purpose

WHY? (Local Version)


State approaches State approaches Most states have embraced outcomes

measurement and are collecting outcomes data for their own purposes.

Many states are building bigger systems than needed to produce the federal data.

Go to www.the-eco-center.org for more information about what other states are doing


How are states collecting child How are states collecting child outcomes data? outcomes data?

Possible state approaches to collection of child data Child Outcomes Summary Form (COSF) Publisher’s online assessment system Single assessment statewide Other approaches

ApproachApproach Part C Part C

(56 states)(56 states)

PreschoolPreschool

(59 states)(59 states)

One tool One tool statewidestatewide 8/56 (14%)8/56 (14%) 13/59 (22%)13/59 (22%)

Multiple Multiple Publishers’ online Publishers’ online toolstools

2/56 (4%)2/56 (4%) 3/59 (5%)3/59 (5%)

COSF 7 pt. scaleCOSF 7 pt. scale 40/56 (71%)40/56 (71%) 36/59 (61%)36/59 (61%)

Other Other 6/56 (11%)6/56 (11%) 7/59 (12%) 7/59 (12%)

State Approaches to Measuring State Approaches to Measuring Child OutcomesChild Outcomes


Checking in – How is it going?Checking in – How is it going?

Implementing the Child Outcomes Implementing the Child Outcomes Summary Form ProcessSummary Form Process

What is going well? What is not going well? Are there other issues or concerns that

need to be addressed about the process?


Let’s Review:Let’s Review:

Child Outcome Child Outcome Summary FormSummary Form


Learning from Each OtherLearning from Each Other

1. Did you attend a training on the COSF?

2. Have you participated in a COSF team rating process?

For more than 10 children? 20? 30?

3. Have you reviewed COSF forms completed by others?

4. Have you provided training on the COSF?

5. How confident are you in your knowledge of the COSF process?


Essential Knowledge for Completing Essential Knowledge for Completing the Child Outcomes Summary Form the Child Outcomes Summary Form

Between them, team members must:

1. Know about the child’s functioning across settings and situations

2. Understand age-expected child development

3. Understand the content of the three child outcomes

4. Know how to use the rating scale

5. Understand age expectations for child functioning within the child’s culture


Important pointImportant point

It is not necessary that all team members be knowledgeable in all 5 areas

Especially, no expectation that parents understand the rating scale or typical child development

But the professionals have to!


1. Know about the child’s functioning 1. Know about the child’s functioning across settings and situationsacross settings and situations

How we learn about the child’s functioning across settings and situations:

Good Good assessmentassessment


DEC* recommended practices DEC* recommended practices for assessment for assessment

Involve multiple sources Examples: family members, professional team

members, service providers, caregivers

Involve multiple measures Examples: observations, criterion- or curriculum-

based instruments, interviews, norm-referenced scales, informed clinical opinion, work samples

*Division for Early Childhood


Assessment practices appropriate for Assessment practices appropriate for outcomes measurement: ASHA*outcomes measurement: ASHA*

ASHA recommended practices: Gather information from families, teachers,

other service providers Collect child-centered, contextualized,

descriptive, functional information

(*American Speech-Language-Hearing Association)


Assessment instrumentsAssessment instruments Assessment the tool

vs. assessment the process

Assessment tools can inform us about children’s functioning in each of the three outcome areas

Challenge:

There is no assessment tool that assesses the three outcomes directly


Resources for understanding age-Resources for understanding age-expected child developmentexpected child development

ECO linkhttp://www.fpg.unc.edu/~eco/pdfs/Age-expected_child_dev_9-5-07.pdf

(under “ECO Tools”)

New course coming soon Watch ECO web site

www.the-eco-center.orgwww.the-eco-center.org


Outcomes JeopardyOutcomes Jeopardy

Pointing to the cabinet for

cereal

Reading the letter “S” on the

Stop sign

Washes hands before lunch

BitingPlays by himself in the classroom

Plays with rhyming words

Building a castle from blocks with

a friend

Problems sleeping

Sharing a cookie at lunchtime

$100

$200

$100

$300

$200

$300

$200

$100

$300


Children have positive social Children have positive social relationshipsrelationships

Involves: Relating with adults Relating with other children For older children, following rules related to groups

or interacting with othersIncludes areas like:

Attachment/separation/autonomy Expressing emotions and feelings Learning rules and expectations Social interactions and play


Children acquire and use knowledge Children acquire and use knowledge and skills and skills Involves

Thinking Reasoning Remembering Problem solving Using symbols and language Understanding physical and social worlds

Includes: Early concepts—symbols, pictures, numbers Imitation Object permanence Expressive language and communication Early literacy


Children take appropriate action to meet Children take appropriate action to meet their needstheir needs

Involves:

Taking care of basic needs Getting from place to place Using tools (e.g., fork, toothbrush, crayon) In older children, contributing to their own health

and safety Includes:

Integrating motor skills to complete tasks Self-help skills (e.g., dressing, feeding, grooming,

toileting, household responsibility) Acting on the world to get what one wants


The two COSF questionsThe two COSF questions

a. To what extent does this child show age-appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7)

b. Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No)


7 – Completely7 – Completely

The child shows behaviors and skills expected in all or almost all everyday situations that are part of the child’s life Home, store, park, child care, with

strangers, etc. The child’s functioning is considered

appropriate for his/her age No one has significant concerns about the

child’s functioning in this outcome area


6 – Between completely 6 – Between completely and somewhatand somewhat

The child’s functioning generally is considered appropriate for his or her age

but there are some significant concerns about the child’s functioning

in this outcome area


5 – Somewhat5 – Somewhat

The child shows functioning expected for his/her age some of the time and/or in some situations

The child’s functioning is a mix of age-appropriate and not appropriate functioning

The child’s functioning might be described as like that of a slightly younger child


4 – Between a 5 and a 34 – Between a 5 and a 3

Child shows some age appropriate functioning some of the time or in some situations or settings but most of the child’s functioning would be described as not yet age appropriate.

The child’s functioning might be described as like that of a younger child


3 – Emerging 3 – Emerging

The child does not yet show functioning expected of a child his/her age in any situation

The child’s behaviors and skills include immediate foundational skills on which to build age-appropriate functioning

The child’s functioning might be described as like that of a younger child


2 – Between 3 and 12 – Between 3 and 1 The child does not yet show functioning

expected of a child his/her age in any situation

The child’s behaviors and skills does have some the immediate foundational skills on which to build age-appropriate functioning but these are not displayed very often

The child’s functioning might be described as like that of a younger or even much younger child


1 – Not yet1 – Not yet The child does not yet show functioning

expected of a child his/her age in any situation

The child’s skills and behaviors also do not yet include any immediate foundational skills on which to build age-appropriate functioning

The child’s functioning might be described as like that of a much younger child

Children with 1 ratings still have skills, just not yet at an immediate foundational level


Rating Scale JeopardyRating Scale Jeopardy

Age appropriate functioning – no

concerns

Mix of age appropriate and not

age appropriate functioning

No age appropriate functioning – not yet showing immediate foundational skills

Some age appropriate

functioning but very little

No age appropriate functioning – lots of

immediate foundational skills

Age appropriate functioning – some

concerns

Rarely shows age appropriate functioning

No age appropriate functioning – some

immediate foundational skills

Age appropriate functioning

$100

$200

$100

$300

$200

$300

$200

$100

$300


Essential Knowledge for Completing Essential Knowledge for Completing the Child Goals Outcome Form the Child Goals Outcome Form








Point of clarificationPoint of clarification

Process is NOT about comparing groups of children – it IS about asking how close children are to being able to do what is expected at their age

Early learning guidelines Kindergarten and access to the

general curriculum


What we are learning nationallyWhat we are learning nationally

The process of training for child outcomes data collection has uncovered other areas of significant need related to professional development.


Providers need to know more about:Providers need to know more about:

Assessment How to gather assessment data to reflect

functioning across settings and situations, especially how to gather child functioning information from families

Understanding the results of the assessment Sharing assessment results sensitively and

honestly with families


Providers need to know more about:Providers need to know more about:

Functional outcomes What are they? How do they differ from outcomes organized

around domains? What do they mean for how professionals from

different disciplines operate as a team? Typical child development

What are the functional expectations for children at different ages with regard to each of the 3 goal statements?


Let’s meet Ava

illinois october, 2008 early childhood outcomes: using data for program improvement kathy hebbeler...

Documents

early childhood data

system outcomes

functional outcomes

outcomes x

early childhood years

use of child outcomes

families good outcomes

child outcomes data