early childhood outcomes center1 using data for program improvement christina kasprzak, nectac/eco...

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 1

Using Data for Program

Improvement

Christina Kasprzak, NECTAC/ECO

Ann Bailey, NCRRC

July 2010

Measuring Child and Family Outcomes


Systems Thinking:

Systems are made up of interrelated, interconnected components

Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders

Desired changes in one part or level of the system must be accompanied by changes in other parts or levels



SPP as long term plan for systems change:

EI/ECSE are systems of complex, interrelated components with goal of achieving outcomes for children and families

Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results



Developing improvement activities:

Effective Policies, Procedures, and MOU Data Systems and Monitoring TA and Professional Development Fiscal Management Evaluation



Evaluating SPP/APR Improvement Activities


6

The Need for Evaluation

Provides an organized way of assessing work in progress and results obtained

Assesses the impact of an activity on the area targeted for improvement

Identifies strengths and weaknesses in the implementation of the improvement activity


7

Types of Evaluation

Process evaluation

Evaluates the improvement process itself

Impact evaluation

Evaluates the results produced by the process


8

Process Evaluation Questions

Process evaluation questions might answer To what extent is the improvement activity

being implemented as intended? To what extent is the improvement activity

reaching the target audience (i.e., children, staff, parents)?

Is everyone doing what they said they would do?

Are resources still available to adequately support this improvement activity?


9

Impact Evaluation Questions Impact evaluation questions might include

Did the improvement activity accomplish what it was supposed to?

Which parts of the improvement activity worked well?

Which parts of the improvement activity did not work well?

Should the agency continue the improvement activity?

What has changed as a result of implementing the improvement activity?


10

Reviewing Improvement Activities Improvement activities are aligned to the

indicators, including if reflected across related indicators.

Improvement activities reflect state priorities.

Improvement activities are actionable.

Improvement activities are realistic.


11

Reviewing Improvement Activities Improvement activities include measures of

performance.

Improvement activities include timelines.

Improvement activities identify responsibility for implementation.

Improvement activities include technical assistance needs.


12

Categorizing Improvement Activities

Training and Professional Development Improve Data Collection Improve Systems Administration and Monitoring Improve Collaboration and Coordination Program Development Clarify/Examine/Develop Policies & Procedures Provide Technical Assistance Increase/Adjust FTE Evaluation


13

Developing a Plan for Evaluation

Identify the goal of the evaluation.

Frame the evaluation questions to be answered.

Identify evaluation methods and measurement options.

Identify data sources.

Determine data analysis techniques.

Establish timelines.


What’s Next?

Moving beyond data quality

Understanding and manipulating the data you have

Using data to make program improvements



Looking at Data:

To better understand issues and areas of concern to focus improvement activities... What does the data analysis say? What are the

‘root causes’ of issues or challenges?



Using data for improvement

Evidence

Inference

Action



Evidence

Evidence refers to the numbers, such as

“89% of families reported ...”

The numbers are not debatable



Inference

How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news?

Bad news? News we can’t interpret?

To reach an inference, sometimes we analyze data in other ways (ask for more evidence)



Action

Given the inference from the numbers, what should be done?

Recommendations or action steps

Action can be debatable – and often is

Another role for stakeholders



Program improvement:

At the state level – TA, policy

At the regional or local level – supervision, guidance

At the service/classroom level– implement high quality individualized family centered services



Key points

Evidence refers to the numbers and the numbers by themselves are meaningless

Inference is attached by those who read (interpret) the numbers

You have the opportunity and obligation to attach meaning



Continuous Program Improvement

Plan (vision) Program characteristics

Child and family outcomes

Implement

Check(Collect and analyze data)

ReflectAre we where we

want to be?



Tweaking the System

Plan (vision) Program characteristics

Child and family outcomes

Implement

Check(Collect and analyze data)

ReflectAre we where we

want to be?

Is there a problem?

Why is it happening?

What should be done?

Is it being done?

Is it working?



Outcome questions for program improvement, e.g.

Do outcomes vary by

Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received?



Looking at Family Outcomes by Subgroups

System Characteristics

Family Characteristics

Child Characteristics

Service Characteristics



Are there differences in outcomes across family characteristics?

Race/ethnicity

Family Income

Primary language

Family structure

Etc



Are there differences in outcomes across child characteristics?

Race/ethnicity

Type of disability

Length of time in services

Etc



Examples of process questions

Are ALL services high quality?

Are ALL children and families receiving ALL the services they should in a timely manner?

Are ALL families being supported in being involved in their child’s program?

What are the barriers to high quality services?



Working Assumptions There are some high quality services

and programs being provided across the state.

There are some families who are not getting the highest quality services.

If we can find ways to improve those services/programs, these families will experience better outcomes.



Action Given the inference from the numbers,

what should be done? Develop improvement activities that

are: Targeted based on data analysis Based on evidence based practices Interconnected, work together to

accomplish the desired result



Small Group Scenarios



Small Group Scenarios

Each table is assigned a scenario Choose a table / scenario As a group, walk through the scenario.

Discuss and answer questions. Jot down your ideas and be prepared to

share back some highlights.

early childhood outcomes center1 using data for program improvement christina kasprzak, nectac/eco...

Documents