welcome to the child outcomes data workshop!
Post on 04-Jan-2016
30 Views
Preview:
DESCRIPTION
TRANSCRIPT
Welcome to
the Child Outcomes Data Workshop!
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2
Child Outcomes Data WorkshopChild Outcomes Data Workshop
Pre-Meeting Workshop at the OSEP National Early Childhood Conference
Washington, DC December 2007
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 3
Today’s Cast:Today’s Cast:
SRI International
Kathy Hebbeler
Donna Spiker
The State of Connecticut
Alice Ridgway
The State of Minnesota
Lisa Backer
Frank Porter Graham Child Development Institute
Lynne Kahn
Robin Rooney
Christina Kasprzak
Courtney Valdes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 4
Objectives Objectives
Participants will begin to understand:
1. How to examine the validity of state child outcomes data
2. How to interpret and use valid data
3. How to talk about early and future data with the media
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 5
The Layout of the DayThe Layout of the Day
Morning: Early Data Are the data valid? What can we say to the media (and how do we
say it) about less than ideal data? Lunch On Your Own (12:30 – 2:00)
During which all data become valid. Remember what you ate…
Afternoon: Using Quality Data What can we learn from outcomes data?
Data Analysis Data Analysis as a Tool to as a Tool to
Promote Data Promote Data QualityQuality
Data Analysis Data Analysis as a Tool to as a Tool to
Promote Data Promote Data QualityQuality
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 7
Keeping our eye on the prize:
High quality services for children and
families that will lead to good outcomes.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 8
High Quality Data on OutcomesHigh Quality Data on Outcomes
Data are a piece of a system that helps to achieve overarching goals for children and families
Data yield Findings that can be interpreted as
having a particular meaning that leads to specific actions to improve the system.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 9
Prof’l Development•Preservice•Inservice
System for Producing Good Child and System for Producing Good Child and Family OutcomesFamily Outcomes
Good Federal policies and programs
Good State policies and programs
High quality services and supports for children 0-5 and their families
Good outcomes for children and families
Good Local policies and programs
Adequate funding
Strong Leadership
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 10
The Vision: Using Data as a Tool for The Vision: Using Data as a Tool for Program ImprovementProgram Improvement
State will have quality data available on an ongoing basis about multiple components of the system Child and family outcomes Services provided Personnel (types, qualifications, etc.) Etc.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 11
FMAFMA
Findings
Meanings
Action
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 12
FindingsFindings Findings are the numbers
10% of families responded …… 45% of children were in OSEP category b
The numbers are
not debatable
(assuming the numbers
are correct…)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 13
MeaningMeaning
The interpretation put on the numbers Is this finding
Credible? (Based on valid data?) Good news? Bad news? News we can’t
interpret?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 14
MeaningMeaning
Meaning is debatable. Reasonable people can reach different conclusions from the same set of numbers
Stakeholder involvement can be helpful in making sense of findings
To interpret meaning, sometimes we analyze data in other ways (ask for more findings)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 15
Putting Meaning on the DataPutting Meaning on the Data What are alternative explanations for
the finding? Are there other ways of looking at the data that might provide insight into a possible explanation?
(i.e., should we run more analyses?)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 16
ActionAction Given the meaning put on the findings,
what should be done? Possible actions:
Continue quality assurance activities to improve the quality of the data
Accept the data as credible and develop recommendations based on the findings
Action is always debatable – and often is debated
Another role for stakeholders
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 17
Building quality into the state system Building quality into the state system
Keep errors from occurring in the first place
Develop mechanisms to identify weaknesses that are leading to data collection errors
Provide ongoing feedback including reports of the data to programs and providers
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 18
Procedures to Promote QualityProcedures to Promote Quality Preparing for data collection
Adequate training and communication
During data collection Commitment to the data collection System of supports for the “data providers”
After data collection Data entry Data follow up Data analysis
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 19
Quality of the process: Quality of the process: Preparing for data collectionPreparing for data collection
Training and Communication
Is there a process for checking whether all of the
[data] providers understand what they are to do?
Is there a process for checking whether they do it?
Do they know why they are doing it?
**What do we know about one shot trainings??**
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 20
Quality of the Process:Quality of the Process:During Data CollectionDuring Data Collection
Commitment to the data collection Do providers understand the importance of the activity? Has the system been designed so providers (and families)
will receive benefit from collecting and providing data? Do providers know someone will be checking on what they
are doing?Supports
Has the process been designed to make it as easy and to take as little time as possible? (Can any part be streamlined?)
Is a knowledgeable person observing or tracking data collection activities and providing feedback in a timely manner?
Is there a way for providers to get ongoing questions addressed?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 21
Quality of the Process: After Data Quality of the Process: After Data CollectionCollection
Data entry
Are there safeguards to minimize data entry errors?
Data follow up
Verification: Is there a process in place for checking [a sample of] records for accuracy and completeness?
Is there a process for providing timely feedback when errors are discovered?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 22
Quality of the Process: After Data Quality of the Process: After Data CollectionCollection
Data analysis
Cleaning individual data: Are there procedures for identifying out of range values, anomalies, incomplete data?
Is there a plan for looking at the aggregated data in various ways to identify unexplainable variations, strange patterns, etc.?
Is there a process for providing timely feedback when errors are discovered?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 23
ValidityValidity
Validity refers to the use of the information Does evidence and theory support the
interpretation of the data for the proposed use?
Or Are you justified in reaching the conclusion
you are reaching based on the data? Standards for Educational and Psychological Testing (1999) by
American Educational Research Association, American Psychological Association, National Council on Measurement in Education
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 24
Validity ArgumentValidity Argument
Accumulation of evidence from a series of “if-then” propositions about the data
If the data are valid, then……, e.g., Data should not vary wildly across programs
serving the same kinds of children Data for children with certain kinds of disabilities
should look different than data for other children Etc.
Are there sensible patterns in the data?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 25
In Search of In Search of ValidityValidity
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 26
Take Home MessageTake Home Message
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
Meaning = Data not yet valid
Action = Continue to improve data collection and quality assurance
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 27
The validity of your data is The validity of your data is questionable if:questionable if:
?
Validity Exercise 1Validity Exercise 1
Acquisition and use of knowledge and skills (including early language/communication and early literacy):
Number of children
% of children
a. Percent of preschool children who did not improve functioning 1 4
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 5 22
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 7 30
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 6 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 4 17
Total N= 23 100%
Validity Exercise 1Validity Exercise 1Acquisition and use of knowledge and skills (including early language/communication and early literacy):
% of children
a. Percent of preschool children who did not improve functioning 4
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 22
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 30
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 17
Total 100%
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 30
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
If your n is small, make sure all of your tables and text show the n prominently.
Validity Exercise 2Validity Exercise 2
Acquisition and use of knowledge and skills (including early language/communication and early literacy):
Number of children
% of children
a. Percent of preschool children who did not improve functioning 210 12
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 210 12
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 526 30
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 456 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 351 20
Total N= 1753 100%
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 32
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange.”
Strange = Unexplainable VariationStrange = Unexplainable Variation
But how do you know?But how do you know?
Compared to what?Compared to what?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 33
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange.”
Compared to what you would expect.
Validity Exercise 2Validity Exercise 2
Acquisition and use of knowledge and skills (including early language/communication and early literacy):
Number of children
% of children
a. Percent of preschool children who did not improve functioning 210 12
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
210 12
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 526 30
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 456 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 351 20
Total N= 1753 100%
Validity Exercise 3Validity Exercise 3
Acquisition and use of knowledge and skills (including early language/communication and early literacy):
Number of children
% of children
a. Percent of preschool children who did not improve functioning 35 2
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 210 12
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 228 13
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 456 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 824 47
Total N= 1753 100%
Validity Exercise 3Validity Exercise 3
Acquisition and use of knowledge and skills (including early language/communication and early literacy):
Number of children
% of children
a. Percent of preschool children who did not improve functioning 35 2
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers 210 12
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach 228 13
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers 456 26
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers 824 47
Total N= 1753 100%
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 37
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange.”
Compared to what you would expect. Compared to other data you have
available.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 38
What else do you know (or can find What else do you know (or can find out?)out?)
Child outcomes: D + E = 73% of children exiting meeting
age expectations (reasonable?)
Part C 618 Exit Data: Do the exit data support this?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 39
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange.”
Compared to what you would expect. Compared to other data you have
available. Compared to other states (that you
would expect to be similar).
Validity Exercise 4Validity Exercise 4Acquisition and use of knowledge and skills (including early language/communication and early literacy):
State 1(n= 1,753)
%
State 2(n = 6,543)
%
State 3 (n=2,451)
%
State 4(n=487)
%
a. Percent of preschool children who did not improve functioning 3 2 1 2
b. Percent of preschool children who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
28 21 10 24
c. Percent of preschool children who improved functioning to a level nearer to same-aged peers but did not reach
20 24 46 26
d. Percent of preschool children who improved functioning to reach a level comparable to same-aged peers
27 25 19 20
e. Percent of preschool children who maintained functioning at a level comparable to same-aged peers
22 28 24 28
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 41
State Data SharingState Data Sharing
(AKA Looking (AKA Looking
for Red Flags)for Red Flags)
Validity Exercise 4Validity Exercise 4
Outcome ___
State 1(n=)
%
State 2(n =)
%
State 3 (n=)
%
State 4(n=)
%
State 5(n =)
%
State 6 (n=)
%
State 7(n=)
%
State 8(n=)
%
a. Did not improve functioning
b. Improved but not sufficient to move nearer to functioning comparable to same-aged peers
c. Moved nearer to same-aged peers but did not reach
d. Reached a level comparable to same-aged peers
e. Maintained functioning comparable to same-aged peers
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 43
Validity and GeneralizabilityValidity and Generalizability
Which group do these findings apply to? Is the group with data representative of
children served in the program statewide? By geography By demographics By types of disabilities and
delays By length of time in service
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 44
Non-representative dataNon-representative data
If data does not include: All areas of state
Dallas ≠ Texas All kinds of families in state
No minority families All kinds of children served in program
Only children with severe disabilities
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 45
Non-representative dataNon-representative data
If data does not include: Children who have been in program
the maximum length of time, .e.g., 36 months for Part C.
All states have non-representative child outcomes data in 2008 (and 2009…).
Percentage of Infants & Toddlers Entering Percentage of Infants & Toddlers Entering Services by Age at Entry *Services by Age at Entry *
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36Age in Months
Per
cent
* Age at development of the Individualized Family Service Plan (IFSP).
Eligibility by Age at EntryEligibility by Age at EntryEligibility by Age at EntryEligibility by Age at Entry
32
75
91
36
16
6
32
93
0
10
20
30
40
50
60
70
80
90
100
<12 months 12-24 months > 24 months
Developmental Delay Diagnosed Condition At-risk
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 48
Non-representative dataNon-representative data
If data does not include: Children who have been in program
the maximum length of time, .e.g., 36 months for Part C.
This applies to every state’s data. Complete the “Earliest possible
date….” worksheet.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 49
Implications of Implications of non-representative datanon-representative data
The findings may be valid
BUT
only for the group represented in the data
The findings are not valid for your state overall.
Validity is related to use of the data.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 50
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange.”
3. The data are not representative of the state and the conclusions being drawn suggest they are.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 51
The validity of your data is The validity of your data is questionable if:questionable if:
1. The n is too small.
2. The overall pattern in the data looks “strange” (= unexplainable variation)
3. The data are not representative of the state and the conclusions being drawn suggest they are.
4. The pattern for subgroups looks “strange” (=unexplainable variation).
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 52
Validity Exercise 5Validity Exercise 5
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 53
Data ExplorationData Exploration
Examine the data to look for inconsistencies
If and when you find something strange, look for some other data you have that might help explain it. Is the variation caused by something other than bad data?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 54
Data ExplorationData Exploration
If the variation can be explained, lower the red flag and consider the data valid. Proceed to analyze the data for program improvement (come back after lunch…)
If you conclude the variation is caused by poor quality data (M), develop a targeted plan to improve the data collection (A).
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 55
So when can you trust your data?So when can you trust your data?
When you can’t find any more red flags.
When the errors that remain will not lead to incorrect conclusions.
Improving data collection is a continuous process.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 56
The data can be considered valid for The data can be considered valid for conclusions related to program effectiveness conclusions related to program effectiveness and program improvement when:and program improvement when:
1. The n is sufficiently large.
2. The overall pattern in the data looks reasonable (no unexplainable variation)
3. The data are representative of the state.
4. The pattern for various subgroups in the data looks reasonable (no unexplainable variation)
By locality By disability By ?, ?, ?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 57
How far along is your state?How far along is your state?
Quality of Child Outcomes Data
Time
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 58
Take Home MessageTake Home Message
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
Meaning = Data not yet valid
Action = Continue to improve data collection and quality assurance
top related