batya elbaum, ncseam pam roush, west virginia part c osep national early childhood meeting
DESCRIPTION
How to Explain the Numbers: Helping Staff, Parents, and Other Stakeholders Understand the Results of the NCSEAM Surveys for Part C and 619. Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting Arlington, VA, December 2007. Purpose of the session. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/1.jpg)
How to Explain the Numbers:Helping Staff, Parents, and Other
Stakeholders Understand the Results of the NCSEAM Surveys for
Part C and 619
Batya Elbaum, NCSEAMPam Roush, West Virginia Part COSEP National Early Childhood
MeetingArlington, VA, December 2007
![Page 2: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/2.jpg)
Purpose of the session To provide participants with
strategies for explaining the measures and percentages that come from the NCSEAM rating scales addressing Indicators C4 and B8.
![Page 3: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/3.jpg)
Part C Indicator #4
“Percent of families participating in Part C who report that early intervention services have helped the family a) know their rights, b) effectively communicate their children’s needs, and c) help their children develop and learn.”
![Page 4: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/4.jpg)
Part B Indicator #8
“Percent of parents with a child receiving special education services who report that schools facilitated parent involvement as a means of improving services and results for children with disabilities.”
![Page 5: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/5.jpg)
WV Part C Indicator 4 4A – Know rights
2006 76.5% 2007 73.8%
4B – Communicate 2006 72.8% 2007 71.0%
4C – Help child 2006 86.3% 2007 78.7%
![Page 6: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/6.jpg)
Change in mean measure by region
Region 2006
N 2006 Mean
2007 N
2007 Mean
Change in Mean
1000 Part C Impact on Family Measures 62 716.41 44 705.32 -11.09
1100 Part C Impact on Family Measures 64 702.91 44 715.36 +12.45
1200 Part C Impact on Family Measures 50 665.87 47 670.60 +4.73
1300 Part C Impact on Family Measures 93 717.19 58 688.85 -28.33
1400 Part C Impact on Family Measures 28 755.98 24 637.00 -118.98
1500 Part C Impact on Family Measures 25 615.88 25 627.71 +11.84
1600 Part C Impact on Family Measures 48 695.17 33 651.53 -43.63
1700 Part C Impact on Family Measures 90 706.44 53 681.74 -24.69
![Page 7: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/7.jpg)
Percent on 4A by Region in 2006 and 2007
2006 2007
Region N 4A % N 4A %
1000 62 72.58 44 70.45
1100 64 71.87 44 86.36
1200 50 72.00 47 74.47
1300 93 .78.49 58 77.59
1400 28 92.86 24 58.33
1500 25 60.00 25 64.00
1600 48 79.17 33 63.64
1700 90 81.11 53 79.25
![Page 8: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/8.jpg)
Why use a measurement analysis?
![Page 9: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/9.jpg)
Why use a measurement analysis?
Use of a measurement framework ensures that measures will mean the same thing, regardless of how many items, or which specific items, are administered.
![Page 10: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/10.jpg)
Why use a measurement analysis?
We can’t assume that all survey items are equally agreeable.
A measurement analysis gives us a measure of each item’s overall agreeability.
![Page 11: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/11.jpg)
Order of Impact on Family items from least to most agreeable
Item Calibration
Item Stem: Over the past year, Early Intervention services have helped me and/or my family:
678 Participate in typical activities for children and families in my community.
656 Know about services in the community.
640 Know where to go for support to meet my family's needs.
625 Keep up friendships for my child and family.
609 Know where to go for support to meet my child's needs.
584 Be more effective in managing my child's behavior.
576 Make changes in family routines that will benefit my child with special needs.
576 Do activities that are good for my child even in times of stress
570 Improve my family's quality of life.
565 Feel that I can get the services and supports that my child and family need.
563 Get the services that my child and family need.
562 Feel that my family will be accepted and welcomed in the community
559 Feel more confident in my skills as a parent.
559 Feel that my child will be accepted and welcomed in the community.
556 Communicate more effectively with the people who work with my child and family.
553 Understand how the Early Intervention system works.
546 Understand the roles of the people who work with my child and family.
539 Know about my child's and family's rights concerning Early Intervention services.
534 Be able to evaluate how much progress my child is making.
516 Understand my child's special needs.
498 Feel that my efforts are helping my child.
498 Do things with and for my child that are good for my child's development.
![Page 12: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/12.jpg)
Why use a measurement analysis?
Use of a measurement framework allows us to test whether all the items are measuring the same thing.
![Page 13: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/13.jpg)
Measuring the 3 subindicators
NCSEAM’s measurement analysis of pilot data from thousands of families showed that items that are related to the three OSEP outcomes all fit into a single scale of family outcomes.
![Page 14: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/14.jpg)
Measuring the 3 subindicators
Families appear to achieve these outcomes in a very consistent order. Families who report that EI helped them
know their rights also report that EI also helped them help their child develop and learn.
Families who report that EI helped them effectively communicate their children’s needs also report that EI also helped them know their rights and help their child develop and learn.
![Page 15: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/15.jpg)
Location of key items related to Indicator C4
Indicator 4c: Help their children develop and learn. IFS items: “Understand my child's special needs.” [516] “Do things with and for my child that are good for my child's development.” [498]
Indicator 4b: Effectively communicate their children’s needs. IFS Item: “Communicate more effectively with the people who work with my child and family.” [556]
Indicator 4a: Know their rights.IFS Item: “Know about my child's and family's rights concerning Early Intervention services.” [539]
556
539
516
![Page 16: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/16.jpg)
Why use a measurement analysis?
We can’t assume that the response choices (very strongly disagree, strongly disagree, disagree, agree, strongly agree, very strongly agree) indicated by equidistant circles or numbers on a page are really equidistant.
![Page 17: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/17.jpg)
Spacing of response choices
◘ ◘ ◘ ◘ ◘ ◘ 1 2 3 4 5 6
![Page 18: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/18.jpg)
Spacing of response choices
◘ ◘ ◘ ◘ ◘ ◘ 1 2 3 4 5 6 1K 5K 10K ½ Mar. Marathon Ultra
![Page 19: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/19.jpg)
The logic of distributions
![Page 20: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/20.jpg)
Example: Distribution of height
![Page 21: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/21.jpg)
Distribution of Measures of Reading Achievement - 1
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X >5’ 5’0 5’2 5’4 5’6 5’8 5’10 6’0 6’2 6’4 6’6 6’8 6’10 7’ >7’
![Page 22: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/22.jpg)
Distribution of Measures of Reading Achievement - 2
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X >5’ 5’0 5’2 5’4 5’6 5’8 5’10 6’0 6’2 6’4 6’6 6’8 6’10 7’ >7’
![Page 23: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/23.jpg)
West Virginia results 2006
![Page 24: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/24.jpg)
West Virginia results 2007
![Page 25: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/25.jpg)
WV without extreme cases 2006
![Page 26: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/26.jpg)
WV without extreme cases 2007
![Page 27: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/27.jpg)
Instrumentation and approach to data analysis make a difference
![Page 28: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/28.jpg)
Choice of instruments: Part B NCSEAM K-12 and 619 9 NCSEAM K-12 21 Customized NCSEAM survey 11 State-developed or adapted 18 ECO Family Outcomes Survey 1
![Page 29: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/29.jpg)
States’ reported baseline data onIndicator 8B
![Page 30: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/30.jpg)
Choice of instruments: Part C NCSEAM Survey 25 ECO Family Outcomes Survey 18 State-developed or adapted 10 Combination 1
![Page 31: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/31.jpg)
Analysis of states’ baseline data:Indicator C4
Sub-Indicator a. Know their
rights b. Communicate children’s needs
c. Help children develop and learn
Mean 79% of families 78% of families 85% of families
Range 45% - 99% 42% - 99% 53% - 99%
![Page 32: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/32.jpg)
Analysis of states’ baseline data:Indicator C4
Figure 2: Baseline by Survey Tool and Criteria for Positive Response
0
10
20
30
40
50
60
70
80
90
100
ECO 3 & above
N=2
ECO5 & above
N=16
NCSEAM alternate analysis
N=6
NCSEAM standard analysis
N=17
State Survey
N=11
Tool and Analysis
% a
% b
% c
![Page 33: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/33.jpg)
APR Family Outcome Results – Part C States Using NCSEAM Survey and RASCH analysisAPRs submitted February 2007
4A 4B 4C Response Rate
#
California 48% 42% 71% * 5,413 Connecticut 73% 71% 84% 27% 712 Florida 56% 52% 57% 25% 402 Idaho 56% 52% 68% 14.74% 355 Kentucky 82% 73% 89% 22% 786 Mass. 80% 77% 86% 30% 290 Michigan 56% 51% 73% 40.8% 2,439 Nebraska 76% 73% 86% 48% 921 New Mexico 78% 72% 92% 23.8% 183 New York 71% 65% 82% 25.6% 580 Pennsylvania 67% 63% 79% 23.81% 3,018 Utah 74% 71% 79% 25.4% 605 Wyoming 78% 85% 86% 30% 230 South Carolina 74% 70% 86% ? ?
![Page 34: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/34.jpg)
Reference to a standard
![Page 35: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/35.jpg)
Percent of measures above the adopted standards
standards
![Page 36: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/36.jpg)
What is the relationship between % at or above the NCSEAM standard and % agreement on the “threshold item?”
![Page 37: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/37.jpg)
Sample descriptive interpretation related to Indicator 4A Approximately 90-95% of families agreed,
with approximately two-thirds of families expressing strong or very strong agreement, that early intervention helped them:
Understand their child’s special needs. Do things with and for their child that
are good for their child’s development. Feel that they can handle the
challenges of parenting a child with special needs.
![Page 38: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/38.jpg)
Part C Example: WV data
![Page 39: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/39.jpg)
Q44 - know about my child's and family's rights
AKnow Response Category Frequency Percent
% Agree, SA, VSA
1 5 6.0 2 1 1.2 3 1 1.2 4 65 78.3 5 9 10.8
6 2 2.4 91.5 Total 83 100.0
Below cut-score on this standard
Missing 3
Total 86 4 15 6.4 5 57 24.5
6 161 69.1 100.0 Total 233 100.0 Missing 9
At or above cut-score on this standard Total 242
![Page 40: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/40.jpg)
Q42 - communicate more effectively
BComm Response Category Frequency Percent % Agree, SA,
VSA 1 4 4.8 2 3 3.6 3 6 7.1 4 67 79.8
5 4 4.8 84.6 Total 84 100.0 Missing 11
Below cut-score on this standard Total 95
3 1 .5 4 15 7.0 5 62 29.0
6 136 63.6 99.5 Total 214 100.0 Missing 19
At or above cut-score on this standard
Total 233
![Page 41: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/41.jpg)
Q46 - understand my child's special needs
CHelp Response Category Frequency Percent
% Agree, SA, VSA
1 6 9.2 2 1 1.5 3 4 6.2 4 50 76.9 5 4 6.2 83.1 Total 65 100.0 Missing 5
Below cut-score on this standard
Total 70
3 1 .4 4 24 10.4 5 52 22.5
6 154 66.7 99.6 Total 231 100.0 Missing 27
At or above cut-score on this standard Total 258
![Page 42: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/42.jpg)
Part B ExampleItem: “The school explains what options parents have it they disagree with a decision of the school.”
SEPPS Measure
% VSD
% SD
% D
% A
% SA
% VSA
Total % A/SA/VSA
>=350<400 21.9 8.8 47.4 20.2 0.9 0.9 22.0 >=400<450 7.6 6.7 51.4 33.3 0.0 1.0 34.3 >=450<500 4.1 2.0 27.6 57.1 7.1 2.0 66.2 >=500<550 3.3 3.3 18.0 62.0 8.7 4.7 75.4 >=550<600 0.0 0.0 16.0 54.0 16.0 14.0 84.0 >=600<650 0.0 3.1 4.6 26.2 44.6 21.5 92.3 >=650<700 0.0 0.0 3.3 30.0 20.0 46.7 96.7 <600 (n=704) 704/903 = 78%
22.2 6.3 28.4 35.9 4.4 2.8 43.1
>=600 (n=199) 199/903 = 22%
0.5 1.0 4.5 15.6 24.1 54.3 94.0
ALL 17.4 5.1 23.1 31.5 8.7 14.2 54.4
![Page 43: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/43.jpg)
The NCSEAM standard is a stringent standard Cut score for B at 600 Cut scores for C at 539, 516, 556 The standard is set so as to ensure
that approximately 95% or more of the responses will be in one of the agree categories on the item designated as the “threshold item”
Simple agreement with the “threshold item” is not enough
![Page 44: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/44.jpg)
How precise are the percentages that states are reporting?
![Page 45: Batya Elbaum, NCSEAM Pam Roush, West Virginia Part C OSEP National Early Childhood Meeting](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c00550346895da55cec/html5/thumbnails/45.jpg)
Error in estimation Whenever data are based on a
sample (whether owing to sampling or to a less than 100% return rate), there is some amount of error in generalizing to the population.
“Error,” or imprecision, can be represented by a confidence interval.
As sample size decreases, error of estimate - and its representation as a confidence interval - increases.
The decrease is not linear!