Accountability and Students with Disabilities: Assuring Valid Inferences about Teachers and Schools

CCSSO National Conference on Student AssessmentNew Orleans, LAJune 27, 2014Strand 11A

Heather Buzick

Research ScientistEducational Testing

ServiceCenter for Validity

Research

Ann Schulte

Research ProfessorArizona State University

James Ysseldyke

Birkmaier Professor of Educational LeadershipUniversity of Minnesota

Daniel Wiener

Administrator of Inclusive Assessment

Massachusetts Department of Elementary and

Secondary Education

IES-funded 5-state longitudinal study on

student growth 

Research on teacher evaluation

Research supported by IES-funded National

Research and Development Center on Assessment and Accountability for Special

Education

Observations from Massachusetts and discussant remarks

National advisory panel member for NCAASE

NCEO research

Assuring valid inferences about teachers and schools…

What do we know about the academic growth of students with disabilities?

What choices do we have for modeling,

reporting, and evaluating?

How can we make valid inferences

about teachers and schools educating

students with disabilities?

Copyright © 2014 by Educational Testing Service. 4

Describing and Using Growth from Students with Disabilities on Summative Assessments

Heather BuzickEducational Testing ServicePrinceton, NJ

A portion of this research was funded by the Institute for Education Sciences (Award #R324A120224)

Copyright © 2014 by Educational Testing Service. 5

Current research•Describi

ng Summative Academic Growth for Students with Disabilities: A Summary from 5 States

Study 1

•Using Test Scores from Students with Disabilities in Teacher Evaluation

Study 2

Copyright © 2014 by Educational Testing Service. 6

Motivation and importance

• Approximately 14% of students have a diagnosed disability• The majority spend most of instructional time in general

education classroom*• Approximately 80% of teachers have at least one student

with a disability in their classroom**• At least 75% of students with disabilities take the general

assessment* • Students’ disabilities can have an impact on access to test

content, student may require testing accommodations, teaching and learning may differ from other students

• How should students’ test scores be included in accountability systems?

*from Historical State-Level IDEA Data Files (http://tadnet.public.tadnet.org/pages/712**Estimate. Sources available from author

Copyright © 2014 by Educational Testing Service. 7

Study 1 Research Questions

What does growth on state summative

assessments look like for students with

disabilities?

Does growth differ by grade, content area,

and disability subtype?

How does the growth of students with

disabilities compare to students without a

disability?

Where is growth for students with

disabilities occurring relative to the

proficiency cutscore?

Copyright © 2014 by Educational Testing Service. 8

Some definitions of growth within individual students

• Differences in vertically scaled scores (gains)– E.g., 300 scaled score in grade 3, 320 scaled

score in grade 4

• Transitions across proficiency levels– E.g., “basic” in grade 3, proficient in grade 4

• Student growth percentiles – From grade 3 to grade 4, the student grew as

much as or more than 70 percent of other students in the state peers who had similar grade 3 test scores

3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Grew below proficiency Grew across proficiency Grew above proficiency

ELA

0

5

10

15

20

25Transitions across proficiency levels

NoneAutismEmotional ImpairmentOther health impairmentSpecific Learning DisabilitiesSpeech or language impairment

Pe

rce

nt

3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Grew below proficiency Grew across proficiency Grew above proficiency

Math

0

5

10

15

20

25Transitions across proficiency levels

NoneAutismEmotional impairmentOther heath impairmentSpecific learning disabilitiesSpeech or language impairment

Pe

rce

nt

3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Low Moderate High

ELA

0

10

20

30

40

50

60

70Student growth percentiles

NoneAutismEmotional impairmentOther health impairmentSpecific learning disabilitiesSpeech or language impairment

Pe

rce

nt

3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8 3-4 4-5 5-6 6-7 7-8Low Moderate High

MATH

0

10

20

30

40

50

60

70Student growth percentiles

NoneAutismEmotional impairmentOther health impairmentSpecific learning disabilitiesSpeech or language impairment

Pe

rce

nt

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3Difference in 5th Grade Math Teachers’ Value-added Scores

Classrooms with no students with disabilitiesMixed classroomsSpecial education-only classrooms

Percent of students with disabilities in the classroom

[Mo

de

l w

ith

dis

ab

ilit

y-r

ela

ted

va

ria

ble

s]-

[Mo

de

l w

ith

ou

t d

isa

bil

ity

-re

late

d v

ari

ab

les

]

Copyright © 2014 by Educational Testing Service.

Conclusions

• The model matters for the inferences we make about schools, teachers, and individual students (sometime is subtle, but important ways)– Policy makers: identify the claims they wish to make– Measurement experts: help policymakers understand the

meaning derived from a particular model

• How much growth is enough?– Norms based on accumulated growth data from multiple

sources– Prediction associated with college- and career-ready standards

Accountability Dilemmas for Students with Disabilities and Policy Alternatives

Ann SchulteArizona State University

Natalie MurrNorth Carolina State University

Joseph StevensUniversity of Oregon

17

National Center on Assessment & Accountability for Special Education

• NCAASE www.ncaase.com

• Institute of Education Sciences, 2011-2016• Co-PI’s

– Stephen Elliott & Ann Schulte, Arizona State Univ– Joseph Stevens & Gerald Tindal (Project Director), Univ of

Oregon

This work is supported by the Institute of Education Sciences, U.S. Department of Education, through grant R32C110004 awarded to the University of Oregon. The opinions expressed are those of the authors and do not necessarily represent views of the Institute or the U.S. Department of Education.

19

NCAASE 2011-2016:Our Key Research Questions

1. What is the natural developmental progress in achievement for students with disabilities?

2. What models best characterize achievement growth for students with disabilities who are participating in general achievement tests?

3. How do various growth models represent school effects for students with and without disabilities, and how do results compare to those derived from the status models now in use?

4. How do results from different types of interim assessments of students’ achievement meaningfully contribute to a model of academic growth for students with disabilities?

5. How can information about opportunity to learn and achievement growth be used to enhance academic outcomes for students with disabilities?

Persistent Accountability Dilemmas

• Bias introduced by including only current students with disabilities in students with disabilities (SWD) subgroup (Ysseldyke & Bielinski, 2002)

• “One shot” model of assessing proficiency and SWD performance variability—retests to assure assessment fairness (Wei, 2012)

• Students start at differing levels, status measures do not consider student progress relative to starting point—importance of looking at growth to assess school and teacher effects (Dunn & Allen, 2009; Stevens, 2005)

Data Sources for Presentation

• North Carolina test data (NCAASE also looking at AZ, OR, PA)

• Cross sectional-2010– Allowed retests for non-proficient student, inclusion of

students who had exited special education for two years or less

– State-level growth metric—residual gain score using two prior years’ test scores, z-score score based on mean gain and sd in standard setting year

• Longitudinal—Math 2001-2005 cohort, Reading 2003-2007 cohort

Impact of Two Specific Policies

• Including students who have exited special education

• Allowing retesting for students who do not reach proficiency

23

3 4 5 6 7-1.20

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

Special Education in Current Year SWD

Special Education at Wave 1 SWD

Grade

Eff

ect

Siz

e

Mathematics Achievement Gap

Stable Subgroup Membership Matters

Change in Mean Number of Students Reaching Proficiency

Rdg/LA Mathematics

Mean SD Mean SD

Current Participation in Special Education Only 40 21 59 20

Including Students Two Years Post Dismissal 47 19 65 18

Net Change in Percent Proficient

+7 +6

24

Change in School-level Percent Proficient for SWD w/ Exiters Included

Mathematics Reading0

10

20

30

40

50

60

70

80

90

100

Current onlyWith 2-Yr Exited

SWD’s Reaching Math Proficiency With and Without Retest

Grade 3 Grade 4 Grade 50

10

20

30

40

50

60

70

80

90

100

Single TestW/ Retest

SWD’s Reaching Reading Proficiency With and Without Retest

Grade 3 Grade 4 Grade 50

10

20

30

40

50

60

70

80

90

100

Single TestW/ Retest

Growth vs. Proficiency

• What does growth across grades look like for specific exceptionalities?

• Relationship between status and growth for students with and without disabilities

Mathematics Growth by Exceptionality

3 4 5 6 7235

240

245

250

255

260

265

270

275

General Education

Speech-language Impairment

NC Proficiency Cutpoint

Grade

Mat

hem

atic

s S

cale

Sco

re

Mathematics Growth by Exceptionality

3 4 5 6 7235

240

245

250

255

260

265

270

275

General Education

Speech-language Impairment

Autism

Hearing Impairment

NC Proficiency Cutpoint

Grade

Mat

hem

atic

s S

cale

Sco

re

Mathematics Growth by Exceptionality

3 4 5 6 7235

240

245

250

255

260

265

270

275

General Education

Speech-language Impairment

Autism

Other Health Im-pairment

Hearing Impairment

Specific Learning Disability

NC Proficiency Cutpoint

Grade

Mat

hem

atic

s S

cale

Sco

re

Mathematics Growth by Exceptionality

3 4 5 6 7235

240

245

250

255

260

265

270

275

General Education

Speech-language Impairment

Autism

Other Health Im-pairment

Hearing Impairment

Emotional Dis-turbance

Specific Learning Disability

Intellectual Disability

NC Proficiency Cutpoint

Grade

Mat

hem

atic

s S

cale

Sco

re

Reading Growth by Exceptionality

3 4 5 6 7230

235

240

245

250

255

260

265

General Education

Speech-language Impairment

NC Proficiency Cutpoint

Grade

Rea

ding

Sca

le S

core

Reading Growth by Exceptionality

3 4 5 6 7230

235

240

245

250

255

260

265

General Education

Speech-language Impairment

Autism

Hearing Impairment

NC Proficiency Cutpoint

Grade

Rea

ding

Sca

le S

core

Reading Growth by Exceptionality

3 4 5 6 7230

235

240

245

250

255

260

265

General Education

Speech-language Impairment

Autism

Other Health Im-pairment

Hearing Impairment

Specific Learning Disability

NC Proficiency Cutpoint

Grade

Rea

ding

Sca

le S

core

Reading Growth by Exceptionality

3 4 5 6 7230

235

240

245

250

255

260

265

General Education

Speech-language Impairment

Autism

Other Health Im-pairment

Hearing Impairment

Emotional Dis-turbance

Specific Learning Disability

Intellectual Disability

NC Proficiency Cutpoint

Grade

Rea

ding

Sca

le S

core

Growth by Starting Proficiency Level-Math

General Ed SWD

Growth by Starting Proficiency Level-Rdg

General Ed SWD

Conclusions

• SWD subgroup is not stable and policy changes allowing longer time to “count” in subgroup improve school SWD outcomes

• Retesting benefits SWDs and may also be likely to benefit other groups characterized by large achievement gaps

• SWDs show growth mathematics and reading achievement across grades, although improvement may not be reflected in changes in status (Non-proficient/proficient)

• Large differences in starting point achievement skills within SWD group, smaller differences in growth

Accountability and Students with Disabilities: Assuring valid inferences about teachers and schools

Jim Ysseldyke

Purposes of Monitoring Student Growth

• District/State Accountability• Individual Progress

Monitoring/instructional planning• Teacher evaluation (value added)

Typical Accountability Models for SWD

• Cross-sectional• Cohort Static• Cohort Dynamic

Typical Scores

• Scaled Scores• Proficiency Levels• Effect Sizes• More recently Student Growth

Percentiles (ala Betebenner) or Student Deciles in some of our work

Main Issues

• Reducing achievement gap (GE v SE)• Nobody wants SWD in their

accountability profile• How long should SWD count?• What model should be used?• What scores should be used?

Major Points I Heard

• Students start at differing levels, so status measures do not consider student progress relative to starting point

• Use of cross sectional dangerous• SWD are growing, but many may not

meet proficiency standards

Major Points I Heard

• We have limited data on growth norms for SWD (small Ns)

• Much concern about how long to count SWD (at district or state level).

Achievement Gap Using 3 Analytic Methods over 6 Years

21.1

27.3

32.0

40.646.2

47.4

22.0

28.4 31.0

36.3

47.0

45.4

22.6 22.725.1 26.2

21.7

0

10

20

30

40

50

60

3 4 5 6 7 8

Per

cent

age

of S

tude

nts

abov

e P

rofic

ienc

y Le

vel

Grade

Cross Sectional Cohort Dynamic Cohort Static

Spring-Fall SGP Growth by Category v National Norms

STAR Reading Growth for Grade 10 SED Students

Spring to Fall SGP for Students with SED in Differing Programs

Big Ideas

• We can identify expectations for SWD, and we can build growth norms for SWD, but it will take large databases to do so. State tests change too often. Suggest large databases such as Renaissance Learning STAR databases, AIMSweb, Easy CBM. FAST or Oregon Behavioral Research & Teaching Institute

Big Ideas

• Use Cohort Static Model and Effect Sizes• Include SWD in SWD group for two years

after exit as a policy rule• SWD should count in teacher evaluation

systems• More research on how accommodation use

should be factored into growth models• The promise of Goal Setting Tool

Accountability and Students with Disabilities:

Assuring Valid Inferences About Teachers and Schools

Dan Wiener

Administrator of Inclusive Assessment

CCSSO NCSA

June 27, 2014

MASSACHUSETTS:Student Growth Percentiles (SGP)+ District-Determined Measures

(DDM)= Student Impact Rating

Students with disabilities (SWD) may demonstrate high growth, but are more likely to have a lower SGP.

Therefore, teachers of SWD would be more likely to meet the definition for low growth, if student growth was the only measure.

Determine Student Impact Rating for those educators using professional judgment of multiple measures over multiple years.

DDMs should reflect the different ways SWD show learning and growth.

Massachusetts Department of Elementary and Secondary Education

54

Multiple Measures

Districts are required to use median SGPs as one measure for evaluating teacher effectiveness

One or more DDMs must also be used as a measure.

Consider the learning setting, shared responsibilities of teachers, creative use by districts of composite measures

Educators should be involved in setting these parameters.

Massachusetts Department of Elementary and Secondary Education

55

Massachusetts Department of Elementary and Secondary Education

Statewide MCAS ResultsStudents with Disabilities

Change in Performance 2009 to 2013 (5-Year Trends)

Percent of Students with Disabilities Scoring Needs Improvement and Higher

English Language Arts

Mathematics Science & Tech/Eng.

2009 2013 Change 2009 2013 Change 2009 2013 Change

Grade 3 65 68 +3 59 57 -2

Grade 4 58 56 -2 62 61 -1

Grade 5 70 62 -8 51 50 -1 66 63 -3

Grade 6 70 63 -7 47 51 +4

Grade 7 70 70 0 38 46 +8

Grade 8 72 76 +4 38 43 +5 47 50 +3

Grade 10 81 88 +7 68 73 +5 70 79 +9

Massachusetts Department of Elementary and Secondary Education

57

ELA

% of Students Scoring Needs Improvement or

Higher

Between-Group Gap:Percentage Point Difference

Students with

Disabilities

Non-disabledStudents

Students with Disabilitiesvs. Non-disabled Students

Grade 3 69 97 28Grade 4 54 94 40Grade 5 63 96 33Grade 6 64 96 32Grade 7 72 98 26Grade 8 70 97 27Grade 10 88 99 11

Spring 2013 MCAS ELA:Achievement Gap between Students with Disabilities and Non-disabled

Students:

Massachusetts Department of Elementary and Secondary Education

58

Mathematics

% of Students Scoring Needs Improvement or

Higher

Between-Group Gap:Percentage Point Difference

Students with

Disabilities

Non-disabledStudents

Students with Disabilitiesvs. Non-disabled Students

Grade 3 63 94 31Grade 4 65 96 31Grade 5 54 93 39Grade 6 50 92 42Grade 7 39 87 48Grade 8 40 88 48Grade 10 70 97 27

Spring 2013 MCAS Mathematics:Achievement Gap between Students with Disabilities and Non-disabled

Students:

End of Gr. 10 (2011)

End of Gr. 11 (2012)

End of Gr. 12 (2013)

0

20

40

60

80

100

9397

99

61

7681

43

58

82

Percent of Students in Class of 2013 Scoring Needs Improvement or

Higher on MCAS ELA, Math, and STE

Non-Disabled Students Students with Disabil-itiesEnglish Language Learners

Perc

en

t

59

Massachusetts Department of Elementary and Secondary Education