accountability: where are we going? martha musser, coordinator nys education department

Accountability: Where Are We Going?

Martha Musser, Coordinator

NYS Education Department

• Contract for Excellence

• NCLB Reauthorization

• NYC and other local accountability initiatives

This may be the Dawning of the Age of Accountability

Contract for Excellence– Governor has proposed a Contract for Excellence that

calls for an enhanced State accountability system, including:

• New accountability standards based on State assessments and other indicators of progress, such as graduation rates or college attendance and completion rates.

• Growth model by 2008-09.• Value-added model by 2010-2011 based on new or

revised state assessments.• Expanded SURR system, resulting in the

identification of up to 5% of State school’s by 2011-2012 for restructuring or reorganization.

Contract for Excellence: Plans for Intervention

• School Review Teams conduct resource, program and planning audits of SINI and SRAP schools and assist all SINI and SRAP schools in development of improvement plans.

• Joint School Intervention Teams, whose members are either appointed by Commissioner or educators from the district, review and recommend plans for reorganizing or reconfiguring schools that are to be closed.

• Distinguished Educators assist low-performing schools and districts.

• The services of all the above are a charge to the school district.

Contract for Excellence: Now it’s Personal

• Commissioner shall define in regulation deficient district performance.

• School districts hiring a superintendent after the effective date of regulation must include a provision in superintendent’s contract that after two years of deficient district performance the superintendent will fully cooperate with a distinguished educator.

• After four years of deficient performance, school board must seek to remove a superintendent or provide a rationale to the Commissioner for why board should not take such action.

• After six years of deficient district performance, the Commissioner shall commence action to remove the board or board members.

Distinguished Educators and Deficient District Performance

• Consult with Commissioner on removal of superintendent.• Serve as ex officio, non-voting members of Board of

Education.• Review all school and district improvement plans and either

endorse or make recommendations to the Board of Education for change.

• Boards of Education must make changes as required by Distinguished Educators unless upon petition by Board of Education Commissioner decides there is a compelling reason not to implement the modification.

Status, Growth and Value- Added Models Defined

Contract for Excellence requires SED to implement growth and value-added models and meet any Federal requirements for such models.

Status Models Defined

Status or improvement models, the current requirement for measuring Adequate Yearly Progress (AYP) under No Child Left Behind, measure progress by tracking the improvement at the same grade levels within the school over time. (This year’s grades 3-5 compared to last year’s grades 3-5.)

Growth Models Defined

Growth models generally refer to accountability models that assess the progress of a cohort of individual students over time with the intent of measuring the progress these students have made (Performance in fourth grade compared to performance in third grade).

Value-Added Models Defined

Value-added models generally refer to a specific type of growth model in which student demographic data or other statistical controls are used to attempt to analyze the specific effects of a particular school, program, or teacher on student learning.

These models ask whether the school has increased the measured achievement of students more than expected based on data from similar schools.

Purpose of Status, Growth, and Value Added Models

• Status model: Determine whether an increasing percentage of students are gaining proficiency over time.

• Growth model: Determine how much progress groups of students are making over time.

• Value-added model: Determine success of schools, programs, or teachers by measuring student growth over time while controlling for non-school variables that impact on student performance.

Value-Added Models: The Holy Grail of Accountability?

– Much controversy about value-added models.– Some claim that good value-added models exist (Sanders),

statistical models not based on vertical scales.– Some claim that the state of the psychometric art does not

currently support vertical scales.– Value-added models are not easy:

• Less sophistication is required to make accurate determinations about status then growth.

• Acquisition of knowledge is not linear.• Value-added models are heavily dependent upon the robustness of their

assumptions.• Value-added models break down more quickly as groups become smaller.

Growth Model System Requirements

Growth Model System Requirements:

• Annual assessments in successive grades

• A unique student identification system

• Vertically aligned or scaled assessments

• Grade by grade standards

• A minimum of two years of assessment data per student

• Value-added systems require additional data on those demographic or resource factors for which the model seeks to control

NY and Growth Models

• NYS Grades 3-8 Testing Program uses Vertically Moderated Standards (VMS)

• Student progress is measured from grade-to-grade relative to proficiency in meeting the standards (rather than in terms of change in scale scores).

• VMS is applicable to some but not all growth model designs.

• NY will by the end of 2006-07 have only two years of test data, limiting our ability to create growth trajectories for individual students.

• SED’s goal is to select a growth model architecture by April.

NCLB: Status Report

• Revised State standards and Annual Measurable Objective’s (AMOs) established.

• 05-06 Accountability Decisions Released.• Development of elementary/middle level

attendance standards in progress.• Regents continuing deliberations regarding raising

high school graduation standards.• Regents engaging in NCLB reauthorization

advocacy.

State Standards

State Standards for 06-07 are:

– 155 for Grade 3-8 ELA and math

– 165 for HS ELA and math

State Standards for 07-8 are:

– 160 for Grade 3-8 ELA and math

– 170 for HS ELA and math

State standard is used to determine which schools must do Local Assistance Plans (LAPs) and which schools and districts may be high performing.

School Year Elementary-Level Middle-Level Secondary-LevelELA Math ELA Math ELA Math

2002-03 123 136 107 81 142 1322003-04 123 136 107 81 142 1322004–05 131 142 116 93 148 1392005–06 138 149 126 105 154 1462006–07 146 155 135 117 159 1522007–08 154 162 144 129 165 1592008–09 162 168 154 141 171 1662009–10 169 174 163 152 177 1732010–11 177 181 172 164 183 1802011–12 185 187 181 176 188 1862012–13 192 194 191 188 194 1932013–14 200 200 200 200 200 200

Original Annual Measurable Objectives for2002–03 to 2013–14

School Year Elementary & Middle-Level Secondary-Level

ELA Math ELA Math

2005–06 122 86 154 146

2006–07 122 86 159 152

2007–08 133 102 165 159

2008–09 144 119 171 166

2009–10 155 135 177 173

2010–11 167 151 183 180

2011–12 178 167 188 186

2012–13 189 183 194 193

2013–14 200 200 200 200

Revised Annual Measurable Objectives for2005–06 to 2013–14

Calculating AMOs

• In 2000-2001 and 2001-2002 combined, 20% of NY’s public school students were enrolled in school’s with PI’s below:– Grade 4 ELA: 123– Grade 8 ELA: 107– Grade 4 Math: 136– Grade 8 Math: 81

• In 2004-05, the percent of students below the 2004-2005 AMO’s were:– Grade 4 and 8 ELA combined: 13.3%– Grade 4 and 8 math combined: 4.2%

• 2005-2006 Grade 3-8 ELA results:– AMO at 13.3% = 122– AMO at 20.0% = 133

• 2005-2006 Grade 3-8 math results:– AMO at 4.2% = 86– AMO at 20.0% = 132

Limited English Proficient (LEP) Students

All LEP students in grade K–12 must take the NYSESLAT annually. LEP students in grades 3 through 8 enrolled in U.S. schools (not

including Puerto Rico) for less than one year (enrolled on or after January 2, 2006) were not required to take the NYSTP ELA assessment in January 2007. For such students who did not take the ELA assessment, valid scores on the NYSESLAT Reading/Writing and Speaking/Listening components will meet the ELA participation requirement.

The eligible LEP students must be identified in the repository using Program Service 0242.

NYSESLAT performance levels will not be used in calculating the Performance Index. LEP students meeting the criteria to use the NYSESLAT in lieu of the ELA will not be included in the Performance Index calculation.

2002 Graduation-RateCohort Definition

This cohort will be used to determine if the district or school meets the graduation-rate requirements for the 2006–07 school year. The 2002 graduation-rate cohort consists of all students, regardless of their current grade status, who were enrolled in the school on October 6, 2005 (BEDS day) and met one of the following conditions:

first entered grade 9 (anywhere) during the 2002–03 school year (July 1, 2002 through June 30, 2003); or

in the case of ungraded students with disabilities, reached their seventeenth birthday during the 2002–03 school year.

The State will exclude the following students when reporting data on the 2002 cohort:

students who transferred to another high school (excluded from the high school graduation-rate cohort) or district (excluded from the district graduation-rate cohort) or criminal justice facility after BEDS day 2005;

students who transferred to an approved alternative high school equivalency preparation (AHSEP) or high school equivalency preparation (HSEP) program (CR 100.7) after BEDS day 2005 and met the conditions stated on the next slide;

students who left the U.S. and its territories after BEDS day 2005 and before August 30, 2006; and

students who died after BEDS day 2005 and before August 30, 2006.

2002 Graduation-Rate Cohort Definition (cont’d)

Students will be removed from the cohort for the school and district from which they transferred to an AHSEP or HSEP program if the final enrollment record shows that on June 30, 2006 the student

a) earned a high school equivalency diploma; or

b) was enrolled in an AHSEP or HSEP program.

Students will be removed from the school cohort if the enrollment records showed that the student transferred to a different high school and was working toward or earned a high school diploma.

Students will be removed from the district cohort if the enrollment records show that the student transferred to a high school in a different district and was working toward or earned a high school diploma.

2002 Graduation-Rate Cohort(Transfers to GED

Removed from Cohort)

2003 Graduation-Rate Cohort

Beginning with the 2003 graduation-rate cohort (used for accountability in 2007-08):

students are included in the cohort based on the year they first enter grade 9 (or for ungraded students, the year they turn 17).

students who have spent at least five months in a district/school during year 1, 2, 3, or 4 of high school are part of the district/school cohort unless they transfer to another diploma-granting program.

Inclusion Rules for the 2003 Graduation-Rate Cohort

A student will be included in the district/school cohort if the student’s last enrollment record in the district or school shows:

that the student was enrolled for at least five continuous (not including July and August) months and the ending reason was not one of the following: transferred to another New York State district or school, died, transferred by court order, or left the U.S.

fewer than five month’s enrollment and an ending reason indicating that the student dropped out or transferred to a GED program and the student’s previous enrollment record in that district/school (assuming one exists):

– indicates that the student dropped out or transferred to a GED program, and

– that the student was enrolled in the district/school for at least five months.

School Status for All State Schools

2005-06 2006-07

Number Percent Number Percent

Good Standing 3,743 84.4% 3,749 84.3%

In Improvement Status under Title 1

502 11.3% 506 11.4%

Requiring Academic Progress 192 4.3% 193 4.3%

Total in improvement status 694 15.6% 699 15.7%

Total Schools 4,437 4,448

NYC Rest of State Total

Schools Enrollment Schools Enrollment Schools Enrollment

Requiring Academic Progress - Year 1 9 10845 36 40043 45 50888







In Need of Improvement - Year 1 62 49523 46 28107 108 77630

In Need of Improvement - Year 2 58 49617 33 27957 91 77574

In Corrective Action 44 32074 23 223305 67 54379

Planning for Restructuring 46 61551 31 30436 77 91987

Restructuring - Year 1 22 17050 20 20459 42 37509



Total Title I SINI 336 309013 170 140465 506 449478

Total SRAP 86 143570 107 132129 193 275699

Grand Total 422 452583 277 272594 699 725177

Note: Enrollment is based on counts from October 2005 BEDS survey.

Student Enrollment by Accountability Status

Statewide Status of SINI and SRAP Schools 2005-06 vs. 2006-07

Status 05-06 Status Improved or Stayed the Same

Status Became Worse Total Schools In Status

Percent Improving or Staying the Same

SINI 1 61 70 131 46.6

SINI 2 33 51 84 39.3

Corrective Action 19 76 95 20.0

Planning for Restructuring

12 31 43 27.9

Restructuring 1 17 58 75 22.7

Restructuring 2 11 62 73 15.1

SRAP 1 44 28 72 61.1

SRAP 2 28 14 42 66.7

SRAP 3 15 44 59 25.4

SPAP 4 6 9 15 40.0

SRAP 5 or 6 3 1 4 75.0

Total 249 444 693 35.9

Note: A school’s status would improve or stay the same if the school made AYP in the area(s) of identification which gave the school its status. A school’s status becomes worse if it fails to make AYP in one or more of the area of identification which gave the school its status.

WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS

3-8 ELA 3-8 Math HS ELA HS Math

Acct StatusAcct

Enrol>30Wtd PI

Acct Enrol>30

Wtd PI

Acct Enrol>30

Wtd PI

Acct Enrol>3

0 Wtd PI

In Good Standing 920,862 163 914,553 168 109,948 183 109,948 185

In Need of Improvement - Year 1 37,283 132 37,069 137 3,694 152 3,694 159

In Need of Improvement - Year 2 41,845 133 41,132 136 4,386 146 4,386 150

In Corrective Action 24,988 127 24,747 125 3,989 144 3,989 149

Planning for Restructuring 32,143 117 31,639 110 9,971 138 9,971 145

Restructuring - Year 1 20,826 119 20,447 114 1,619 122 1,619 130

Restructuring - Year 2 37,304 116 36,921 116 874 132 874 139

Restructuring - Year 3 41,208 116 40,601 111 867 103 867 124

Requiring Academic Progress - Year 1 9,314 154 9,214 154 8,346 180 8,346 184


Requiring Academic Progress - Year 3 9,700 150 9,602 145 6317 169 6,317 170



Requiring Academic Progress - Year 6 443 114 427 99 419 116 419 118

Requiring Academic Progress - Year 7 1,236 94 1,220 91 212 118 212 118

Has No Status - Regulations Do Not Apply 316 145 308 140 0 0 0 0

Total Title I SINI 235,597 123 232,556 122 25,400 140 25,400 147

Total SRAP 53,498 148 52,967 147 44,773 161 44,773 164

Grand Total 1,210,273 154 1,200,384 158 180,121 171 180,121 175

NCLB Reauthorization:

Regents have developed positions regarding:- Schools and districts with multiple federal designations- Growth Models- Targeted interventions- AYP Methodologies- Additional time to meet graduation standards- ELL testing issues- Implementation of choice and SES- Safe Schools

NCLB Reauthorization: What we are Hearing

– Serious discussions about NCLB are beginning but reauthorization more likely to occur in 2009 than 2007, very unlikely to occur in 2008.

– Key issues being discussed:• Full funding• Growth models• N size and confidence intervals• Required interventions, including SES and choice• Assessments: LEP, SWD, high school• Highly Qualified Teachers vs. High Quality Teaching• National Standards• Science Assessments

Other Accountability Initiatives: NYC

• Children First:– Valued Added Model.

– More Ragu than KISS.

– Each school receives letter grade (A-F) on school progress report.

– Progress report consists of four categories:• School Environment (15%)

• Student Performance (30%)

• Student Progress (55%)

• Additional Credit

Children First Progress Report

• School’s grade is based upon how well the school performs relative to other City schools (1/3 of grade) and to a peer group of approximately 40 schools with similar demographics (2/3 of grade).

• State assessments used to measure performance of students over time.

• Additional credit given to improved performance among low achieving students and various disaggregated groups.

• Factors such as parent, student, and teacher surveys; credit accumulation; PSAT scores used in determining school grade.

Challenges for SED

• Growing Pains

• Testing Times

• Vision 2020

• A Cart Before the Horse?

Challenges Ahead for School Districts

Immediate:

• Single Grade 3-8 Performance Index makes schools and districts responsible for more disaggregated groups.

• New standard setting for grade 3-8 assessments may challenge middle schools even more.

• Changes in testing practices for LEP students require that bilingual education programs emphasize rapid acquisition of English as well as fluency in the native language.

• New graduation standards will raise expectations for middle schools and force high schools to reveal “hidden students.”

Longer term:

• If Governor and Legislature show school districts the money, they are also going to demand that districts show results.

Challenges Ahead: Lists, Lists, Lists

How do we make all of these work together:• SURR• SINI, DINI• SRAP, DRAP• IDEA Districts• Title III AMAO’s• Persistently Dangerous Schools• High School Initiative• Highly Qualified Teachers

Key Questions:

• How do we design accountability models that compel movement from awful to adequate without impeding the movement from good to great?

• How do we move from beating the odds to changing the odds?

This may be the Dawning of the Age of Accountability

More Information

Ira Schwartz, Coordinator

Office of School Improvement and Community Services (NYC)

[email protected]

718 722-2796

Accountability PowerPoint for 2006-07:

http://www.emsc.nysed.gov/irts/accountability/2005-06/accountability-rules-Nov2006_files/frame.htm

Manuals for NYS Student Identifier System and the Repository System: http://www.emsc.nysed.gov/irts/SIRS/home.shtml

accountability: where are we going? martha musser, coordinator nys education department

Documents

accountability models

school board

growth models

improvement models

status models

district improvement

age of accountability

years of deficient performance