Education Commission of StatesJuly 2, 2008

Austin, TexasDr. Charity Smith, Assistant Commissioner

Arkansas Department of Education

Measuring Academic Gains in Arkansas

US Department of EducationGrowth Model Pilot Study

7 core principles of state-proposed growth models

100% proficiency by 2014 Establish appropriate growth targets at student level Separate accountability decisions for English Language

Arts and Mathematics Inclusion of all students State assessment system and methodology Tracking student progress Student participation rates and additional academic

indicators (USDOE, 2005)

Two Systems for Measuring Growth

Arkansas Federal Growth Model

Difference in Gain Scores Model (DGSM)

Arkansas Act 35 Gains Model

Transition Matrix Model (TMM)

Difference in Gain Scores Model

Difference between a student's current score and the score that would meet standards in a set number of years

Dividing the difference by the number of years gives the annual gain needed. A student's actual gain can be compared to the target growth to see if the student is on track to meet standards.

Different scores can be aggregated to the school or district level to obtain a group growth measure.

(CCSSO, 2007)

Arkansas Model DGSM

Arkansas (USDOE, 2007) Scale Score and Proficient Level is grade and subject

specific Annual growth increment is a function of the following:

Proficiency Scale Score Standard for Subsequent Grade Proficiency Scale Score Standard for Current Grade Proficiency Scale Score Standard at Grade 8 Student’s current grade scale score

CG

G

CGG

CGSG

SS

PSS

PSSPSS

PSSPSSGI 8

8

Technical Advisory Committees (Act 35 Assessment)

William J. Brown, Jr., Brownstar, Inc.

Gregory J. Cizek, University of North Carolina at Chapel Hill

Huynh Huynh, University of South Carolina, Chair

Robert W. Lissitz, University of Maryland

William A. Mehrens, Michigan State University (Emeritus

Professor)

E. Roger Trent, Trent Consulting Inc.

William Brown, Jr., Brownstar, Inc., Chair

Thomas Fisher, Fisher Education Consulting LLC

Huynh Huynh, University of South Carolina, Chair

Eugene Kennedy, Louisiana State University

Robert Kennedy, University of Arkansas for Medical Sciences

E. Roger Trent, Trent Consulting Inc.

Arkansas Act 35 Accountability Team

Charity Smith, Assistant Commissioner

Willie Morris, Associate Director

Louis Ferren, Data Manager

Kimberly Millins, Special Projects Director

Oliver Dillingham, Program Manager

School Rating System

The Department of Education shall prepare an annual report which shall describe the school rating system. The annual report shall designate two (2) category levels for each school.

The first category, annual performance, is based on performance from the prior year on the criterion-referenced test and end-of-course exams.

The second category, growth, shall be based on the schools’ improvement gains tracked longitudinally and using value-added calculations on the criterion-referenced test.

This presentation will focus only on the second category.

Intention of Act 35

Promoting student learning at all levels so that “all students have an opportunity to demonstrate increased learning” and “meet the expected academic standards.”

Improving student achievement through school accountability and recognition

Requiring that each school be classified into “two (2) category levels” as follows:

-- “Category One” for the “school’s improvement gains” tracked longitudinally using value-added calculation known as the annual improvement category level and

-- “Category Two” based on “performance from the prior year” referred to as the annual performance category (or “status”).

Annual Improvement Category

The annual improvement category for rating schools will report each school’s improvement gains by tracking student’s longitudinal achievement gains on the state’s augmented criterion-referenced tests.

The base year for the growth model is 06-07. The first reports will be available in December of 2008.

A school’s annual improvement gain is based upon the changes in student achievement from one year to the next.

No value will be added if instruction does not move a student’s achievement from a given performance category to a higher performance category.

If a student’s achievement moves to a lower performance category, then value is lost.

Initial Design Considerations

Growth (“Improvement Gain”) will be based on the tests included in the Arkansas Comprehension Testing, Assessment and Accountability Program (ACTAAP).

An annual improvement gain model will be based on an aggregation of student changes in literacy and mathematics for grades 3-8 and for literacy in grade 11.

So far work has been conducted using test data for Grades 3-8 only.

The annual improvement gain model shall be designed with the expectation that (a) students who are proficient or higher will either maintain or improve their performance classification, and (b) students who are basic or below basic will reach proficiency.

Annual Improvement Category Levels

Level 1: Schools in need of immediate improvement

Level 2: Schools on alert

Level 3: Schools meeting improvement standards

Level 4: Schools exceeding improvement standards

Level 5: Schools of excellence for improvement

Calculations for Annual Student Gains

Student growth is based upon changes in student performance levels across two adjacent years.

To assess annual changes more precisely each student performance level (Below Basic, Basic, Proficient, and Advanced) is split into two sub-categories.

The subcategories are: BB1, BB2, Basic1, Basic2, Prof1, Prof2, Adv1, and Adv2.

For example, if a student in third grade earns a achievement rating of Proficient 2 and the following year the same student in fourth grade receives an achievement rating of advanced 1, he/she would be given a value-added score of +0.5. Similar value-added scores would be calculated for each child in each school.

Table 1: Student Performance Sub-categories

Student Performance Sub-categories

Points Assigned

Below Basic 1 1

Below Basic 2 1.5

Basic 1 2

Basic 2 2.5

Proficient 1 3

Proficient 2 3.5

Advanced 1 4

Advanced 2 4.5

Table 2: Value-added Point for Changes inStudent Achievement

Previous Year

Current Year

1 1.5 2 2.5 3 3.5 4 4.5

1 0 .5 1 1.5 2 2.5 3 3.5

1.5 -.5 0 .5 1 1.5 2 2.5 3

2 -1 -.5 0 .5 1 1.5 2 2.5

2.5 -1.5 -1 -.5 0 .5 1 1.5 2

3 -2 -1.5 -1 -.5 0 .5 1 1.5

3.5 -2.5 -2 -1.5 -1 -.5 0 .5 1

4 -3 -2.5 -2 -1.5 -1 -.5 0 .5

4.5 -3.5 -3 -2.5 -2 -1.5 -1 -.5 0

• In 2007, the school improvement gain index ranged from approximately -0.50 to 0.50.

• Conceptually, all students in the highest performing schools moved up one sub-category.

• Conceptually, all students in the lowest performing schools moved down one sub-category.

Literacy Grade 4 Math Grade 4

Gain Points

Count Total Gains Count Total Gains

-2.5

-2.0 1 -2.0*1 = -2

-1.5 2 -1.5*2 = -3

-1.0 3 -1.0*3 = -3 2 -1.0*2 = -2

-0.5 4 -0.5*4 = -2 3 -0.5*3 = -1.5

0 5 0.0*5 = 0 5 0.0*5 = 0

0.5 3 0.5*3 = 1.5 4 0.5*4 = 2

1.0 3 1.0*3 = 3 3 1.0*3 = 3

1.5 2 1.5*2 = 3

2.0

Sum 21 -5.5 19 4.5

Sum of all total gains = -5.5 + 4.5 = -1.0 Sum of Counts = 21 + 19 = 40

School Improvement Gain Index = Sum of Total Points / Sum of Counts

= -1.0/40 = -0.025

Multiply the number of students (5) by the points assigned to the category. For example multiply (5) times (4) to get the points produced by the students in the

Advanced category. Add up the points for each category. Divide the total points for the school (285) by the number of students (100) to get a rating score (2.85),

which is meeting standards.Number of Students

Categories Points Assigned to Categories

Total

5 Below Basic 1

1 5

5 Below Basic 2

1.5 7.5

10 Basic 1 2 20

20 Basic 2 2.5 50

30 Proficient 1 3 90

20 Proficient 2 3.5 70

5 Advanced 1 4 20

5 Advanced 2 4.5 22.5

Total Points for the school for all categories 285

How School Ratings Are Calculated

Computation of Annual Improvement Gain Index

For each school, the Annual Improvement Gain Index is the average of all value-added points across subject areas and grades.

Potentially, the range of the Annual Improvement Gain Index is from -3.5 to +3.5, but in practice the range is much smaller.

The value of zero indicates no growth, positive values indicate improvement, and negative values indicate a decline in achievement.

Stakeholder Field Review of Gain Index

Procedures for Setting Cut Scores for Annual Improvement Category Levels (Gains)

Schools will be classified into one of five Annual Improvement Gain Levels. Therefore, four (4) cut scores will need to be set.

The steps are: (1) preliminary work by the

Technical Advisory Committee on Accountability, (2) advice by Arkansas stakeholders at a meeting, and (3) adoption by the Arkansas State Board of Education.

Preliminary Work by the Technical Advisory Committee on Accountability

 The distribution of the School Annual Improvement Gain Index was compiled based on the matched Spring 2006 and Spring 2007 data.

Improvement Gain index was computed only for schools with complete data on Literacy and Math for at least 40 students.

Preliminary Work by the Technical Advisory Committee on Accountability

 The distribution of the School Annual Improvement Gain Index was compiled based on the matched Spring 2006 and Spring 2007 data.

Improvement Gain index was computed only for schools with complete data on Literacy and Math for at least 40 students.

Schools with 40+ Students 2006/2007School Mean Change

N 798 Mean 0.0277 Median 0.0300 Mode 0.0900 Std Dev 0.1468 Skewness -0.0800 Min -0.512 Max 0.504

Histogram

# Box plot0.525+ * 2 0 . . * 3 0 . ** 6 | . ***** 13 | . ********** 28 | . **************** 48 | 0.175+********************** 65 | . ******************************* 91 +-----+ . ************************************** 112 | | . *********************************** 105 *--+--* . ********************************* 99 | | . ****************************** 90 +-----+ . ****************** 54 |-0.175+************* 37 | . ******** 23 | . **** 10 | . *** 7 | . * 1 0 . * 3 0 .-0.525+* 1 0 ----+----+----+----+----+----+----+---

Understanding the School Improvement Gain Index

 In 2007, the school improvement gain index ranges from approximately -0.50 to 0.50.

Conceptually, all students for a school at the lowest index of -0.50 move down one sub-category.

Conceptually, all students for a school at the highest index of 0.50 move up one sub-category.

Technical Characteristics of School Improvement Gain Index

  Reliability assessed via the Spearman-Brown formula = 0.922 Global standard error of measurement = 0. 041 Conditional standard error of measurement (Rulon Method) School Number of Mean Gain CSEM

Group Schools Index 1 79 -0.234 0.055

2 80 -0.123 0.038 3 80 -0.071 0.040 4 80 -0.030 0.048 5 80 0.012 0.043 6 80 0.050 0.036 7 80 0.087 0.032 8 80 0.123 0.035 9 80 0.181 0.037 10 79 0.282 0.039

TAC Rationale for Setting Preliminary Cut Scores

 

Four cut scores are needed. A cut score is the lowest mean change for a school in each school level.

Cut Score 1 for Level 1/Level 2: 25% of the students move down one sub-category and the other 75% remain in the same performance sub-category.

Cut Score 2 for Level 2/Level 3: all students remain in the sub-category where they were in the previous year. This cut reflects no growth on the school improvement scale.

Cut Score 3 for Level 3/Level 4: 25% of the students move up one sub-category and the other 75% remain in the same performance sub-category.

Cut Score 4 for Level 4/Level 5: 50% of the students move up one sub-category and the other 50% remain in the same performance sub-category.

TAC preliminary cut scores are: Cut score 1 = -0.12; Cut score 2 = 0.01; Cut Score 3 = +0.13; and Cut Score 4 = +0.25.

Creating Ordered School Booklet

 The global standard error of measurement is about 0.04.

The TAC preliminary lowest cut score is -0.12 and highest cut score is +0.25, with a range from -0.12 to +0.25.

In creating the Ordered School Booklet (OSB), a decision was made to consider to extend this range by two SEMs on both ends.

All schools with a mean rating between -0.21 to +0.32 were candidates from the OSB.

Schools were stratified using mean rating (rounded to 0.01).

One school was selected to represent each stratum.

There are 54 representative schools in the OSB.

Base Year School ImprovementGain Index Definitions

• In 2007, the school improvement gain index ranged from approximately -0.50 to 0.50.

• Conceptually, all students in the highest performing schools moved up one sub-category.

• Conceptually, all students in the lowest performing schools moved down one sub-category.

Data for School Profile

• Information provided:– Type of School– Grade range– 2007 enrollment– Racial composition– Mean of school improvement gain– Performance status in 2006 and 2007– Percentage of students on free or reduced price

lunch, with limited English proficiency, or in special education

– Percentage of mobile and migrant students

School Profile in Each Preliminary Level

• Level 1: “Schools in need of immediate improvement”

(116 schools)

• Level 2: “Schools on alert” (228)

• Level 3: “Schools meeting standards” (261)

• Level 4: “Schools exceeding standards” (140)

• Level 5: “Schools of excellence” (53)

Note: This preliminary level analysis includes high schools.

Initial Considerations for Preliminary Cut Scores: Conceptual

Framework

• Spread of mean changes by level:

– Level 1: -0.51 to -0.13– Level 2: -0.12 to 0.00– Level 3: 0.01 to 0.12– Level 4: 0.13 to 0.24– Level 5: 0.25 to 0.50

Panelists set cut points where they felt comfortable.

Composition of the Panel

• Facilitators: black female, black male, Hispanic male, white female, and white male

• Panelists also racially and geographically diverse: PTA, business, AAEA, AEA, ASBA

• Each group named 12 representatives, for a total of 60 Panelists (a small number were unable to participate)

• Monitored by TAC/Accountability

Procedures for Setting Cut Scores for Annual Improvement Category Levels

(Gains)

Schools will be classified into one of five Annual Improvement Gain Levels. Therefore, four (4) cut scores will need to be set.

The steps are: (1) preliminary work by the Technical Advisory Committee on Accountability, (2) advice by Arkansas stakeholders at a meeting, and (3) adoption by the Arkansas State Board of Education..

Advice by Arkansas Stakeholders

A meeting of Arkansas stakeholders was convened to recommend Annual Improvement Gain cut scores to the Arkansas Department of Education February 7, 2008 at the Crowne Plaza Hotel in Little Rock.

Arkansas PTA Louis Ferren

Arkansas Education Association Dr. Andre’ Guerrero

Arkansas Association of Educational Administrators

Willie Morris

Economics Arkansas Dr. Charity Smith

Arkansas School Boards Association Janine Riggs

Standard Setting RecommendationsStakeholder Advisory Panels

Cut Scores Cut 1/2 Cut 2/3 Cut 3/4 Cut 4/5

Administrators

-0.20 -0.01 0.10 0.20

Teachers -0.15 0.00 0.11 0.24

Business -0.12 0.01 0.13 0.25

Parents  -0.09 0.01 0.15 0.27

School Board -0.12 0.01 0.13 0.25

Median -0.12 0.01 0.13 0.25

Recommended Cut Scores by Arkansas Stakeholder Advisory Team

 Recommended cut scores are:

• Cut score 1 = -0.12

• Cut score 2 = 0.01

• Cut Score 3 = +0.13

• Cut Score 4 = +0.25

Note: Recommended cuts are the median scores from all five stakeholder groups.