Proposed Accountability Index

Pete Bylsma, EdD, MPAConsultant to the State Board of Education

February 25, 2009

http://www.sbe.wa.gov/spa.htmbylsmapj@comcast.net

1

Overview of Presentation• Rationale for creating a new system• Guiding principles• AYP vs. new system• Process used to develop the proposed system• Proposed indicators and outcomes• Ratings and tier assignments• School and district results for 2007• Handling special cases• Recognition categories and criteria• Identifying “Priority” schools and districts• Next steps

2

Why a state accountability system?• Legislative mandate for the SBE to “adopt objective,

systematic criteria” for successful schools and districts and for those in need of assistance

• Accountability is the “3rd leg” of a comprehensive state education system after establishing standards and assessments

• Federal accountability system (AYP) is viewed as unfair, too complicated, and punitive

• State provides most of the education funding• Supports legislative efforts to revise funding system

3

Guiding Principles1. Be fair, reasonable, and consistent

2. Be transparent and simple to understand

3. Be valid and accurate

4. Use existing data

5. Rely on multiple measures

6. Include state assessment results from all grades and subjects

7. Apply to as many schools & districts as possible

8. Provide multiple ways to show success and earn recognition

9. Rely mainly on criterion-referenced measures

10.Use familiar concepts

11.Use concepts of AYP when appropriate

12.Be flexible enough to accommodate future changes4

Role of AYP• Widespread belief that AYP is overly complex, less

inclusive, too narrow, not transparent, and not valid for identifying success or most in need

• Recommend using disaggregated results to help determine which schools and districts need the most help (“Priority” designation)

• Will submit proposal to US Ed. Dept. to use this system to determine AYP

5

Advantages of Proposed System• More valid

Uses performance of all students in more subjects, more differentiated than a “Yes/No” system, does not count students multiple times, looks at improvement and how the outcomes compare with similar schools

• More inclusive/comprehensive Uses a smaller minimum N (10 students across the entire school/district), includes the results of all students (regardless of how long they have been enrolled), includes both writing and science (this helps prevent a narrowing of the curriculum), uses Learning Index to measures performance across the range of assessment results (reduces the focus on bubble kids)

• Less volatile over timeAssessment results are combined across all grades (not individual grades) so the N is larger and the students being assessed do not change much from year to year

• More transparent Does not include a margin of error, benchmarks are the same over time and among the different subjects, fewer subgroups and rules, schools and districts are evaluated using the same criteria, N is the same for all groups and both schools and districts

The results will provide recognition and will identify fewer schools and districts in the lowest tier because improvement will occur

6

Process to Date• Initial proposal reviewed by an advisory panel• Convened diverse set of advisors to provide input on

technical details for a revised proposal• Applied proposed rules to data to ensure validity, made

adjustments to increase precision• Ongoing feedback received from OSPI and others• Presentations and feedback at SBE meetings and ESDs• Further refinements taking place

7

Proposed Outcomes/Indicators

Five OutcomesResults from 4 assessments (reading, writing, math, science)

aggregated together from all gradesExtended graduation rate for all students

Four Indicators1. Achievement by non-low income students (% meeting

standard/ext. grad rate)

2. Achievement by low income students (eligible for FRL)

3. Achievement vs. Peers (Learning Index and ext. grad rate controlling for ELL, low-income, special ed., gifted, mobility)

4. Improvement (change in Learning Index from previous year)

Creates a 5x4 matrix with 20 outcomes8

Outcome/Indicator Matrix

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

Low inc. ach.

Ach. vs. peers

Improvement

Average Index

9

Outcome/Indicator Benchmarks & Ratings

10

Indicator Reading Writing Math Science Ext. grad rate

Achievement of- Non-low inc.- Low income (% met standard)

% MET STANDARD RATING90 – 100% 780 – 89.9% 670 – 79.9% 560 – 69.9% 450 – 59.9% 340 – 49.9% 2< 40% 1

RATERATING> 95 790 – 95% 685 – 89.9% 580 – 84.9% 475 – 79.9% 370 – 75% 2< 70% 1

- Achievement vs. Peers(Learning Index)

DIFFERENCE IN LEARNING INDEX RATING> .20 7.151 to .20 6.051 to .15 5-.05 to .05 4-.051 to -.15 3-.151 to -.20 2 < -.20 1

DIFFERENCEIN RATE RATING> 12 76.1 to 12 63.1 to 6 5-3 to 3 4-3.1 to -6 3-6.1 to -12 2< -12 1

Outcome/Indicator Benchmarks & Ratings

11

Indicator Reading Writing Math Science Ext. grad rate

- Improvement(Learning Index)

CHANGE IN LEARNING INDEX RATING> .15 7.101 to .15 6.051 to .10 5-.05 to .05 4-.051 to -.10 3-.101 to -.15 2< -.15 1

CHANGEIN RATE RATING> 6 74.1 to 6 62.1 to 4 5-2 to 2 4-2.1 to -4 3-4.1 to -6 2< -6 1

Achievement vs. Peers• Considers contextual differences• Makes “apples to apples” comparisons by controlling for

level of ELL, low-income, special education, mobile, & gifted students (+ funding level in district analysis)

• Currently no data generated on this type of indicator• Others have done this type of analysis

– Seattle Times school guides

– Simple scatterplots with trend lines (only one variable at a time)

– Other states using this concept in accountability systems

• Complexity presents a communications challenge

12

Linear Regression

0.0 25.0 50.0 75.0 100.0

Pct low income

0.000

1.000

2.000

3.000

4.000

Mat

h L

earn

ing

Inde

x, 2

007

Math Learning Index, 2007 = 3.26 + -0.01 * PctLowIncR-Square = 0.70

A

B

Illustration of Ach. vs. Peers

13

7

1

4

Tier Index Range

Exemplary 5.50 – 7.00

Very Good 5.00 – 5.49

Good 4.00 – 4.99

Acceptable 2.50 – 3.99

Struggling 1.00 – 2.49

Priority (eligible for Innovation Zone)

1.00 – 2.49

14

Tier Names and RangesAssigned to a “tier” based on index score

School and District Results (2007)Tier Index

RangePct. of schools

Pct. of districts

Exemplary 5.50 – 7.00 4.0% .3%

Very Good 5.00 – 5.49 6.5% 3.1%

Good 4.00 – 4.99 29.4% 29.9%

Acceptable 2.50 – 3.99 48.7% 60.8%

Struggling 1.00 – 2.49 11.3% 5.8%

Priority (eligible for Innovation Zone)

1.00 – 2.49 TBD TBD

228 schools were in the Struggling tier with 74,000 students (1 in every 14 students); 98 were alternative schools or served other special populations.Over 2-year period, 7.5% of all schools averaged < 2.50 (Struggling tier)

17 districts were in the struggling tier in 2007 (average 1,000 students) 15

Key Changes Made or Proposed• Slow down implementation schedule

• Avoid double-counting students by creating separate ratings for low income and non-low income students

• Move to a 7-point scale, add another tier

• Exclude results for ELL students in first 3 years or until reaching Level 3 on WLPT (whichever comes 1st)

• Think of other ways to hold alternative school accountable• Exclude improvement indicator for those with very high

performance• Simplify recognition system

16

Indicator Reading Writing Math ScienceGrad Rate Average

Non-low inc. ach. 6 6 6 4 5.50Low-inc. ach. 5 4 3 1 3.25Ach. vs. peers 7 7 7 7 7.00Improvement 7 7 6 4 6.00Average 6.25 6.00 5.50 4.00 5.44

Example Results(Elementary School, 2008)

17

Showing Results Over Time

(Elementary school)

18

Indicator/Outcome 2006 2007 2008 Non-low inc. ach. 4.25 5.25 5.50 Reading 6 6 6 Writing 3 5 6 Math 6 6 6 Science 2 4 4 Extended grad. rate       Low-income ach. 3.00 2.25 3.25 Reading 5 4 5 Writing 3 2 4 Math 3 2 3 Science 1 1 1 Extended grad. rate       Ach. vs. peers 5.50 5.25 7.00 Reading 7 5 7 Writing 2 3 7 Math 7 6 7 Science 6 7 7 Extended grad. rate       Improvement 2.00 3.00 6.00 Reading 4 1 7 Writing 1 3 7 Math 2 1 6 Science 1 7 4 Extended grad. rate      

INDEX 3.69 3.94 5.44

Graphing Results Over Time

19

1 .0 0

2 .0 0

3 .0 0

4 .0 0

5 .0 0

6 .0 0

7 .0 0

No n -lo w in c o m e

A c h ie ve m e n t

L o w -in c o m e A c h ie ve m e n t

A c h . vs. P e e rs Im p ro ve m e n t A c c o u n tab ility In d e x

2 0 0 6 2 0 0 7 2 0 0 8

District Accountability

• Uses the same indicators, outcomes, benchmarks, ratings, recognition system as schools

• Combines all the grades together(no separate results for each grade band)

• N is 10 for entire district (very few left out)

• Control for funding level for “peer” analysis

20

0

2

4

6

8

1 0

1 2

Ele m e n tary M id d le /Jr H i H igh K -1 2

Num

ber o

f sch

ools Ex e m p lary

Ve ry Go o d

Go o d

A c c e p tab le

Stru gglin g

21

Showing Results for All Schools(Average of 2007 and 2008)

0

5 0 0

1 0 0 0

1 5 0 0

2 0 0 0

2 5 0 0

3 0 0 0

3 5 0 0

4 0 0 0

Ele m e n tary M id d le /Jr H i H igh K -1 2

Num

ber o

f stu

dent

s Ex e m p lary

Ve ry Go o d

Go o d

A c c e p tab le

Stru gglin g

22

Showing Results for All Schools(Average of 2007 and 2008)

23

Showing Results for All Schools

1 .0 0 2 .0 0 3 .0 0 4 .0 0 5 .0 0 6 .0 0 7 .0 0

A K -1 2

K Ele m e n taryJ Ele m e n taryI Ele m e n tary

H Ele m e n taryG Ele m e n taryF Ele m e n taryE Ele m e n taryD Ele m e n taryC Ele m e n taryB Ele m e n taryA Ele m e n tary

D M id d leC M id d leB M id d leA M id d le

C H igh Sc h o o lB H igh Sc h o o lA H igh Sc h o o l

2 -YEA R A VERA GE IND EX (2 0 0 7 & 2 0 0 8 )

24

Propose exempting ELL results in the first 3 years of enrollment or until advanced proficiency achieved on the WLPT (whichever comes first)

• Results currently count in AYP in 2nd year of enrollment

• Research found it usually takes at least 3 years to achieve “academic” proficiency in English

• OSPI requested this policy but was denied; WA could still use this policy when calculating the index

• ELLs would still take the test in their 2nd year, WLPT results would be made public to increase accountability

• Would not affect many students (most ELLs in tested grades have been in US for 3 years or have achieved intermediate proficiency); would have a small positive impact on index where there are many ELLs present

Special Case #1 – ELL Exemption

25

Propose using other means for holding alternative schools accountable

• Many different types exist in the state, so no “peer” indicator computed for schools with this designation

• Represent < 4% of enrollment but many serve students facing significant challenges, generating low index score (More than half the “schools” serve at-risk students in 9-12)

• Many need to be held accountable through more than index

Two Options Being Considered

1: Use regular process, use in-depth analysis to determine if school is using best practices, showing progress, and their role/status/resource level within the district

2: Allow schools serving high-risk/special populations to use additional measures to determine their tier (e.g., credits earned, attendance, gains on pre-post tests)

Special Case #2 – Alternative Schools

26

Propose excluding improvement indicator when reaching very high achievement levels

• Improvement is difficult when achievement is very highCannot receive the highest rating (7) after the Learning Index

reaches 3.85/4.00 and when the graduation rate reaches 94%

• Would go into effect when achievement is in the top range two years in a row, making it impossible to receive the maximum rating in the second year

• No school or district has yet to met these criteria for the assessments; 11% met these criteria for graduation rate

• School/district could decide to include the results

Special Case #3Improvement by High Performers

Recognition System• Guiding principles

– Multiple ways to demonstrate success, earn recognition

– Criterion-based system

Theory of Change: People are motivated more by successthan guilt or blame; need clear, challenging, attainable goals

• Use same accountability matrix, receive recognition when meeting specific benchmarks

• Based on 2-year average

27

Suggested Cells and Criteria

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

5.50*Low inc. ach.

Ach. vs. peers

Improvement

Average 5.00*

28

Recognize results in 20 cells + Index (21 total)

* Minimum 2-year average rating to earn recognition

Complement Existing Recognition

Federal recognition (competitive)– Blue Ribbon Schools– Academic Achievement Award– Distinguished Schools Award– Title I Improvement Award

OSPI recognition– Schools of Distinction (top 5%)– Improvement Award (schools and districts)

No recognition exists yet based on meeting achievement levels or “beating the odds”

29

Current Recognition

Outcomes

Indicator Reading Writing Math Science Ext. G.R. Avg.

Non-low inc. achievement

Low inc. ach.

Ach. vs. peers

Improvement

Average Index

30

Note: Improvement is measured in different ways.

Identifying Priority Schools

• Those in “Struggling” tier undergo deeper analysis to determine which need more help

• Many issues to examine:– Contextual issues– In-depth analysis of disaggregated WASL/WAAS– AYP results– Other data

• Identification process similar to that used now by OSPI

31

Issues to be Resolved

• Determine how a school/district “makes AYP”

• How to include other outcome measures for HS

• Continue receiving input from stakeholders- Includes review by OSPI’s external panel of advisors

• Collaborate with OSPI- Develop proposal to US Education Dept. to use the index for

federal accountability purposes- Create options for showing results on Report Card- Determine how index results relate to recognition/state

assistance

• Identify ways to show “reciprocal” accountability

• Timing of implementation32