next-generation accountability

NEXT-GENERATION ACCOUNTABILITYDesigning a Differentiated Accountability System for Michigan

Presentation to the Michigan Educational Research AssociationNovember 22, 2011

The Challenge• Design a school accountability system that:

• Sets a high proficiency standard (where proficiency is based on career and college ready standards) AND

• Rewards schools for achieving growth with students, regardless of starting point

• Moves Michigan toward a higher level of preparation for career and college

• Fair and equitably applied

WHERE ARE WE NOW?Assessing Michigan’s Current Situation

College going rates• Statewide:

• 71% of 2008-2009 graduates enrolled in an IHE• 73% of those who enroll earn at least one year’s worth

of credits• Gives a total of 52% of 2008-09 graduates who earned

at least one year’s worth of credits• By individual school:

• Median = 63%• 25th percentile: 40%• 75th percentile: 75%

0.5

11.

52

2.5

Den

sity

0 .2 .4 .6 .8 1

Percentage of Graduates Enrolled at an IHE

Relationship between new cut scores and college going0

2040

6080

100

Per

cent

Enr

olle

d in

an

IHE

0 20 40 60 80 100

2008 MME Math Percent Proficient (based on new cut scores)

Takeaways• Michigan students are going to college• Even if students are not proficient on new cut scores on the MME, they are enrolling in college.

Question: Will those students be successful? Will they pursue challenging majors?

Achievement Gap• Since 2001, schools have been held accountable on overall student performance… AND the performance of the nine traditional subgroups

• Put the focus on achievement of all students, as defined by demographic characteristics

• Caveat: IF you had a sufficient number of students!

Economically disadvantaged gaps: Math

Ethnicity gaps: Math

Economically disadvantaged gaps: Reading

Ethnicity gaps: Reading

Career and College Readiness in Our Schools: Math

0.0

1.0

2.0

3.0

4D

ensi

ty

0 20 40 60 80 100mmemath2011

50th percentile 95th percentile

CCR: Reading0

.01

.02

.03

Den

sity

0 20 40 60 80 100mmeread2011

50th percentile 95th percentile

Tension: Is it really important that our students be career and college ready?• Evidence points to the importance of higher education.

• Competitive job market, especially in states like Michigan.

• Fast-growing occupations (health care, technology) require higher education/specific training

• Educational inflation

NEXT GENERATION ACCOUNTABILITYCharting a New Path

Necessary Components• Focused consequences and interventions for schools most in need

• Achievement gap• Differentiated accountability = differentiated interventions

• Fair and equitable• Ambitious AND achievable goals

Focused Consequences and Interventions

• Priority Schools• Lowest 5% of the Top to Bottom list• Priority schools = PLA Schools• Aligns federal and state accountability• Priority schools must enter a three year cycle of school

improvement, with the most highly targeted interventions

PLA Schools: Anecdotal Evidence• Two “cohorts” of PLA schools: 2010 and 2011.• 2010 schools: first year of implementation• 2011 schools: planning• 2010 schools:

• About half experienced an increase in percent proficient and increased their improvement rate

• More meaningful data in another 1-2 years

PLA Schools: Anecdotal Evidence• What are they saying?

• ERA Unit doing PLA data visits• Being named a PLA school was extremely difficult• However, it has fundamentally altered the way the

schools are approaching achievement• Pushing a fundamental redesign• Impetus to address crucial issues• Innovative strategies (i.e. flipped learning)

Achievement Gap as Central Focus• Achievement gaps have not closed to the extent that we need

• Proposal:• Focus on the bottom 30% of students, regardless of

demographic, not the traditional subgroups• Puts the attention firmly on the lowest achieving

students• By improving that group, increase school’s overall

achievement, and improvement rate

Achievement Gap• Pros

• All schools have a subgroup• At least 700 schools have no subgroup under AYP

traditional subgroups• Unmask low performance in high performing subgroups• Asks that all schools consider their lowest performing

students• Schools cannot mask low-performance with overall high

performance

Achievement Gap• Cons:

• Concern that we will lose focus on demographic subgroups.

• In the lowest 30% subgroup—approximately 70% of that group are also a member of one or more traditional subgroups.

• High-achieving schools do not like it• People think that “lowering the ceiling on our highest

achieving students” will help the schools

Achievement Gap: Focus Schools• Need to identify the schools with the largest achievement gaps.

• Using the bottom 30% subgroup, would rank the bottom 10% in terms of largest gap.

• Using traditional subgroups in a ranking (normative) setting is complicated:• Not all schools have a subgroup• Comparing schools with the same subgroup• Unfairly focuses on students with disabilities• Still allows for “masking”

Achievement Gap: Our Belief• If Michigan is serious about raising the achievement of ALL students, then the bottom 30% is the correct way to go.

• Distributing accountability to traditionally high achieving schools and asking them to achieve those same results with all students is appropriate.

• Michigan cannot leave students behind any longer.

Differentiated Accountability For All Schools

• Priority schools = 5% and Focus Schools = 10%--so what about the other 85%?

• Need a more nuanced system than pass/fail AYP.• Need to integrate performance for all students, bottom 30%, and all subjects (not just reading and mathematics)

Use this system to set a proficiency goal with improvement• Proficiency target = AMO

• Set for each school as the increase in percent proficient necessary for that school to reach the overall target proficiency

• Improvement target• If school does not meet proficiency target, can meet an

improvement target• Set as the increase in percent proficient demonstrated

by a high-improvement school in the base year

Example with Data• End proficiency target: 85% • School is at 10% proficient now• Need to improve 75% in 10 years, or 7.5% per year.

• Proficiency target in year 1: 17.5% proficient• If does not meet it, must have improved by 3.5% (which is improvement rate for school at the 90th percentile in base year)

Bottom 30% Subgroup as Accountable Subgroup

• Only one “accountable” subgroup now (still report on nine traditional subgroups)

• Need to meet a proficiency target for the bottom 30% subgroup (unlikely…) OR the improvement target

Notes on this system• Offers differentiated AMOs by school• Keeps a clear proficiency target in the system• Proficiency target is actually an improvement target as well

• Many of the increases demanded of schools will be greater than we have historically seen, so need the improvement (safe harbor) target

Questions• What is an “ambitious and attainable” end goal? 100%? 85%? 70%? How do you determine this?

• Should meeting the target based on improvement be equivalent to meeting it based on straight proficiency?

• Should we reset each year?

Participation• Necessary to keep a firm participation target in the system

• Schools will begin to “game” on who they assess if no clear participation target

“Green” SchoolSchool Name: ABC Schools

 Reward

   Proficienc

y Improvement Participation Overall Math All Students Yes -- Yes

   Bottom 30% No Yes  Reading All Students Yes -- Yes

   Bottom 30% Yes --  Writing All Students Yes -- Yes

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No Yes  Social Studies All Students Yes -- Yes

   Bottom 30% Yes --      Rate Improvement OverallGraduation Rate   Yes --              OverallCompliance          Educator Evaluations          

“Yellow” SchoolSchool Name: XYZ Schools

   Proficienc

y Improvement Participation Overall Math All Students No Yes Yes

   Bottom 30% No Yes  Reading All Students Yes -- Yes

   Bottom 30% Yes --  Writing All Students No Yes Yes

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No Yes  Social Studies All Students Yes -- Yes

   Bottom 30% Yes --      Rate Improvement OverallGraduation Rate   Yes --              OverallCompliance          Educator Evaluations          

“Red” SchoolSchool Name: MNO Schools

 Priority

    Proficiency Improvement Participation Overall Math All Students No Yes Yes

   Bottom 30% No No  Reading All Students No No Yes

   Bottom 30% No No  Writing All Students No Yes No

   Bottom 30% No Yes  Science All Students Yes -- Yes

   Bottom 30% No No  Social Studies All Students No No Yes

   Bottom 30% No No      Rate Improvement OverallGraduation Rate   No Yes              OverallCompliance          Educator Evaluations          

Rules for Colors• Need to be green on all indicators• This makes “green” a more rare indicator; it means that there are no areas of concern

• To be red, need to be red on all five academic indicators; makes it a more rare indicator

• Yellow—largest category—can have some red, some green; is indicative of “intervention” needed; use colors within to target

• Final color is not the key determiner for consequences; priority/focus status is more critical

Questions?• How to determine the final colors?• Balancing public desire for “one” rating with internal knowledge that “one” rating is difficult.

• Other indicators that should be included?

Contact Information• Venessa A. Keesler, Ph.D.

• Evaluation, Research and Accountability• Office of Psychometrics, Accountability, Research and

Evaluation (OPARE)• keeslerv@michigan.gov