beyond the numbers - harvard university · 850 hungerford drive | rockville, md 20850 | (301)...

850 Hungerford Drive | Rockville, MD 20850 | (301) 309-6277

Presentation for the SDP Spring Convening 2014

BEYOND THE NUMBERS Breakout Session 1: Analytics to Inform Early Warning

Indicators | The K‐12 Pipeline http://www.montgomeryschoolsmd.org/departments/sharedaccountability/

Geoff Sanderson Office of Shared Accountability

Montgomery County Public Schools

http://www.montgomeryschoolsmd.org/departments/sharedaccountability/



MCPS EWIS Background

• Previous OSA research examined secondary outcomes consistent with the broader area of study.

• In the fall of 2012 the Superintendent of Schools commissioned OSA to examine earlier grade levels.

• In the spring of 2013 that study was published and had investigated linkages between grades 1, 3, 6, 9, and final graduation status for two cohorts of students (Class of 2011 and Class of 2012; ~22,000+ students).

• Findings presented in July 2013 at the NCES Stats-DC Conference by the study’s lead researcher.

• Design phase for the EWIS commenced in August 2013.

• First prototypes created in November 2013 after Marking Period 1.

• Plan/build phase underway with goal of launch date in Fall 2013.



Early Warning Indicators (EWIs)

• OSA Publication: Just the Right Mix

• Analyzed MCPS data from the Class of 2011 and 2012

• Focused on attendance, behavior, and coursework patterns

• Compared completers versus non-completers



The ABCs and Student Engagement

• Proxy measures of student engagement

• Attendance (class absences vs. daily attendance)

• Behavior (in- and out-of-school suspensions)

• Course Performance (mathematics and English grades vs. grade point averages)



Summary of Key Findings Early Warning Indicator (EWI) Grade and Marking Period

Attendance Behavior Coursework

Grade 1 Marking Period 3

Students absent from school nine or more times are twice as likely to drop out of high school

Students suspended (in- or out-of-school) one or more times can be up to five times as likely to drop out of high school

(1) Students below grade level in reading and/or mathematics are twice as likely to drop out of high school; (2) Students having a calculated third marking period grade point average (GPA) below a 1.20 are twice as likely to drop out of high school


Students absent from school three or more times are twice as likely to drop out of high school

(1) Students suspended (in- or out-of-school) one or more times can be up to nine times as likely to drop out of high school; (2) Students receiving a ‘Needs Improvement’ on completing homework on-time are twice as likely to drop out of high school

(1) Students below grade level in reading and/or mathematics are twice as likely to drop out of high school; (2) Students having a calculated first marking period grade point average (GPA) below a 3.00 are twice as likely to drop out of high school



Summary of Key Findings (Continued) Early Warning Indicator (EWI) Grade and Marking Period

Attendance Behavior Coursework


Students absent from a class three or more times are twice as likely to drop out of high school

Students suspended (in- or out-of-school) one or more times are three times as likely to drop out of high school

(1) Students receiving a grade of ‘D’ or below in mathematics and/or English are one and a half times more likely to drop out of high school; (2) Students having a first marking period grade point average (GPA) below a 3.00 are at least five times as likely to drop out of high school


Students absent from a class three or more times are three times as likely to drop out high school

Students suspended (in- or out-of-school) one or more times are twice as likely to drop out of high school

(1) Students receiving a grade of ‘D’ or below in mathematics and/or English are at least three times as likely to drop out of high school; (2) Students having a first marking period grade point average (GPA) below a 3.00 are at least five times as likely to drop out of high school



EWI Factors at a Glance Factors in EWI Models for Kindergarten to Grade 12

Factors K–Grade 2 Grade 3–5 Grade 6–8 Grade 9 Grade 10–12

Academic

MAP-P MAP-M MPA MPA MPA

Instructional Reading

level

MAP-R MAP-R Grade 8 MAP-R PSAT/NMSQT Critical

Reading

MSA math Grade 8 MSA math PSAT/NMSQT Math

Behavioral

Tardiness Tardiness Tardiness Retention Retention

Absence Absence Absence Absence Absence

Suspension Suspension Suspension Suspension Suspension

Other

Unexpected entry &

withdrawal

Unexpected entry &

withdrawal

Unexpected entry &

withdrawal

Unexpected entry &

withdrawal

Unexpected entry &

withdrawal

New to MCPS New to MCPS

New to MCPS from

outside of U.S.

New to MCPS from

outside of U.S.

Business Rule

At least one

achievement factor

At least one

achievement factor

At least two

achievement factors

At least two

achievement factors

At least two

achievement factors

At least four out of

six factors

At least four out of

six factors

At least five out of

seven factors

At least seven out of

nine factors

At least seven out of

nine factors



BEYOND THE NUMBERS Breakout Session 1: Analytics to Inform Early Warning

Indicators | The K‐12 Pipeline http://www.montgomeryschoolsmd.org/departments/sharedaccountability/

Geoff Sanderson Office of Shared Accountability

Montgomery County Public Schools

http://www.montgomeryschoolsmd.org/departments/sharedaccountability/

Student

Monitoring and

Intervention

System

Taniesha A. Woods, PhD Say Yes to Education, Inc. April 23, 2014

1

Say Yes to Education

• A national, non-profit organization

• Helps students (and their families) in low-income communities achieve high school graduation and readiness for postsecondary programming

• Provision of postsecondary scholarships ensures that students are ready and succeed in affording, attaining, and completing an educational experience beyond high school

2

27 Years of Experience Leading to

two Say Yes Citywide

Implementations

3

Cohort Strategy: Philadelphia, Hartford, Cambridge, and Harlem

Citywide Turnaround Strategy: Syracuse and Buffalo

Factors That Say Yes Considers

Crucial to Student Success

4


Student Monitoring and Intervention

System (SMIS)

• New way of doing business -- we focus on supporting students’ academic achievement, social, emotional, and health outcomes.

• A research-based Student Monitoring and Intervention System developed by the American Institutes for Research, a key partner

• Continually updated profile for each student on 13 key indicators in domains of academic, social, emotional, mental, and physical health

• Based on SMIS growth plan, students and families can enroll in intervention(s) before low-achievement outcomes occur

5

SMIS Indicators

• Physical Well-Being

• Social Capital

• Cultural Capital

• Social-Emotional Learning

• Behavior

• Citizenship

• Peers

• Learning Supports

• English Language Arts, Literacy

• Mathematics, Numeracy

• Attendance

• Grade Retention

• College Eligibility

6


Student Monitoring and Intervention System

(SMIS) Inputs

MONITORING AND INTERVENTION SYSTEM

*Research Based Indicators that track to post

secondary readiness

*AIR developed algorithms

*Technology based interactive system

School District data

Attendance, grades, discipline,

test scores

Parent surveys

Teachers surveys Student surveys

Other Provider

input

7

MONITORING AND INTERVENTION SYSTEM

*Research Based Indicators that track to post secondary

readiness

*AIR developed algorithms

*Technology based interactive system

Individual Student

Growth Plans including

interventions

Reports on the overall effectiveness of the provider and of the specific

interventions provided

Aggregate reports that support

planning at the class, grade,

school, and district level

8


Student Monitoring and Intervention System

Outputs

Student Growth Plans

• Includes information about whether students on are on-track for high school graduation and postsecondary matriculation

• Displays students’ status (on-track, on-track to thrive, and off-track) for the 13 indicators

• Students who are off-track are referred for services by the student support team (SST). Services include:

– School-day academic supports

– Extended day and year

– Counseling

– College preparation

– Family, health, and legal services

• Aggregate reports also available: Class, grade, school, and district

Site Facilitator and Student Support

Team (SST)

• Say Yes Site Facilitators work in schools in a case management type role as part of the Student Support Team (e.g., psychologist, guidance counselor, social worker)

• SSTs use Student Growth Plans to identify students who are off track and the appropriate intervention(s) for their needs

• Using SMIS data it is possible to look at the relation between a student’s participation in the intervention, the dosage and quality of the intervention, and achievement outcomes

• SSTs are provided with on-going professional development

10

Demographics, Buffalo Public

Schools

2009-10 2010-11 2011-12

Enrollment PreK - 12 35,039 34,191 33,556

Free-Reduced Price Lunch 77% 79% 77%

English Language Learners 9% 10% 11%

American Indian 1% 1% 1%

Black/African American 56% 55% 53%

Hispanic/Latino 15% 15% 16%

Asian/Pacific Islander 4% 5% 6%

White 23% 23% 22%

Annual Attendance Rate 86% 86% Pending release

Suspensions 20% 18% Pending release

Source: New York State Education Department

High School Graduation and Dropout

Rates, Buffalo Public Schools

50%

55%

48%

32%

27% 29%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2009-10 2010-11 2011-12

4-year Graduation Rate

Drop-out Rate

Source: New York State Education Department

Lessons Learned and Next Steps

1. Response rate for family surveys

– Students: 81.25%

– Faculty: 96.35%

– Parents/Families: 18%

2. Technology

– 3rd party IT vendor

– Pulling data from multiple sources

– Translating 6 languages for the electronic surveys

3. Timing of Survey Administration

– Faculty’s knowledge of students

4. Building capacity for the SMIS to interface with postsecondary data bases

5. Establishing a shared understanding of the purpose of SMIS

13

Early Results - Buffalo Public Schools,

Postsecondary Matriculation Rates the Fall

Immediately after High School

14

21%

25% 27%

25% 28%

25%

31%

38% 35%

33% 30%

34% 32%

35%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2007 2008 2009 2010 2011 2012 2013

2-year Institutions

4-year Institutions

Source: National Student Clearinghouse

THANK YOU!

15

beyond the numbers - harvard university · 850 hungerford drive | rockville, md 20850 | (301)...

Documents