What is program success?

Wendy Tackett, Ph.D., Evaluator

Valerie L. Mills, Project Director

Adele Sobania, STEMOakland Schools MSP, Michigan

Mathematics Education Resource Center

MSP Project Goals1. Provide high quality professional

development to teachers in math or

science

2. Positively impact student academic

achievement in math or science

MSP Program Logic

Oakland Schools

Variables to Consider

• Teacher attendance & participation in:– Coaching (lesson planning & reflection)– Mathematics institutes/courses– Mathematics instructional materials

workshops– Formative assessment workshops (data

analysis & planning)– Building based professional development

• Teacher attitudes about project

Variables to Consider

• Principal participation in:– Lenses on Learning program– Supporting teachers– Building professional development

• Building climate– Perceived collegiality– Teacher turnover– Administrative turnover– Support for time out of the classroom

Variables to Consider

• Student– Level of participation of teachers – Years in classrooms of teachers participating in

project

• Knowledge– Teacher math content knowledge– Teacher pedagogical content knowledge– Teacher instructional knowledge– Student math knowledge– Principal math knowledge

Unintended variables

• STEM– Learnings from developing professional

development for teachers– Learnings from teaching professional

development to teachers

• Coaches– Professional growth of coaches– Impact on the rest of the district if coaches

are employed full-time by district

Primary Measures• Teacher content & pedagogical knowledge

– Learning Mathematics for Teaching (LMT)– sitemaker.umich.edu/lmt

• Teacher instructional knowledge– Science and Mathematics Program Improvement (SAMPI)

lesson observations– www.wmich.edu/sampi

• Principal knowledge– Lenses on Learning (LOL)– http://mi.gov/mde/0,1607,7-140-22709_31168---,00.html

• Student knowledge– MEAP (state assessment)– Oakland quarterly assessments (locally developed)

• Other– Success Case Method teacher interviews– Climate survey – Lesson planning questionnaire

Comparison Group

• Identified non-participating school district with similar demographics as 5 districts in Oakland Schools

• Agreement with comparison district:– Test comparison teachers at beginning, middle, & end

of project– Collect MEAP information annually– Use Oakland quarterly assessment to test students in

5th grade in fall 2006 & 6th grade in spring 2008– Pay teachers $200 per testing period & pay district for

collecting student achievement data

Treatment Group

• 5 districts in Oakland Schools– All 3rd-8th grade math teachers participate in a building that

is participating

• Agreement with treatment district:– Test treatment teachers at beginning, middle, & end of

project– Observe teachers formally at beginning, middle, & end of

project– Collect MEAP information annually– Use Oakland quarterly assessment to test students in 5th

grade in fall 2006 & 6th grade in spring 2008– Pay teachers $200 per testing period

Results: SAMPI

• Statistically significant improvements in: – Overall rating for classroom instruction– Subcategory ratings of lesson implementation,

lesson content, & classroom culture– Substantive student-student interaction– Appropriate pacing for all student levels– Adequate reflection time– Differences between teachers using standards-

based math curricula vs. traditional curricula

Using Data Formatively

• Differences between teachers using standards-based math curricula vs. traditional curricula– MERC staff shared results with partner districts– Those districts using traditional curricula have all

now switched to either Everyday Math or the Connected Mathematics Project

Results: SAMPI & Climate

• 2006 Spring MSP conference in Boston– Iris Weiss discussed impact of building climate on

student learning– We developed survey that addressed five

elements of building climate

• Found statistically significant differences (p<.05) in behaviors observed through SAMPI observations in the classroom between teachers in buildings with high climate ratings (3.0 or higher) and the rest of classrooms

Results: SAMPI & Climate

IMPLEMENTATION Positive Climate

Others

Periods of teacher-student interaction were probing and substantive. 4.63 3.61

Classroom management was effective in engaging students in the lesson. 4.93 4.09

Periods of student-student interaction were focused on pertinent lesson content and enhanced individual understanding of it.

3.83 3.00

Overall rating for Implementation of Lesson 4.43 3.70 CONTENT Students were intellectually engaged with important ideas related to the focus of the lesson. 4.67 3.76

The lesson included connections between concepts/content in this lesson and/or previous or future lessons in the overall unit or topic being addressed.

4.57 3.21

The lesson included abstractions as appropriate. 3.83 2.58 CLASSROOM CULTURE Active participation of all students was encouraged and valued. 5.67 4.61

The teacher showed respect for and valued studentsÕ ideas, questions, and/or contributions to the lesson. 5.59 4.36

Student-student interactions reflected collaborative working relationships. 3.79 2.76

Overall rating for Classroom Culture 4.93 4.06

Using Data Formatively

• Statistically significant differences in behaviors observed through SAMPI observations in the classroom between teachers in buildings with high climate ratings and the rest of classrooms– Coaches started focusing more on creating a sense of

collegiality among teachers through building based professional development

– MERC project director met with all building principals, curriculum coordinators, and superintendents to more clearly explain the MERC project, potential benefits, and current evaluation findings

Results: LMT

• Attendance at three or more content-focused or curriculum-based courses had a statistically significant positive impact on the pedagogical content knowledge of teachers

LMT

Attend 3 or more courses 69%

Attend 1-2 courses 54%

Comparison group 48%

Using Data Formatively

• Attendance at three or more content-focused or curriculum-based courses had a statistically significant positive impact on the pedagogical content knowledge of teachers– Offered courses at various times to allow flexibility

for teachers who couldn’t previously take them– More concerted efforts to encourage all

participants to take at least 3 courses

– Developed a fifth content course based on areas of persistently low performance

Results: Coach Observations

• Teachers are exhibiting deeper understandings of math - leading to more math talk & student discourse

• Teachers strongly support the professional development sessions

• Teachers report greater instructional focus and alignment to state standards

• Teacher concerns moved from logistics to student learning

Using Data Formatively

• Teachers are exhibiting deeper understandings of math - leading to more math talk & student discourse– Coach observations helped shape future sessions

in the Mathematics Institutes and building based professional development

– Coaches discuss observations at monthly meetings, sharing ideas, and creating synergies between the coaches

– Revised Lesson Planning Protocol based on SAMPI and Success Case interview results

MSP Program Logic

Oakland Schools

Our Two Cents

• Evaluation isn’t just to be done to satisfy grant reporting - more critically it should be used for program improvement

• Look strategically at the data you’re collecting to show impact along the way…if your logic model is true, it will lead to improved student academic achievement

• Continue to ask questions and be prepared to adjust your evaluation collection along the way

For More Information

• Wendy Tackett, Ph.D., iEval– www.ieval.net– ieval@mac.com

• Valerie Mills, Oakland Schools– valerie.mills@oakland.k12.mi.us– http://www.oakland.k12.mi.us/

Departments/LearningServices/MathematicsEducationResourceCenter/tabid/287/Default.aspx

For a PDF of this presentation:

Go to www.ieval.netclick on “presentations”The instructional practices and assessments discussed or

shown in this presentation are not intended as an endorsement by the U.S. Department of Education.