cera 87 th annual conference- effective teaching & learning: evaluating instructional practices...

CERA 87th Annual Conference- Effective Teaching & Learning: Evaluating Instructional Practices

Rancho Mirage, CA – December 4, 2008

Noelle C. Griffin, PhD.

Implementation of a Formative Assessment-based Middle School Math Intervention: Findings, Considerations and Lessons Learned

CRESST/UCLA

Victoria Schumacher, PhD. Norwalk/ La Mirada Unified School District

• NLMUSD Background and Considerations

• POWERSOURCE© Overview and Implementation

• POWERSOURCE© Findings and Trends

• Broader Math Achievement Context

Presentation Overview

NLMUSD: Background and Considerations

• Priority 1! 9 out of 10 students reading at grade level.

• Math: initial focus on middle school

• Overarching district framework: “The Work”

Instructional Context

• Authentic student engagement

• Effective questioning

• Making meaning/text analysis

• Assessment driven instruction

• Standards-based instruction across content areas

Components of the Work

Comparison of Math CST 2008 Results across Schools with Free or Reduced Lunch and English Learners

Using CRESST Collaboration to Support “The Work”

• Initial conversations/planning re: implementation and experimental design (POWERSOURCE and “alternative groups”)

• Coordination with other district math partners (Dr. Marty Bonsangue, CSUFullerton)

• Key consideration: How can POWERSOURCE© project compliment existing district initiatives?

District Stakeholder Involvement

• Key role of stakeholders in decision to participate

• Joint CRESST/NLMUSD presentations/question answer sessions to middle school principals/teachers

• Involvement of Teachers’ Association (TANLA)

• Inclusion of CRESST Senior Researcher (Terry Vendlinski) in other district math professional development activities and sessions

• Teachers invited to participate, vote for “site coordinators” for each school

POWERSOURCE© : Overview and

Implementation

POWERSOURCE© Background

• Five-year research project

• Objective: Develop formative assessments and associated instructional support materials to improve student performance in Algebra/pre-Algebra

• Funder: U.S. Department of Education, Institute of Education Sciences (National Assessment Center Grant)

• Emphasis on empirical rigor and experimental testing

• Involves participation of multiple districts over multiple years

POWERSOURCE© Foundations: Formative Assessment

• Formative assessments can be used to guide teaching and instruction

• When used correctly, it can produce increased learning and achievement. (e.g., Black & William, CRESST)

• Many teachers don’t have the time, resources, or knowledge to use formative assessments effectively. (e.g., Herman)

POWERSOURCE© Foundations: Cognition

• Expert knowledge is schema-based, organized around complex, interconnected concepts

• Expert knowledge organization supports problem solving, inferences, transfer

• Experts provide the basis for material design, scoring, and formative assessment process

• Application of what we know about experts to support content area expertise

POWERSOURCE© Foundations: “Big Ideas”

• Fundamental principles that underlie Algebra operations, when learned, will enhance the speed and quality of learning

• Context of expert knowledge/ cognition

• Identified/vetted through participation of math experts (theory and instruction)

• POWERSOURCE Focus: Rational Number Equivalence, Arithmetic Properties, Solving Equations, Functions

Big IdeasMap of Algebra Knowledge

POWERSOURCE© Components

• Four “units” during the school year (one for each “Big Idea”)

• A series of brief formative assessment tasks

Focused on core concepts in algebra and their interconnection, cognitive demands

Using a variety of formats to demonstrate understanding (computation, problem-solving, explanations, word problems, partially worked examples)

• Focused instructional resources for teachers

Teacher Handbook, exemplar lessons, web-based support

• Professional development activities

Explanation Task-Example• An open-ended question to see if students can

transfer the ideas about the multiplicative identity to a novel situation.

Sample Student Responses

Sample Student Responses (Cont’d)

Sample Teacher Resources (Introduction to Handbook)

Information to say to the class

Information to write on the board

Overheads

Reconciling Design/ Rigor with District Needs

• How to integrate POWERSOURCE© content with on-going initiatives?

• How to be respectful of teacher time but maintain implementation fidelity?

• How to make project useful for all participants while maintaining experimental rigor? (I.e., random assignment)

• How to collect data needed while minimizing district burden?

Alternative Professional Development in NLMUSD

• Developed specifically for implementation in NLMUSD

• Focus on use of district benchmark data (DataDirector)

• Teachers randomly assigned to POWERSOURCE© or Alternative

• Provide a meaningful experience for all teachers (including “controls”) while still maintaining empirical rigor

• Collaboration with districts to find workable approaches to control group

POWERSOURCE© : Findings and Trends

POWERSOURCE© 6th Grade Field Testing

• Random assignment of teachers, 4 districts (6th grade)

• Preliminary results:

POWERSOURCE© students perform significantly better than control students on math learning outcomes.

Based on HLM analyses, there were significant differences between PS and control students for all districts and content areas. The average effect size was .88 (with a district by content area range of .44 to 1.75), and the effect size averages by district ranged from .75 to 1.01.

No initial broad impact on CST (follow-up being tracked).

POWERSOURCE© 2007-08 6th Grade Extended Pilot Test

• Multiple Districts (7, including NLMUSD)

• Mixed Design (random assignment by school or teacher)

• Start with 6th grade in 2007-08, then phase in 7th and 8th grades

• Analysis in progress (multiple student and teacher outcomes)

• Preliminary trends NLMUSD

Preliminary Findings: Change in Proficient and Above 2006-07 to 2007-08 (Grade 6 Math CSTs)

Broader Math Achievement Context

NLMUSD Math Achievement

• POWERSOURCE© part of larger initiative, starting at middle school and eventually moving to other grades

• Overall trajectory of math gains across grades in the district from 2006-07 to 2007-08

Math 6 Benchmark Comparison 2006 - 07 to 2007 - 08

Percent Proficient and Above = 1%

Percent Proficient and Above = 5%

Percent Proficient and Above = 6%%

Percent Proficient and Above = 11%

Percent Proficient and Above =17%

Percent Proficient and Above = 23%

Change +10%

Change +12%

Change +17%

Math 7 Benchmark Comparison 2006 - 07 to 2007 - 08

Percent Proficient and Above = 6%

Percent Proficient and Above = 2%

Percent Proficient and Above = 3%

Percent Proficient and Above = 10%

Percent Proficient and Above =6%

Percent Proficient and Above = 12%

Change +4%

Change +4%

Change +9%

Math 8 - Algebra I Benchmark Comparison 2006 - 07 to 2007 - 08

Percent Proficient and Above = 2%

Percent Proficient and Above <1%

Percent Proficient and Above <1%

Percent Proficient and Above = 15%

Percent Proficient and Above =12%

Percent Proficient and Above = 5%

Change +13%

Change +12%

Change +5%

Grade 6CST Comparsion2006 - 07 to 2007 - 08

10%

15%

20%

25%

30%

35%

40%

Total Number of Students

% Proficient &

Above1766 28%

28%

32%

Total Number of Students

% Proficient &

Above1702 32%

14% increase

Grade 7CST Comparison2006 - 07 to 2007 - 08

10%

15%

20%

25%

30%

35%

40%

26%

37%

Total Number of Students

% Proficient &

Above1752 26%

Total Number of Students

% Proficient &

Above1735 37%

42% increase

10%

15%

20%

25%

30%

35%

40%

13%

24%

Total Number of Students

% Proficient &

Above1778 13%

Total Number of Students

% Proficient &

Above1741 24%

Grade 8CST Comparison2006 - 07 to 2007 - 08

85% increase

griffin@cse.ucla.edu