measuring teachers’ use of standards-based instructional materials
DESCRIPTION
Measuring Teachers’ Use of Standards-based Instructional Materials. Karen D. King - PowerPoint PPT PresentationTRANSCRIPT
Measuring Teachers’ Use of Standards-based Instructional MaterialsKaren D. KingThis material is based upon work supported by the National Science Foundation under Grant No. DRL-0732184. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation
OverviewOur framing of the problem -
measuring textbook useResearch questionData collectionResultsInterpretation Implications for practice and
future researchOther areas of research
Instructional Materials vs. Curriculum• Instructional materials are not curriculum• Curriculum is a plan or pathway envisioned from
students' current understanding to target understandings as outlined in Standards
• Instructional materials are tools and resources comprised of a series of instructional tasks and exposition teachers can use to teach the curriculum
• Often, without curriculum guidance, the textbook/instructional materials become the curriculum, substituting instructional materials designers' goals and plans for that of the district or teacher
Why does this distinction matter?A major concern in
implementation is alignment with often conflicting policies◦State Standards◦District Curriculum◦District/School Textbook/Instructional
Materials Adoption◦State Assessments
Our Perspective on Measuring Implementation• Considers the alignment of textbook
to curriculum (Mitchell, King & Gearhardt, in revision)
• Considers alignment of curriculum to state standards (Mitchell, et al., in revision; Porter, 2002)
• Considers alignment of textbook, curriculum, and state standards to state achievement tests (King, et al., in revision)
An Example
0.00 0.20 0.40 0.60 0.80
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
Fig 3. Grade 8 Cognitive Demand
NJ Stnds 2009 Gr. 8 CMP2 All Units Gr.8
Research Question• What is the relationship between middle grades mathematics teachers' use and adaptation of Standards-based instructional materials and students' achievement?
Research Setting• Newark, NJ in 2008-2009 school year• Had adopted Connected Mathematics
Project (CMP) in 2002, started using CMP 2 in 2008-2009
• Had an NSF-funded Local Systemic Change project that provided substantial professional development on CMP and middle grades mathematics education through 2008
Data Collection • Part of a larger mixed methods study• Surveyed 159 middle grades mathematics
teachers and the student achievement data from 2528 of their students in their first mathematics class of the week
• In Spring 2009, teachers completed the Surveys of Enacted Curriculum Instructional Content and Instructional Practices surveys and project-developed CMP
• Students took the NJ ASK during Spring 2009
Surveys of Enacted CurriculumDemographic information on the teacher,
teacher qualifications, and professional development experiences
Information on the teachers’ instructional practices
Information on the content and cognitive demand of the instruction
To ensure accurate responses, we conducted a professional development session on assessing the cognitive demand of tasks for students prior to teachers completing the survey
SEC Instructional Content SurveyTime on Topic For 188 content areas ranging from
number to advanced algebra, the SEC-IC asks what proportion of time is spent teaching the topic
Emphasis For each content area that the teacher indicates he or she teaches, the SEC-IC asks what proportion of time spent teaching the topic was spent at 5 levels of cognitive demand◦ Memorize facts, perform procedures, demonstrate
understanding, conjecture/generalize/prove, solve non-routine problems/make connections
Prior to survey administration, teachers participated in professional development to ensure that the meanings for each was established and common
CMP Implementation SurveyBackground information on
teachers’ experiences with CMPGeneral questions about the
teachers’ use of other materials and context of teaching
For each lesson organized by units, we asked teachers to describe their use of each lesson in the textbook with the following instruction:
To answer the following questions, use the following definitions of textbook use
Use without modification – Follows the outline of the lesson as described in the teachers’ materials. No changes are made to the task and the structure of the lesson follows the Launch, Explore, Summarize routine described in the CMP materials. The goals and objectives of the lesson remain the same.
Use with adaptation – Follows the outline of the lesson as described in the teachers’ materials with changes made to the task, but not the structure of the lesson. The goals and objectives of the lesson remain the same. The structure of the lesson follows the Launch, Explore, Summarize routine described in the CMP materials, but the task is modified in one of several ways
Use as a one of many resources – Takes tasks from the materials but uses them to create own lesson with own lesson structure.
Replace – Does not use the materials but takes tasks or lessons from other materials to teach the goals and objectives of the lesson.
Does not use – Does not use any materials to teach the goals and objectives of the lesson.
Alignment of NJ ASK to New Jersey Mathematics Standards and Instructional Materials
NJASK2009
State Mathematics Standards
Connected Mathematics 2 Instructional Materials
Grade 6 .29 .21*
Grade 7 .30 .26
Grade 8 .30 .25
Alignment considers both the content and cognitive demand alignment (see Porter, et al., 2008)
Data Analysis• Includes 143 teachers and their 2107
students• Removes teachers we did not have
information on: • their position type (general or special
education), • teachers whose classrooms were comprised of
very mobile students• students for whom we did not have 2008 math
scores • About 1/3 of teachers in the sample were
special education teachers
Measure construction• To measure use of the materials, we
created the curriculum coverage index (CCI) and to measure adaptation, we created the lesson modification index (LMI). • CCI is a simple proportion of the CMP lessons
that were not taught during the year• LMI represents the extent to which the teacher
has modified the lesson, expressed on a three-point scale where the low score represents all lessons as used without adaptation and the high score represents all lessons used as one of many resources.
Measure ConstructionThe SEC creates cognitive demand
measures that sum to 1 based on emphasis of each of the types of cognitive demand Memorize facts, perform procedures, demonstrate
understanding, conjecture/generalize/prove, solve non-routine problems/make connections
Using factor analysis on these SEC measures, we created a new cognitive demand measure which is characterized by stressing "procedures" and avoiding "non-routine problems"
Results: Incomplete table ofHLM Fixed Effects Estimates for a Model of 2009 NJ ASK Mathematics Scale Score with Robust Errors
Fixed Effect Model
Intercept 203.54 (2.41)
Level 1
2008 Math Score 0.72** (0.03)
Special Education -14.52** (2.39)
LEP -0.73 (3.27)
Black -3.76 (2.43)
Latino -0.96 (2.02)
Level 2
LMI 4.191* (2.06)
CCI -0.26**( 0.07)
Cognitive Demand Factor 2.82* (1.28)
Summary of Results Teachers’ increased use of the CMP instructional
materials as compared to average is significantly related to higher student achievement.
However, in relationship to traditional measures of fidelity, where the focus is on use without adaptation, our findings suggest that greater than average adaptation on the part of the teacher is related to increased achievement of students.
A focus on procedures and decreased emphasis on non-routine problems is significantly related to higher student achievement.
Instructional Practice scales were not statistically significant.
Teachers’ alignment was not statistically significant association with student achievement.
Also, while 2008 achievement and special education classifications of students remained significant factors in predicting 2009 student achievement, race/ethnicity and LEP status did not.
O’Donnell’s 5 Criteria for Measuring FOI
Adherence—whether the components of the intervention are being delivered as designed;
Duration—the number, length, or frequency of sessions implemented;
Quality of delivery— the manner in which the implementer delivers the program using the techniques, processes, or methods prescribed;
Participant responsiveness—the extent to which participants are engaged by and involved in the activities and content of the program; and
Program differentiation—whether critical features that distinguish the program from the comparison condition are present or absent during implementation. (O’Donnell, 2008, p. 34)
Difference in measures of FOI
Our Study O’Donnell’s Criteria Curriculum Coverage (CCI)
Alignment (includes time and cognitive demand emphasis
adaptation to local contexts of the materials (LMI)
Decreased cognitive demand
Extent of use or adherence
Duration
Quality of Delivery
LimitationsWe do not account for teachers’
mathematical knowledge for teaching in our model. ◦ The SEC asks questions about certifications and
degrees, but, there was little variability in teacher qualifications of this type Only 6 teachers (3.8%) had secondary mathematics
certification Only 11 teachers (7.1%) had a major in mathematics or
mathematics education at either the undergraduate or Masters level.
The lack of fine-grained data collection for teachers’ years of experience, especially in the early years when student learning gains are most pronounced (Clotfelter et al., 2007). ◦ However, given our results for years teaching with
the CMP instructional materials, we may have produced the same general results for teaching experience with a better measure.
Limitations Continued
Proficient
Advanced Proficient
Partially Proficient
Returning to the Example
0.00 0.20 0.40 0.60 0.80
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
Fig 3. Grade 8 Cognitive Demand
NJ Stnds 2009 Gr. 8 CMP2 All Units Gr.8
0.00 0.20 0.40 0.60 0.80
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
Fig. 2 Grade 7 Cognitive Demand
NJ Stnds 2009 Gr. 7 CMP2 All Units Gr.7
NJ ASK Item Types by Grade Grade 6 Grade 7 Grade 8
Item Count by
Type
MC 42 42 42
SCR 8 (non-calculator) 10 10
ECR 5 5 5
# of sections 5 5 5
Total raw score
points possible
(excluding field test items)
50 52 52
Approximate total testing time 120 min. 124 min. 133 min.
MC - multiple choice, 1 raw score point
SCR - short constructed response, 1 raw score point
ECR – extended constructed response, 3 raw score points
Implications for Research in an Era of CCSSM To the extent there was lack of alignment between the NJ
ASK, the standards, and the instructional materials, it was to decrease cognitive demand as opposed to content coverage.
How well the outcome measure aligns with the standards, curriculum and the instructional materials is a critical factor in the results of such studies.
How well the PARCC and SBAC assessments will align with the CCSSM, particularly the Standards for Mathematical Practice will have implications for cross-state research on instructional materials use
In an era of openness and the Internet, we need to better understand what teachers use outside the adopted instructional materials, and how this use supports or detracts from a cohesive and coherent curriculum
Suggestions for Policy and Practice
As districts adopt new instructional materials and regimes, implementation strategies and monitoring should focus on both fidelity to structure and process.
As textbook authors design and modify texts, they should consider guidance to districts and teachers that support implementation by providing sufficient description of both the structure and the process of implementation that allow them to make appropriate adaptations.