![Page 1: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/1.jpg)
Measuring Teachers’ Fidelity of Implementationto Algebra Curricula that Emphasize Mathematical PracticesAEA Evaluation 2013October 18, 2013, 2:40 pm – 4:10 pm
![Page 2: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/2.jpg)
04/19/2023 2
Agenda
• Background
• Three projects & approaches
• Lessons
• Questions & discussion
![Page 3: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/3.jpg)
04/19/2023 EDC Powerpoint Template 3
BackgroundThree algebra interventions
1. Transition to Algebra (TTA)
2. Learning and Teaching Algebra (LTA)
3. Mathematical Practices Implementation Study (MPI)
![Page 4: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/4.jpg)
04/19/2023 4
Mathematical Habits of Mind
“…we’d like students to think about mathematics the way mathematicians do….”
- Cuoco, Goldenberg, & Mark (1996, p. 377)
![Page 5: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/5.jpg)
04/19/2023 5
Example
Simplify.
8(992 1) 3(992 1) 11(992 1)
![Page 6: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/6.jpg)
04/19/2023 6
Example cont’dUsing Structure – Chunking: treating an expression, number, or other structural element as a single object. A variable might be substituted for the object or chunk.
8(992 1) 3(992 1) 11(992 1)
![Page 7: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/7.jpg)
04/19/2023 7
Why Examine Fidelity?
• Teacher fidelity as an important predictor or outcome- TTA: teacher fidelity student outcomes
- LTA: PD fidelity teacher fidelity teacher and student outcomes
- MPI: mathematical knowledge teacher fidelity teaching practices
![Page 8: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/8.jpg)
04/19/2023 8
Three Takeaways
1. Content and presentation fidelity are different
• Both should be measured
• Content = “what”
• Presentation = “how”
![Page 9: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/9.jpg)
04/19/2023 9
Three Takeaways
2. Measuring presentation fidelity is hard
• Must identify what teachers do
• Must assess against curriculum intent
• Multiple ways to be consistent with curriculum
- Similar to measuring HQ instruction in general
![Page 10: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/10.jpg)
04/19/2023 10
Three Takeaways
3. Resources affect validity, reliability, and usability
![Page 11: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/11.jpg)
04/19/2023 11
Transition to Algebra Project
The quest for reliable and valid measures of fidelity of implementation in live observation settings
![Page 12: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/12.jpg)
04/19/2023 12
Transition to Algebra Project (TTA)
• Algebra intervention for double-period algebra
• Puzzles & more
![Page 13: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/13.jpg)
04/19/2023 13
TTA Puzzle Examples
![Page 14: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/14.jpg)
04/19/2023 14
Do students with teachers with higher fidelity to TTA have stronger mathematics outcomes?
Teacher content & presentation fidelity
TTA Fidelity Questions
Student algebra performance
Student attitudes toward mathematics
![Page 15: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/15.jpg)
04/19/2023 15
TTA Fidelity Data Sources
• Teacher surveys- For content fidelity
![Page 16: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/16.jpg)
04/19/2023 16
TTA Fidelity Data Sources
• Classroom observations- For content & presentation fidelity- Live, no video- Created protocol & checklist
![Page 17: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/17.jpg)
04/19/2023 17
TTA Instrument Development
• Curriculum developers played primary role- Identified key lesson components (content)
- Identified key activity indicators (presentation)
- Pilot tested and conducted final data collection
![Page 18: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/18.jpg)
04/19/2023 18
TTA Instrument Development
• Iterative rounds of testing- Paired observers
- Revised and refined items in response to inter-rater disagreement
![Page 19: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/19.jpg)
04/19/2023 19
TTA Development Constraints
• Live observations only
• 2 ½ month development period
• Final data collected same school year
• Low-inference items for inter-rater reliability
![Page 20: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/20.jpg)
04/19/2023 20
TTA Instrument Format
• 4 basic lesson components
• Checklist items within each component
• Several recurring activities
• Notes about each component or activity
• Overall fidelity rating (1-5)
![Page 21: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/21.jpg)
04/19/2023 21
TTA Checklist ExcerptsIII. Student problem-solving
1. Did the teacher expose students to problems from the Student Book that were not in the launch materials? Yes No
If yes to 1:2. Did students have a chance to attempt TTA problems on their own? (could be independently or with other students, without the teacher demonstrating each step) Yes No
If yes to 2:3. Did students spend any time collaborating to solve TTA problems? [Does not include simply sharing answers] Yes _ No 4. Did students work on TTA problems for at least 10 minutes during class? Yes _ No
Notes about Student problem-solving:
Low Medium High1 2 3 4 5
Overall fidelity rating for this class:
![Page 22: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/22.jpg)
04/19/2023 22
TTA Instrument Reliability
• Individual items: strong reliability
• Overall fidelity rating not reliable
• Lacked capacity to check for rater drift
![Page 23: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/23.jpg)
04/19/2023 23
TTA Instrument Validity
• Summary score validity questions
• Face validity issues
• A measure of content fidelity
![Page 24: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/24.jpg)
04/19/2023 24
TTA Instrument Usability
• Little training required to use checklist
• Clearer definition of overall fidelity ratings required
![Page 25: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/25.jpg)
04/19/2023 25
Learning and Teaching Algebra ProjectAssessing Teachers’ Implementation of a Curriculum Focused on Mathematical Practices
![Page 26: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/26.jpg)
Learning and Teaching Algebra Project (LTA)
• Comprehensive and integrated CME Algebra I support:
– Live professional development (PD)
– Coaching
– Classroom implementation and coach facilitation guides
04/19/2023 26
![Page 27: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/27.jpg)
04/19/2023 27
• Does LTA PD and coaching impact classroom fidelity?• Does faithful use of LTA materials impact student outcomes?
LTA Fidelity Questions
Teacher content & presentation fidelity
Student Mathematical Habits of Mind
Professional Development and Coaching
Other mediators and moderators:• MKT• Teacher Mathematical Habits of Mind• Academic rigor of instruction
![Page 28: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/28.jpg)
04/19/2023 28
LTA Fidelity Data Sources
• Teacher surveys
• Coaching Implementation Log
• Classroom Observations
- Live
- Video
![Page 29: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/29.jpg)
04/19/2023 29
LTA Instrument Development
• Behavioral indicators identified from PD/curriculum
• Refined, defined, and added to over 2 years
- Year 1: Used in live classroom
- Year 2: Several revision iterations using classroom video
- Year 3 (current year): use live with double-coding of video
![Page 30: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/30.jpg)
04/19/2023 30
LTA Development Constraints
• Use requires CME training and knowledge of math teaching
• Live use
• Use with any CME investigation
• Use across CME investigation over several days
• Reciprocal relationship with LTA development materials
![Page 31: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/31.jpg)
04/19/2023 31
LTA Instrument Format
• 3-part observation:
– Pre-Observation interview
– Observation Protocol
• Field notes
• Content table
• Presentation across 3 dimensions
– Post-Observation interview
![Page 32: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/32.jpg)
LTA Presentation Fidelity Dimension 1: Developing Coherence
• Teacher uses CME features to…
– Build theory or extend student thinking
04/19/2023 32
![Page 33: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/33.jpg)
LTA Sample Behavioral MarkersDeveloping Coherence
Highly consistent with author intentConsistent with author intent
Not consistent with author intent
04/19/2023 33
![Page 34: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/34.jpg)
LTA Sample Behavioral MarkersDeveloping Coherence
Highly consistent with author intent
-Teacher explicitly makes connections to habits of mind
Consistent with author intent
Not consistent with author intent
04/19/2023 34
![Page 35: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/35.jpg)
LTA Sample Behavioral MarkersDeveloping Coherence
Highly consistent with author intent
-Teacher explicitly makes connections to habits of mind
Consistent with author intent
-Teacher encourages the use of existing student skills with novel problems
Not consistent with author intent
04/19/2023 35
![Page 36: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/36.jpg)
LTA Sample Behavioral MarkersDeveloping Coherence
Highly consistent with author intent
-Teacher explicitly makes connections to habits of mind
Consistent with author intent
-Teacher encourages the use of existing student skills with novel problems
Not consistent with author intent
-Teacher provides specialized methods for solving problems
04/19/2023 36
![Page 37: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/37.jpg)
LTA Presentation Fidelity Dimension 2: Developing Habits of Mind
• Teacher uses CME features to…
– Promote mathematics as a way of thinking, not a set of skills
04/19/2023 37
![Page 38: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/38.jpg)
LTA Presentation Fidelity Dimension 3: Experience before Formality
• Teacher uses CME features to…
– Provide students with experience in mathematics before formal definition or process
04/19/2023 38
![Page 39: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/39.jpg)
04/19/2023 39
LTA Instrument Reliability
• Inter-rater consistency calculated using three out of four raters for agreement
- For markers present: 79%
- For all markers: 90%
• Inter-rater reliabilities (2013-2014)
- A portion of live ratings will be double-coded using video to ensure no rater drift
![Page 40: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/40.jpg)
04/19/2023 40
LTA Instrument Validity
• PD providers and implementation guide writers served as instrument developers
• Author served as reviewer on instrument
![Page 41: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/41.jpg)
04/19/2023 41
LTA Instrument Usability
• Specialized knowledge required for rating
• Large number of markers requires multiple ratings for training
• Requires extended time in the classroom
![Page 42: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/42.jpg)
04/19/2023 42
Mathematical Practices Implementation StudyCapturing fidelity of implementation to a curriculum’s content and design principles
![Page 43: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/43.jpg)
04/19/2023 43
Mathematical Practices Implementation Study (MPI)
• 4-year mixed-methods study
• Examining teacher use of new curriculum over two years
• Moderate professional development support
![Page 44: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/44.jpg)
04/19/2023 44
MPI Fidelity Questions
ContentFidelity
PresentationFidelity
Mathematical Knowledge for
Teaching
Instructional Quality
How may the level of implementation of CME Project be related to high school teachers’ mathematical knowledge for teaching and instructional practices?
![Page 45: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/45.jpg)
04/19/2023 45
MPI Instrument Development
• Teacher’s Guide and Implementation Guide• Meetings with authors• Meetings with experienced users and coaches
• Iterations tested with classroom video• Minimal developer/author support
![Page 46: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/46.jpg)
04/19/2023 46
MPI Development Constraints
• Summative scoring
• Balance between usability and depth
• Low fidelity not defined
![Page 47: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/47.jpg)
04/19/2023 47
MPI Instrument Format
• Content Fidelity- Extent of textbook use- Goals and Habits of Mind addressed in the Teacher’s
Edition
• Presentation Fidelity- Four scales - Indicators- Rubrics
![Page 48: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/48.jpg)
04/19/2023 48
MPI Sample Scale
Mathematical Thinking Made Visible
• Teacher provided time for students to discuss mathematics and asked for critique and evaluation.
• Teacher mined student ideas and sought multiple approaches.• Teacher clarified or extended student ideas.• Teacher demonstrated forming conjectures and generalizations.• Teacher asked or encouraged students to explain concepts in their own
words.
Not consistent Moderately consistent Consistent1 2 3
![Page 49: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/49.jpg)
04/19/2023 49
MPI Instrument Reliability
Reliability work is ongoing
![Page 50: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/50.jpg)
04/19/2023 50
MPI Instrument Validity
Face validity developed through cycles of author and coach evaluation
![Page 51: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/51.jpg)
04/19/2023 51
MPI Instrument Usability
• Use by researchers
• Familiarity with the CME approach
• Reliance upon text materials
![Page 52: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/52.jpg)
04/19/2023 EDC Powerpoint Template 52
Lessons
Three Takeaways
![Page 53: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/53.jpg)
04/19/2023 53
1. Content and Presentation Differ
• All three projects
- aim to measure both types of fidelity
- turned to observations for measuring presentation
![Page 54: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/54.jpg)
04/19/2023 54
2. Measuring Presentation Is Hard
• All tried to identify curriculum intent
• Lengthy development periods for LTA and MPI tools
![Page 55: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/55.jpg)
04/19/2023 55
3. Resources Affect Outcomes
• Key resources- Training & knowledge
of raters- Access to video- Development
timeframe- Access to developers
• Instrument outcomes- Validity
• Type of fidelity measured
- Reliability- Usability
![Page 56: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/56.jpg)
04/19/2023 56
Questions & Discussion
• What questions does our work raise for you?
• What have you done to measure content and/or presentation fidelity?
• How have you addressed validity, reliability, and usability of instruments in these efforts?
![Page 57: Measuring Teachers’ Fidelity of Implementation](https://reader036.vdocuments.us/reader036/viewer/2022062422/56812eda550346895d947c2f/html5/thumbnails/57.jpg)
04/19/2023 57
Contact Us
• TTA: [email protected]
• LTA: [email protected], [email protected]
• MPI: [email protected]