relating instructional materials use to student achievement using validated measures and a path...

Relating Instructional Materials Use to Student

Achievement

Amy Cassata, Lead Researcher Dae Kim, Lead Researcher

Outlier Research & Evaluation CEMSE | University of Chicago

September 2, 2014

Using Validated Measures and a Path Analysis

Approach

#R305A110621

“In  education,  we  could  be  collecting  information  at  the  school  and  classroom  levels  on  the  instructional  materials  in  use  and  the  associations  between  those  materials  and  student  achievement…”  

BUT “…we  know  almost  nothing  about  the  instructional  materials  being  used.”  

›  Matthew  Chingos  &  Grover  Whitehurst,  The  Brookings  Institution  (April,  2012)  

Instructional Materials are Important to Education Reform

?

? ?

+/- +/-

+ +

+ +

+ +

A Dearth of Information on Instructional Materials Use

78 Elementary math intervention studies in the What Works Clearinghouse

11 were reviewed

6  showed  positive  impacts  on  math  achievement.  5  showed  no  impact  or  mixed  impact.  

Why?

Implementation Matters!

The Black Box of Implementation

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Structural Components • Structural Procedural (SP) • Structural Educative (SE)

Interactional Components • Interactional Pedagogical (IP) • Interactional Student

Engagement (ISE)

Elementary math & science instructional materials

DRK12 #06280052 2007-2010

�  Measurement ›  Rigorous,  valid  and  reliable  

instruments    to  measure  the  variety  of  ways  that  math  and  science  instructional  materials  are  implemented  in  classrooms.  

�  Analysis ›  An  analytic  framework  that  

can  be  used  to  collectively  learn  about  which  components  are  effective  for  which  students  in  which  contexts,  across  interventions.  

Implementation Research Challenges

Math & Science Instructional

Materials

Customized Implementation

Measures

Validation Analyses

• Everyday Math • FOSS, STC, BSCS Science

Tracks, Local curricula

• Questionnaire • Teacher Log • Observation Protocol • Student Questionnaire

• Reliability • Construct validity • Measurement invariance • Predictive validity

IES Instrument Validation Study 2011-2015

#R305A110621

Math & Science Instructional

Materials

Customized Implementation

Measures

Validation Analyses

• Everyday Math • FOSS, STC, BSCS Science

Tracks, Local curricula

• Questionnaire • Teacher Log • Observation Protocol • Student Questionnaire

• Reliability • Construct validity • Measurement invariance • Predictive validity

IES Instrument Validation Study 2011-2015

#R305A110621

The Study Context

#R305A110621, 2011-2015

REESE, #1109595, 2011-2013 PRIME, #DRL-1118866, 2011-2015

�  2 years of data ›  2011-12 and 2012-13

�  5 districts - 3 states �  Over 800 K-5 teachers in

52 schools

�  Online �  30 minutes �  125 items measuring

44 components at the “unit-level”

“The  following  questions  pertain  to  the  most  recent  complete  unit  you  taught  or  the  unit  you  are  currently  teaching  if  you  have  not  yet  completed  a  unit  this  year.        

Please  consider  your  expectations  for  teaching  this  whole  unit  when  responding  to  the  following  questions.”  

Implementation Questionnaires

District Name N Schools N Teachers Percent

Champaign, IL 11 65 14.25

Evanston, IL 12 41 8.99

Tinley Park, IL 5 61 13.38

Stamford, CT 12 289 63.38

TOTAL 40 456 100.00

Spring 2012 Teacher Sample

•  Evenly distributed across grades K-5 •  Range of teaching experience

(Mean = 12.59 years, SD = 7.52)

Data Reduction: From Components to Constructs

53 items measuring 22 SP components 4 items measuring 4 SE components 35 items measuring 10 IP components 33 items measuring 8 ISE components

Data Reduction: From Components to Constructs

53 items measuring 22 SP components 4 items measuring 4 SE components 35 items measuring 10 IP components 33 items measuring 8 ISE components

Structural Items •  Descriptive    •  Not  intended  to  create  indices  •  Combination  of  time,  checklist  and  

Likert  scale  

Data Reduction: From Components to Constructs

53 items measuring 22 SP components 4 items measuring 4 SE components 35 items measuring 10 IP components 33 items measuring 8 ISE components

Interactional Items •  Some  descriptive    •  Some  intended  to  create  indices  •  Likert  scale  •  2-­‐7  items  per  component  

10-Factor Model Number of Items Cronbach’s Alpha

Facilitation of Small Group Work 3 0.67

Facilitation of Student Discussion 3 0.74

Facilitation of Cognitively Demanding Work

7 0.86

Facilitation of Student Autonomy 3 0.71

Facilitation of Student Risk-Taking 3 0.90

Facilitation of Student Interest 4 0.80

Facilitation of Materials, Manipulatives, and Tools Use

2 0.71

Use of Assessment to Inform Instruction 3 0.81

Differentiation 3 0.73

CFA Baseline Model for IP Items

6-Factor Model Items During the unit, how often did you do the following?

Loading

Facilitation of Small Group Work

Call students’ attention to guidelines for group interaction Encourage all group members to contribute (verbally or non-verbally)

0.51

0.94

Facilitation of Cognitively Demanding Work

Analyze (organize, process, manipulate, evaluate) data Explain their reasoning Consider alternative explanations/arguments

0.58 0.78 0.80

Facilitation of Student Risk-Taking

Encourage students to answer a question even if they are unsure Encourage students to take risks in trying new things Encourage students to take risks in asking questions

0.84

0.96 0.81

Facilitation of Student Interest

Engage student interest by connecting the lesson content with current events and real-world phenomena Engage student interest by making lesson content relevant to students Engage student interests through other means (e.g., tell an interesting story, use humor, bring in a guest speaker)

0.76

0.92

0.59

Use of Assessment to Inform Instruction

Change your instruction based on student work and/or responses Re-teach concepts based on student understanding

0.72

0.70

Differentiation Scaffold ideas and activities for individual students Give students different activities based on ability or learning modality Group students based on their ability or learning modality

0.82 0.80

0.47

CFA Final Model for IP Items

Group Work Cognitive Demand Risk-Taking Interest

Assessment to Inform Instruction Differentiation

Group Work 1.000 Cognitive Demand 0.190 1.000

Risk-Taking 0.241 0.403 1.000

Interest 0.202 0.371 0.350 1.000 Assessment to Inform Instruction

0.142 0.140 0.316 0.432 1.000

Differentiation 0.187 0.319 0.344 0.464 0.821 1.000

Correlations of IP Indices

5-Factor Model Number of Items Cronbach’s Alpha

Students Contribute to Small Group Work

3 0.71

Students Engage in Discussion 4 0.80

Students Engage in Cognitively Demanding Work

7 0.86

Students Demonstrate Autonomy 4 0.79

Students Take Risks 3 0.88

CFA Baseline Model for ISE Items

4-Factor Model Items During the unit, what proportion of your students regularly

did the following?

Loading

Students Contribute to Small Group Work

Managed time efficiently when in groups Worked collaboratively with their peers

0.70 0.68

Students Engage in Cognitively Demanding Work

Interpreted written text Supported conclusions with evidence Considered alternative arguments or explanations Analyzed (organized, processed, manipulated, and evaluated) data Demonstrated reasoning Considered relationships between lesson content and academic topics

0.53 0.68 0.70 0.80

0.83 0.69

Students Demonstrate Autonomy

Worked appropriately without regulation Made appropriate choices during the course of the lesson (e.g., groupings, topics to explore, activity order, games to play, etc.)

0.84 0.86

Students Take Risks

Took risks in answering questions Took risks in trying new things

0.91 0.94

CFA Final Model for ISE Items

Group Work Cognitive Demand Autonomy Risk-Taking

Group Work 1.000 Cognitive Demand 0.546 1.000

Autonomy 0.679 0.488 1.000 Risk-Taking 0.466 0.545 0.504 1.000

Correlations of ISE Indices

Second order CFA

Loading

Group Work 0.752 Cognitive Demand 0.725

Autonomy 0.752 Risk-Taking 0.685

�  Path Analysis approach �  Sub-sample: Stamford teachers ›  Total  N=289  ›  Student-­‐teacher  matched  sample  N=125  (grades  2-­‐5)  ›  District-­‐developed  standardized  math  assessment  

�  June  2011  &  June  2012  

Relating EM Implementation Indices to Student Achievement

Grade N Teachers Percent

2 31 24.8

3 33 26.4

4 31 24.8

5 30 24.0

TOTAL 125 100.0

Spring 2012 Stamford Classroom Sample

•  Teachers represent all 12 schools •  Range of teaching experience (Mean=12.66, SD=6.87) •  Average class size is 21 students (Mean=21.22, SD=2.17)

Race/Ethnicity Average % students

per classroom SD

Asian 8.29 6.13

Black 19.60 8.74

Hispanic 34.52 12.62

White 37.48 13.14

Other 0.11 0.69

Spring 2012 Stamford Student Sample

•  49.9% students receive free/reduced price lunch •  8.1% students receive Special Education •  12.5% students designated ELL

Path Analysis

Dependent variable: Current Year’s math achievement

DV is math achievement from June 2012

Path Analysis Independent Variables

Dependent variable: Current Year’s math achievement

Aggregated classroom average math achievement from prior year (June 2011)

Path Analysis Independent Variables

Dependent variable: Current Year’s math achievement

Aggregated classroom average % free/reduced price lunch

Path Analysis Independent Variables

Dependent variable: Current Year’s math achievement

Years of teaching experience

Path Analysis Mediating Variables

Dependent variable: Current Year’s math achievement

6 Interactional-Pedagogical (IP) Indices

Path Analysis Mediating Variables

Dependent variable: Current Year’s math achievement

1 Interactional- Student Engagement (ISE) Index

1.  Effects of IVs on Achievement 2.  Effects of ISE on Achievement 3.  Effects of IPs on Achievement

Path Analysis Results

Path Analysis Effects of IVs on Achievement

Dependent variable: Current Year’s math achievement

Direct effect (a)

Path Analysis Effects of IVs on Achievement

Dependent variable: Current Year’s math achievement

Indirect effect #1 (c x f)

Path Analysis Effects of IVs on Achievement

Dependent variable: Current Year’s math achievement

Indirect effect #2 (c x d x e)

Path Analysis Effects of IVs on Achievement

Dependent variable: Current Year’s math achievement

Indirect effect #3 (b x e)

Path Analysis Effects of IVs on Achievement

Dependent variable: Current Year’s math achievement

Total effect: a + {(c x d x e) + (c x f) + (b x e)}

Estimated Effect

Variable Direct Indirect Total Prior Math Achievement 0.550*** 0.021 0.571***

Free/Reduced Price Lunch -0.054 -0.068 -0.122*

Teaching Experience 0.023 0.002 0.025

Path Analysis Effects of IVs on Achievement

All  path  coefficients  are  standardized.  ***p<.01  **p<.05  *p<.10  

Path Analysis Effects of ISE on Achievement

Dependent variable: Current Year’s math achievement

Direct effect (e)

Estimated Effect

Variable Direct Indirect Total Student Engagement 0.130*** n/a 0.130***

Path Analysis Effects of ISE on Achievement

All  path  coefficients  are  standardized.  ***p<.01  **p<.05  *p<.10  

Path Analysis Effects of IP on Achievement

Dependent variable: Current Year’s math achievement

Direct effect (f)

Path Analysis Effects of IP on Achievement

Dependent variable: Current Year’s math achievement

Indirect effect (d x e)

Path Analysis Effects of IP on Achievement

Dependent variable: Current Year’s math achievement

Total effect: f + (d x e)

Estimated Effect

Variable Direct Indirect Total Facilitation of Small Group Work -0.148** 0.001 -0.147**

Facilitation of Cognitively Demanding Work

0.147* 0.034*** 0.181**

Facilitation of Student Risk-Taking 0.072 -0.004 0.068

Facilitation of Student Interest -0.237** 0.015 -0.222***

Use of Assessment to Inform Instruction

0.083 0.008 0.091

Differentiation 0.088 -0.003 0.085

Path Analysis Effects of IP on Achievement

All  path  coefficients  are  standardized.  ***p<.01  **p<.05  *p<.10  

�  IP indices as DV ›  What  factors  affect  teacher  instructional  practices?  ›  Are  some  type  of  factors  more  influential  than  others?      

�  ISE index as DV ›  What  factors  affect  student  engagement?  ›  Do  some  types  of  instruction  engage  students  more  than  others?  

Other Analyses

�  Replicate the analyses  ›  With  data  collected  in  Spring  2013  ›  With  data  from  other  districts  ›  With  Science  curricula    

�  Add more variables to the model ›  Structural  components  ›  More  teacher-­‐level  characteristics  ›  School-­‐level  characteristics  

Other Analyses

In conclusion: Implementation is Complicated!

vgfjdkfjfdfd

Environmental Factors

School and District Factors

Questions?

For questions about analysis, contact: Dae Y. Kim, PhD, Lead Researcher daeykim@uchicago.edu 773-834-2778 For questions about theoretical framework and instruments, contact: Amy Cassata, PhD, Lead Researcher acassata@uchicago.edu 773-834-2371

Thank  You!