relating instructional materials use to student achievement using validated measures and a path...

Relating Instructional Materials Use to Student

Achievement

Amy Cassata, Lead Researcher Dae Kim, Lead Researcher

Outlier Research & Evaluation CEMSE | University of Chicago

September 2, 2014

Using Validated Measures and a Path Analysis

Approach

#R305A110621

“In education, we could be collecting information at the school and classroom levels on the instructional materials in use and the associations between those materials and student achievement…”

BUT “…we know almost nothing about the instructional materials being used.”

›  Matthew Chingos & Grover Whitehurst, The Brookings Institution (April, 2012)

Instructional Materials are Important to Education Reform

?

? ?

+/- +/-

+ +

+ +

+ +

A Dearth of Information on Instructional Materials Use

78 Elementary math intervention studies in the What Works Clearinghouse

11 were reviewed

6 showed positive impacts on math achievement. 5 showed no impact or mixed impact.

Why?

Implementation Matters!

The Black Box of Implementation

Opening the Black Box: A Component Approach

E

Elementary math & science instructional materials

Unit Duration Session Frequency Instructional Time Lesson Order Order of Lesson Parts Materials Presence Writing Structures Readings Assessments & Tools Lesson Content Class Structures Instructional Formats Extensions Homework Additional Resources Projects Background on Content Background on Pedagogy Information on Standards Lesson Notes

Facilitation of Discussion Facilitation of Cognitively Demanding Work Facilitation of Group Work Facilitation of Autonomy Facilitation of Risk-Taking Facilitation of Interest Facilitation of Materials Use Use of Assessment to Inform Instruction Differentiation Students do Group Work Students Engage in Discussion Students Demonstrate Autonomy Students Take Risks Students Engage in Cognitively Demanding Work

DRK12 #06280052 2007-2010

Opening the Black Box: A Component Approach

E

Structural Components • Structural Procedural (SP) • Structural Educative (SE)

Interactional Components • Interactional Pedagogical (IP) • Interactional Student

Engagement (ISE)

Elementary math & science instructional materials

DRK12 #06280052 2007-2010

�  Measurement ›  Rigorous, valid and reliable

instruments to measure the variety of ways that math and science instructional materials are implemented in classrooms.

�  Analysis ›  An analytic framework that

can be used to collectively learn about which components are effective for which students in which contexts, across interventions.

Implementation Research Challenges

Math & Science Instructional

Materials

Customized Implementation

Measures

Validation Analyses

• Everyday Math • FOSS, STC, BSCS Science

Tracks, Local curricula

• Questionnaire • Teacher Log • Observation Protocol • Student Questionnaire

• Reliability • Construct validity • Measurement invariance • Predictive validity

IES Instrument Validation Study 2011-2015

#R305A110621

The Study Context

#R305A110621, 2011-2015

REESE, #1109595, 2011-2013 PRIME, #DRL-1118866, 2011-2015

�  2 years of data ›  2011-12 and 2012-13

�  5 districts - 3 states �  Over 800 K-5 teachers in

52 schools

�  Online �  30 minutes �  125 items measuring

44 components at the “unit-level”

“The following questions pertain to the most recent complete unit you taught or the unit you are currently teaching if you have not yet completed a unit this year.

Please consider your expectations for teaching this whole unit when responding to the following questions.”

Implementation Questionnaires

District Name N Schools N Teachers Percent

Champaign, IL 11 65 14.25

Evanston, IL 12 41 8.99

Tinley Park, IL 5 61 13.38

Stamford, CT 12 289 63.38

TOTAL 40 456 100.00

Spring 2012 Teacher Sample

•  Evenly distributed across grades K-5 •  Range of teaching experience

(Mean = 12.59 years, SD = 7.52)

Data Reduction: From Components to Constructs

53 items measuring 22 SP components 4 items measuring 4 SE components 35 items measuring 10 IP components 33 items measuring 8 ISE components



Structural Items •  Descriptive •  Not intended to create indices •  Combination of time, checklist and

Likert scale



Interactional Items •  Some descriptive •  Some intended to create indices •  Likert scale •  2-‐7 items per component

10-Factor Model Number of Items Cronbach’s Alpha

Facilitation of Small Group Work 3 0.67

Facilitation of Student Discussion 3 0.74

Facilitation of Cognitively Demanding Work

7 0.86

Facilitation of Student Autonomy 3 0.71

Facilitation of Student Risk-Taking 3 0.90

Facilitation of Student Interest 4 0.80

Facilitation of Materials, Manipulatives, and Tools Use

2 0.71

Use of Assessment to Inform Instruction 3 0.81

Differentiation 3 0.73

CFA Baseline Model for IP Items

6-Factor Model Items During the unit, how often did you do the following?

Loading

Facilitation of Small Group Work

Call students’ attention to guidelines for group interaction Encourage all group members to contribute (verbally or non-verbally)

0.51

0.94


Analyze (organize, process, manipulate, evaluate) data Explain their reasoning Consider alternative explanations/arguments

0.58 0.78 0.80

Facilitation of Student Risk-Taking

Encourage students to answer a question even if they are unsure Encourage students to take risks in trying new things Encourage students to take risks in asking questions

0.84

0.96 0.81

Facilitation of Student Interest

Engage student interest by connecting the lesson content with current events and real-world phenomena Engage student interest by making lesson content relevant to students Engage student interests through other means (e.g., tell an interesting story, use humor, bring in a guest speaker)

0.76

0.92

0.59

Use of Assessment to Inform Instruction

Change your instruction based on student work and/or responses Re-teach concepts based on student understanding

0.72

0.70

Differentiation Scaffold ideas and activities for individual students Give students different activities based on ability or learning modality Group students based on their ability or learning modality

0.82 0.80

0.47

CFA Final Model for IP Items

Group Work Cognitive Demand Risk-Taking Interest

Assessment to Inform Instruction Differentiation

Group Work 1.000 Cognitive Demand 0.190 1.000

Risk-Taking 0.241 0.403 1.000

Interest 0.202 0.371 0.350 1.000 Assessment to Inform Instruction

0.142 0.140 0.316 0.432 1.000

Differentiation 0.187 0.319 0.344 0.464 0.821 1.000

Correlations of IP Indices

5-Factor Model Number of Items Cronbach’s Alpha

Students Contribute to Small Group Work

3 0.71

Students Engage in Discussion 4 0.80

Students Engage in Cognitively Demanding Work

7 0.86

Students Demonstrate Autonomy 4 0.79

Students Take Risks 3 0.88

CFA Baseline Model for ISE Items

4-Factor Model Items During the unit, what proportion of your students regularly

did the following?

Loading

Students Contribute to Small Group Work

Managed time efficiently when in groups Worked collaboratively with their peers

0.70 0.68

Students Engage in Cognitively Demanding Work

Interpreted written text Supported conclusions with evidence Considered alternative arguments or explanations Analyzed (organized, processed, manipulated, and evaluated) data Demonstrated reasoning Considered relationships between lesson content and academic topics

0.53 0.68 0.70 0.80

0.83 0.69

Students Demonstrate Autonomy

Worked appropriately without regulation Made appropriate choices during the course of the lesson (e.g., groupings, topics to explore, activity order, games to play, etc.)

0.84 0.86

Students Take Risks

Took risks in answering questions Took risks in trying new things

0.91 0.94

CFA Final Model for ISE Items

Group Work Cognitive Demand Autonomy Risk-Taking

Group Work 1.000 Cognitive Demand 0.546 1.000

Autonomy 0.679 0.488 1.000 Risk-Taking 0.466 0.545 0.504 1.000

Correlations of ISE Indices

Second order CFA

Loading

Group Work 0.752 Cognitive Demand 0.725

Autonomy 0.752 Risk-Taking 0.685

�  Path Analysis approach �  Sub-sample: Stamford teachers ›  Total N=289 ›  Student-‐teacher matched sample N=125 (grades 2-‐5) ›  District-‐developed standardized math assessment

�  June 2011 & June 2012

Relating EM Implementation Indices to Student Achievement

Grade N Teachers Percent

2 31 24.8

3 33 26.4

4 31 24.8

5 30 24.0

TOTAL 125 100.0

Spring 2012 Stamford Classroom Sample

•  Teachers represent all 12 schools •  Range of teaching experience (Mean=12.66, SD=6.87) •  Average class size is 21 students (Mean=21.22, SD=2.17)

Race/Ethnicity Average % students

per classroom SD

Asian 8.29 6.13

Black 19.60 8.74

Hispanic 34.52 12.62

White 37.48 13.14

Other 0.11 0.69

Spring 2012 Stamford Student Sample

•  49.9% students receive free/reduced price lunch •  8.1% students receive Special Education •  12.5% students designated ELL

Path Analysis

Dependent variable: Current Year’s math achievement

DV is math achievement from June 2012

Path Analysis Independent Variables


Aggregated classroom average math achievement from prior year (June 2011)



Aggregated classroom average % free/reduced price lunch



Years of teaching experience

Path Analysis Mediating Variables


6 Interactional-Pedagogical (IP) Indices

Path Analysis Mediating Variables


1 Interactional- Student Engagement (ISE) Index

1.  Effects of IVs on Achievement 2.  Effects of ISE on Achievement 3.  Effects of IPs on Achievement

Path Analysis Results

Path Analysis Effects of IVs on Achievement


Direct effect (a)



Indirect effect #1 (c x f)



Indirect effect #2 (c x d x e)



Indirect effect #3 (b x e)



Total effect: a + {(c x d x e) + (c x f) + (b x e)}

Estimated Effect

Variable Direct Indirect Total Prior Math Achievement 0.550*** 0.021 0.571***

Free/Reduced Price Lunch -0.054 -0.068 -0.122*

Teaching Experience 0.023 0.002 0.025


All path coefficients are standardized. ***p<.01 **p<.05 *p<.10

Path Analysis Effects of ISE on Achievement


Direct effect (e)

Estimated Effect

Variable Direct Indirect Total Student Engagement 0.130*** n/a 0.130***

Path Analysis Effects of ISE on Achievement


Path Analysis Effects of IP on Achievement


Direct effect (f)



Indirect effect (d x e)



Total effect: f + (d x e)

Estimated Effect

Variable Direct Indirect Total Facilitation of Small Group Work -0.148** 0.001 -0.147**


0.147* 0.034*** 0.181**

Facilitation of Student Risk-Taking 0.072 -0.004 0.068

Facilitation of Student Interest -0.237** 0.015 -0.222***

Use of Assessment to Inform Instruction

0.083 0.008 0.091

Differentiation 0.088 -0.003 0.085



�  IP indices as DV ›  What factors affect teacher instructional practices? ›  Are some type of factors more influential than others?

�  ISE index as DV ›  What factors affect student engagement? ›  Do some types of instruction engage students more than others?

Other Analyses

�  Replicate the analyses ›  With data collected in Spring 2013 ›  With data from other districts ›  With Science curricula

�  Add more variables to the model ›  Structural components ›  More teacher-‐level characteristics ›  School-‐level characteristics

Other Analyses

In conclusion: Implementation is Complicated!

vgfjdkfjfdfd

Environmental Factors

School and District Factors

Questions?

For questions about analysis, contact: Dae Y. Kim, PhD, Lead Researcher [email protected] 773-834-2778 For questions about theoretical framework and instruments, contact: Amy Cassata, PhD, Lead Researcher [email protected] 773-834-2371

Thank You!

relating instructional materials use to student achievement using validated measures and a path...

Education

facilitation of materials

group work students

discussion students

autonomy students

content background

risks students

demanding work drk12

use of assessment