andrew c. porter vanderbilt university measuring the content of instruction: uses in research and...
DESCRIPTION
Content MatrixTRANSCRIPT
Andrew C. PorterVanderbilt University
Measuring the Content of Instruction:
Uses in Research and Practice
Tools
Teacher surveys of instructionDaily logsEnd-of-semester or end-of-year surveys
Content analyses ofStandardsTestsCurriculum materials
Alignment indices—e.g, alignment between assessment and standards
Categories of Cognitive Demand
TopicsMemorize Perform
ProceduresCommunicateUnderstanding
Solve Non-Routine
Problems
ConjectureGeneralize
ProveMultiple
StepEquations
Inequalities
LiteralEquations
Lines /Slope andIntercept
Operationson
Polynomials
QuadraticEquations
Content Matrix
Instruction
Vertical and Horizontal Alignment
Achievement
Assessment
Assessment Standards
StandardsDistrict
State
Example Matrices to Measure Alignment
.3 0 .10 .1 00 .2 .10 .1 .1
.2 0 .10 .2 0.1 .2 .10 0 .1
Assessment Standards
Topics
Cognitive Demand
Alignment Index =1 -X=Assessment Cell ProportionsY=Standards Cell Proportions
∑ |X-Y|
2
Alignment of Assessments with Standards7th-Grade Math:Goals Study
Standards Assessment States B D E F
B .37 .39 .37 .45 D .35 .37 .36 .40 E .36 .33 .43 .31 F .32 .35 .30 .41
NCTM .34 .40 .33 .47
Average Within-State Alignment = .40Average Between-State Alignment = .39Average State-Test-to-NCTM Alignment = .39
Alignment of Instruction with Assessment8th-Grade Math:SCASS Study
Instruction AssessmentState H J K L E O NAEP
H 0.35 0.22 0.19 0.28 0.21 0.04 0.38J 0.34 0.21 0.18 0.25 0.20 0.05 0.38K 0.42 0.28 0.21 0.29 0.25 0.05 0.39L 0.36 0.24 0.19 0.29 0.22 0.05 0.40E 0.39 0.26 0.17 0.26 0.24 0.04 0.38O 0.35 0.21 0.16 0.26 0.20 0.05 0.38
Average Within-State Alignment = .22Average Between-State Alignment = .23Average State-to-NAPE Alignment = .39
States H J K L E O G I M N
HJ .73
K .59 .66L .56 .64 .67E .65 .71 .78 .70
O .71 .80 .63 .65 .70G .71 .81 .66 .67 .71 .84I .73 .82 .63 .66 .68 .79 .80M .68 .77 .61 .62 .66 .73 .76 .79N .62 .69 .58 .61 .62 .71 .70 .67 .65
Average Alignment = .69
Alignment of Instruction with Instruction 8th-Grade Math: SCASS Study
7th Grade StandardsState E State F NCTM
Quality of Data Response rates Interrater agreement for content
analyses Validity of teacher self-report
Explaining between-teacher variance in alignment to NAEPPredicting student achievement gains
[Note: The need for a reform-neutral language]
Uses of Tools Describing Instructional Practices
ResearchServe as dependent variable in teacher decision-making researchDescribe the implemented curriculum Measure implementation of new curriculaAssess the validity of transcript studies
PracticeInform teacher reflections on their own instructional practices
[Note: Should not be used for teacher accountability.]
Uses of Tools Describing Instructional Materials
ResearchResearch effects of textbooks on instructionAssess the breadth and depth of content in instructional materials
PracticeBuild testsWrite content standardsDevelop national, state, or district indicator systems
Uses of Indices of Alignment Research
Serve as a control variableServe as a dependent variable Serve as a descriptive variable
PracticeAlign state tests to state standardsAlign instructional materials to standards or course frameworks
Increasing Validity and Value Getting the content languagecontent language right Using time samples to describe instruction for an entire
school year Replicating the finding that alignment predicts student
achievement gains Identifying contexts in which teacher self-reportteacher self-report on the
content of instruction is more or less accurate Improving the level and consistency of interrater interrater
agreementagreement in content analyses Understanding the distributional propertiesdistributional properties of the
alignment statistics Building powerful professional developmentpowerful professional development programs
for data-based decision making Developing a content language for reading
Conclusions Much progress has been made in recognizing the
importance of instructional content as a variable in education research.
Some progress has been made in building tools for including content in education research.
There have been several innovative uses of these new tools in both research and practice, and more are on the horizon.
But there is much more work to be done.
7th-Grade Standards--Close ViewState E State F NCTM
Number Sense and Numeration
7th-Grade Standards--Close ViewState E State F NCTM
Data Analysis and Probability
Response Rates for Survey
Eisenhower LongitudinalWave 1 75%Wave 2 74%Wave 3 75%
Eisenhower Cross-Sectional 72%Reform Up Close 75%
Interrater Agreement
Assessment Mean Range Goals Study .51 .77 to .34 CCSSO Study .47 .60 to .37
Standards Goals Study .48 .59 to .33
[Note: In each study, there was one outlier rater.]
Eisenhower Longitudinal Study
42% of variance explained by level (elementary, middle, high school) and subject
27% of variance explained by teachers in the same school
0% of variance explained by between school or between years
Longitudinal data on instruction alignment to NAEP yielded:
Alignment to Predict Achievement Gains
Index Cross-Sectional Gain
Longitudinal Gain
Level .336 (p=.02) .238 (p=.11) Level and Configuration .425 (p=.01) .341 (p=.07)
Level x Configuration .438 (p=.00) .304 (p=.04)
Cell Correlation .335 (p=.02) .240 (p=.11)
Total Configuration .312 (p=.04) .335 (p=.02)
Index Intercorrelations
L LXC CELL TOTAL
Level 1.0
Level x Configuration .69 1.0
Cell Correlation .62 .72 1.0
Total Configuration .77 .59 .86 1.0