content analysis workshop
DESCRIPTION
Content Analysis Workshop. Minneapolis, MN Nov 7-9, 2007. Describing the Content of. Standards & Assessments. John L. Smithson , Director, Measures of the Enacted Curriculum Alissa Minor , Projects Manager, Measures of the Enacted Curriculum - PowerPoint PPT PresentationTRANSCRIPT
Content Analysis Workshop
John L. Smithson, Director, Measures of the Enacted CurriculumAlissa Minor, Projects Manager, Measures of the Enacted Curriculum
Wisconsin Center for Education Research, University of [email protected]
Minneapolis, MN Nov 7-9, 2007
Describing the Content of
Standards & Assessments
The Goal
To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.
The Goal
To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.
Content Descriptions
ContentDescriptions
AlignmentAnalyses
NeedsAssessment
ProgramEvaluation
MonitoringChange
CurriculumManagement
TeacherReports
ContentAnalyses
SECTaxonomy
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.
• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments
Analytic: (Alignment)
To Describe Instructional Content
SEC utilizes a two-dimensional taxonomy based on:Topic
by Cognitive Demand
1
2
3
4
5
B C D E F
Categories of Cognitive Demand
Topics Memorize Conduct
Investigations Communicate Understanding
Analyze Information
Apply / Make Connections
Nature of Science
Science & Technology
Science, Health, Env.
Meas. & Calc. In Sci.
Comp. Of Living
Systems
Botany
The Content Matrix
Categories of Cognitive Demand
Topics Memorize Conduct
Investigations Communicate Understanding
Analyze Information
Apply / Make Connections
Nature of Science
Science & Technology
Science, Health, Env.
Meas. & Calc. In Sci.
Comp. Of Living
Systems
Botany
… adding levels of relative emphasis yields a 3-D construct
MemorizePerform
CommunicateConjecture
Connect0
0.020.040.060.08
0.1
Operations
Number Sense
Measurement
Geometric Concepts
Algebraic Concepts
Data Analysis
Instructional Technology
MemorizePerform
CommunicateConjecture
Connect
State J Grade 8Mathematics Assessment
Content Map Data Displays
To Facilitate Comparisons
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning.
• Predict student achievement gains• Control for content to examine other factors• As an outcome measure for change over time• Examine alignment of Standards & Assessments
Analytic: (Alignment)
Alignment as a Quantity
1.000.00 0.500.25 0.75
0.27 (Avg. Alignment: Test to Standard)
Range of Alignment: Test to Standard)
(Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003)
0.14 0.40
Aligning Tests to Standards
State U Grade 8 Mathematics Alignment: Test to Standard (0.23)
Instructional Alignment
1.000.00 0.500.25 0.750.03 0.31
Avg.0.05 0.29
Based upon results for 168 teachers, across 3 states: MSP PD Study 2004
Fine Grain
Min. Max.0.17
Avg.Min. Max.
Instruction to Standards & Assessments
Standards
Assessments
0.19
Time 0 Time 1 Time 2
Time 0 Time 1 Time 2
Learning Gains by Course Type
Learning Gains Controlling for Content
RegentsAlgebraStretch RegentsMath A/B/UCSMPGen. Mth. / Pre-alg.
From:Upgrading High School Mathematics Instruction,(Gamoran, Porter, Smithson, & White, 1997),EEPAv19n4
12
11
10.5
12
9.5
10
11.5
12
11
10.5
12
9.5
10
11.5
Explaining variation in student learning gains
Alignment Index:Instruction to Standards
MathematicsAcross 4 Districts
Alignment Analyses for School Improvement
Using alignment as an outcome measure
(Measuring change in alignment over time)
Treatment 99
Control 124
Leaders 16
Counts
Content Analysis Procedures
Exploring theDimensions of Content
But first….
Let’s take a 10 minute break!
The Two Dimensions Of Content
What students should know[Topics]
And…
Be Able to Do [Expectations for student performance]
Describing the Cognitive Domain
3
4
5
6
How Many Categories?
Bloom’s
SEC
DOK (Webb)
SCASS Science
Dimensions of Knowing & Inquiry
Acquire Use Extend
(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student StandardsScience Project, Council of Chief State School Officers, 1997)
Depth of Knowledge
* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.
Skill/Concept
Recall
StrategicThinking
ExtendedThinking
Level
1
2
3
4
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
Exploring Cognitive Demand
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
Understanding
Conjecture,Generalize
Prove
Solve non-routine/ makeconnections
MemorizeConduct
InvestigationsCommunicateUnderstanding
AnalyzeInformation
Apply concepts/make
connections
RecallPerform
ProceduresAnalyze/
InvestigateEvaluate
Generate/Demonstrate
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
UnderstandingAnalyze
InformationEvaluate/Apply
Recall Understanding Application Analyze Evaluate Create
Exploring Cognitive Demand
CgD Immersion Activity• Organize into Groups/Tables• Each Table w/ CgD Pie• Each Person w/ Cgd Descriptors
Step 1: Place CgD cards on Pie Slices face-down
Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same sliceif Group Agrees ... discuss key wordsif not … Discuss … operational definition to distinguish
Step 3: Discuss disagreementsif consensus reached put in envelope / if not, set aside
Content Analysis Materials
• Cognitive Demand List• Topics Lists• Comments & Suggestions Worksheet• Coding Forms• Documents to be analyzed
Cognitive Demand Lists
• Five categories of cognitive demand• Slightly different for each subject• Each category is defined by a list of descriptors• The list of descriptors are not exhaustive
• Each category stands on its own• Each category has an associated letter (B-F)
Dimension A: Content Topics
Topics List (In your packet of material)
Organized at two levels:Content Areas
(16 for Mathematics)(27 for Science)(14 for ELAR)
Topics (identified by number)(182 Mathematics Topics)(211 Science Topics)(114 ELAR Topics)
Plus: non-specific & other
Topics Lists
• Topics Lists• Mathematics• Science• English Language Arts & Reading
• Cover grades K-12• Organized into Content Areas• Topics & Content Areas have an associated #
Comments & Suggestion Worksheet
• One for each reviewer - more available• Use to:
• Record coding conventions/decision rules• Suggest/identify additional topics not listed• Suggest/identify additional CGD descriptors• Provide other comments & suggestions
• Be sure to turn in at end of workshop (and with mail-in materials, as necessary).
Coding Forms
• Assessment Coding Forms• Benchmark Coding Forms
• Each is used to record content descriptions
• Each content description consists of• A topic number• A cognitive demand category letter
Rater: ________ Subject:__NC Math Gr 3 Benchmark Test (SF)_ Form: Test 1 Page of
Itm. Desig./Nbr. Itm. Desig./Nbr.TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1 40
2 41
3 42
4 43
5 44
6 45
7 46
8 47
9 48
10 49
11 50
12 51
13 52
14 53
15 54
16 55
17 56
18 57
19 58
20 59
21 60
22 61
23 62
24 63
25 64
26 65
27 66
28 67
29 68
30 69
31 70
32 71
33 72
34 73
35 74
36 75
37 76
38 77
39 78
Content Code 2 Content Code 3Content Code 1 Content Code 2 Content Code 3 Content Code 1
Assessment Coding Forms
Rater: ________ Document:__NAEP Grade 8____Itm. Desig./Nbr. Itm. Desig./Nbr.
Number PropertiesNumber Sense TPC1 CGD1 TPC2 CGD2 TPC3 CGD31a 4a
1b 4b
1d 4c
1e 4d
1f
1g 5a
1i 5b
1j 5c
Estimation TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 5d2a
5e2b
5f2c
Measurement
2d Measuring Physical Attributes TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1bNumber Operations TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
3a 1c
3d 1g
3e 1h
3f 1j
3g 1k
Properties of Number & Operations
TPC1 CGD3
CGD3CGD1 TPC2
CGD1 TPC2 CGD2 TPC3
CGD2 TPC3
Content Code 1 Content Code 2 Content Code 3 Content Code 1 Content Code 2 Content Code 3Ratios & Proportional Reasoning
TPC1
Standards Coding Forms
Standards Coding Forms
Coding Procedures
TPC CGDContent Descriptions arerecorded using pairs ofcodes to identifyparticular intersectionsTopic & CognitiveDemand Categories.
B200203
C
Cognitive Demand Categories arerepresented by letters B-F.
Topics are identifiedusing numbers.
D290
TPC CGD
Large Grain Topic Descriptions:To reference vague content descriptions thatcould cover most topics in a given contentarea, use the content area code (ends in 00)to indicate the relevant content area. (Thereare 7 content areas in K-8 Mathematics and25 content areas in K-8 Science.)
200Grain Size can berepresented using thefollowing conventions:
Coding Conventions
Should none of five categories of cognitivedemand listed on the Cognitive DemandCoding Sheet appear relevant, (e.g. thestandard statement makes no reference tocognitive demand, use the letter ‘Z’. Thisshould only occur rarely, if at all.
Non-SpecificCognitive Demand:
Coding Conventions
TPC CGD200 Z
Coding Conventions
Non-specific Content TPC CGD
0
Z
999 Z
Non-specific target subject 0
EOut of subject area
Cognitive Demand only
(*Please make note on your comments form.)
Non-specific Topics: TPC CGDB290
293 DOther: (rater-defined*)
Other: (topic not listed)
Coding Conventions
Practice Coding Exercise
Content Analyzing Assessments(Three code maximum)
Practice Coding Exercise
Content Analyzing Standards(Six code maximum)
The Content Analysis Process
‘Coding’ Teams of 4-5 Content Experts
Independent Coding by each Analyst w/ Group Discussion
Goal for Process: Generalizability not Inter-rater Reliability
Should not be necessary to discuss every item – select by team
Pick-up and return documents / coding sheets to Alissa
Sign & return to Alissa non-disclosure forms
A neutral content language
Topics by Cognitive Demand
The intended curriculum: State content standards—What students should learn
The enacted curriculum: What teachers teach
The learned curriculum: Student outcomes based on school learning
The assessed curriculum: State (and other) assessments—tested learning
Content Analysis Workshop