assessing the instructional level for writing david parker, kristen mcmaster, and matthew burns
TRANSCRIPT
Assessing the Instructional Level for Writing
David Parker, Kristen McMaster, and Matthew Burns
Activity
1. Topic:– White: Describe why nuclear fission has
been easier to do than nuclear fusion.– Peach: Describe why this conference will
be useful for your practice.– Pink: Describe the events of your last
family vacation.
2. Pencils down: Think for 30 seconds3. Write!!!
Activity
• Count # Words Written
• Results:Sample White Peach Pink
1
2
3
4
5
Average
Activity
• The Findings:– Did the Peach and Pink writers write more?– Who was more on-task?
• Why’d we do it?– Simulate the right amount of challenge
• Think of the kids!!!
Overview
1. Introduction– Why writing?– Why instructional level?– Purpose of this study
2. Method– Who, what, how?
3. Results– What was found
4. Discussion– Why it matters, limitations, what next?
IntroductionWhy Writing??
National Report Cards on Writing, 2003; 2008
Introduction
Why does writing proficiency matter?
– Enhances learning in content area courses (Bangert-Drowns, Hurley, & Wilkinson, 2004).
– College Entrance, Job Obtainment/Performance (National Commission on Writing,
2004; 2005).
Introduction
Problem: Detecting writing problems in late elementary or middle school, makes it more difficult to remediate (Baker,
Gersten, & Graham, 2003)
Solution: Start Intervening Early!!!!
IntroductionEnter the Instructional Level!!!
First, some background knowledge
IntroductionWhat is the Instructional Level?
Introduction• Theoretical Foundation
– Vykotsky (1978)– Betts (1946)– Gravois & Gickling (BP-V; 2008)
• Measurement Tools– Curriculum-based Measurement (CBM; Deno, 1985; Marston, 1989)
• For early writers (Coker & Ritchey, 2009; McMaster, Du, Yeo, Deno, Parker, & Ellis, 2009)
• Assessment – Curriculum-based Assessment (Gickling &
Havertape, 1981; Gickling, Shane, & Croskery, 1989)
IntroductionEmpirical Findings
• Reading:– 93-97% correctly read words (Treptow, McComas, & Burns, 2007; Gickling
& Armstrong, 1978)
• Improved on-task behavior, task completion, and reading comprehension
• 4x Faster growth rates (Burns, 2007)
• Math:– 14-31 Correct Digits (2/3rd Graders); 24-49 Correct
Digits (4/5th Graders)• Highest growth slopes (Burns, VanDerHeyden, & Jiban, 2006).
Introduction
There is NO Instructional Level for writing!
Purpose: To identify potential estimates of the instructional level for writing.
MethodParticipants
– 5 classrooms from 2 urban schools– 85 1st grade students
• 51% male
• 41% White; 28% Black; 26% Hispanic
• 57% Free/Reduced Lunch
• 17% special education services
Setting– Classrooms
MethodMeasures1.Curriculum-based Measurements
– Two Types1. Picture-Word2. Sentence Copy
– Scoring Procedures1. Words Written2. Words Spelled Correctly3. Correct Word Sequences
2.Test of Written Language
Method
Picture-Word Prompt (McMaster, Du, & Petursdottir, 2009)
Method
Sentence-Copy Prompt (McMaster, Du, & Petursdottir, 2009)
MethodProcedure
– Weekly progress monitoring data• 12 weeks• Teacher-administered• Students practiced then completed prompts for 3
minutes
Fidelity and Agreement– Collected for teacher administration as well as
prompt scoring – Teacher administration fidelity: 100%– Agreement: generally > 90%
MethodData Analysis (an 8-step plan)1.Establish Reliability of Accuracy/Fluency Metrics
2.Establish Validity of Promising Metrics
3.Compute Growth Slopes
4. Identify top 1/3rd Slopes
5.Compute Mean Start for top 1/3rd Slops
6.Create Categories
7.Establish Reliability of Categories
8.Establish Validity of Categories
Part 1: Find promising measures and scoring procedures
MethodData Analysis (an 8-step plan)1.Establish Reliability of Accuracy/Fluency Metrics
2.Establish Validity of Promising Metrics
3.Compute Growth Slopes
4. Identify top 1/3rd Slopes
5.Compute Mean Start for top 1/3rd Slopes
6.Create Categories
7.Establish Reliability of Categories
8.Establish Validity of Categories
Part 2: Find Instructional Levels
MethodData Analysis (a 8-step plan)1.Establish Reliability of Accuracy/Fluency Metrics
2.Establish Validity of Promising Metrics
3.Compute Growth Slopes
4. Identify top 1/3rd Slopes
5.Compute Mean Start for top 1/3rd Slops
6.Create Categories
7.Establish Reliability of Categories
8.Establish Validity of Categories
Part 3: Examine promise of instructional levels
Table 1. Means, Standard Deviations, and Correlation Coefficients for Fluency and Accuracy Scores for Sentence Copy and Picture-Word Prompts and Accompanying Scoring Procedures.
Fluency Accuracy
Probe 2 Probe 3 Probe 2 Probe 3
Prompt
ProcedureM SD M SD r M SD M SD r
Picture-Word
Words Written 17.0 8.4 18.4 8.6 .71*
Words Spelled Correctly 13.4 7.7 15.0 8.6 .67* 76.1 23.4 77.8 24.6 .52*
Correct Word Sequences 11.9 8.6 13.1 9.1 .67* 54.2 28.6 55.9 26.5 .46*
Sentence Copy
Words Written 16.7 7.1 16.8 7.7 .71*
Words Spelled Correctly 12.8 6.6 13.3 7.1 .74* 74.6 25.1 78.8 19.7 .60*
Correct Word Sequences 11.9 7.3 12.6 8.1 .70* 59.8 29.6 64.6 26.7 .56*
Correlation with TOWL-3 Total
Prompt
Procedure
Fluency Raw
r1
Category
ρ1
Picture-Word
Words Written .32* .36*
Words Spelled Correctly .48* .46*
Correct Word Sequences .52* .50*
Sentence Copy
Words Written .26 .21
Words Spelled Correctly .42* .46*
Correct Word Sequences .46* .48*
Table 2. Criterion-related Validity Coefficients between Scoring Procedures for Each Prompt and the Test of Written Language-3 (TOWL-3) Total Score.
Prompt
ProcedureMean SD SE
Fluency Criteria
(3 minute probe)
Picture-Word
Words Written 14.46 8.68 1.64 11-18
Words Spelled Correctly 11.43 7.03 1.33 9-14
Correct Word Sequences 10.93 8.56 1.62 8-14
Sentence Copy
Words Written 16.25 6.58 1.24 14-19
Words Spelled Correctly 13.39 6.52 1.23 11-16
Correct Word Sequences 13.32 7.86 1.49 10-16
Table 3. Derivation of and Estimates for Fluency Instructional Level Criteria for Scoring Procedures within Prompt Types.
Probe 2 Probe 3
Frustration Instructional Independent Frustration Instructional Independent.
κPrompt-Procedure
N % N % N % N % N % N %
Picture-Word
Words Written 19 23.8 19 23.8 42 52.5 21 25.3 15 18.1 47 56.6 .46*
Words Spelled Correctly 24 30.0 18 22.5 38 47.5 23 27.7 12 14.5 48 57.8 .46*
Correct Word Sequences 30 37.5 20 25.0 30 37.5 29 34.9 17 20.5 37 44.6 .46*
Sentence Copy
Words Written 24 29.6 30 37.0 27 33.3 23 28.8 24 30.0 33 41.2 .37*
Words Spelled Correctly 29 35.8 24 29.6 28 34.6 29 36.2 28 35.0 23 28.8 .46*
Correct Word Sequences 32 39.5 25 30.9 24 29.6 31 38.8 20 25.0 29 36.2 .47*
Table 4. Number and Percentage of Fluency Scores Categorized as Frustration, Instructional, and Independent and Kappa Coefficients.
Correlation with TOWL-3 Total
Prompt
Procedure
Fluency Raw
r1
Category
ρ1
Picture-Word
Words Written .32* .36*
Words Spelled Correctly .48* .46*
Correct Word Sequences .52* .50*
Sentence Copy
Words Written .26 .21
Words Spelled Correctly .42* .46*
Correct Word Sequences .46* .48*
Table 2. Criterion-related Validity Coefficients between Scoring Procedures for Each Prompt and the Test of Written Language-3 (TOWL-3) Total Score.
Discussion
Conclusion:– Consistent with previous research for reading
(Burns, 2007; Gickling & Armstrong, 1978) and math (Burns, VanDerHeyden, &
Jiban, 2006), criteria are plausible that indicate a student will make optimal growth in writing skill.
Implications:– MORE research!– Instructional decision-making
DiscussionLimitations
– Conceptual issues• CBM (General Outcome Measure) vs. CBA (Specific Subskill
Measure)?• Material difficulty?
– Generalizability?? (only 1st graders?)– Criterion for “high-responders”– Ongoing research with early CBM-Ws
Future Research– Investigate effects of instructional level prospectively (vis.
Intervention)?– Which measure is most informative?– Appropriate criteria?
Questions?
