progress monitoring of oral language using the narrative language measures
DESCRIPTION
Progress Monitoring of Oral Language Using the Narrative Language Measures. Douglas B. Petersen, PhD, CCC-SLP University of Wyoming. Objectives. Describe the key features of a curriculum-based measure Explain why narrative assessment is suitable for progress monitoring - PowerPoint PPT PresentationTRANSCRIPT
Progress Monitoring of Oral Language Using the Narrative Language Measures
Douglas B. Petersen, PhD, CCC-SLPUniversity of Wyoming
Objectives
• Describe the key features of a curriculum-based measure• Explain why narrative assessment is suitable for progress
monitoring• Describe the purposes of the NLM• Administer and score different subtests of the NLM• Describe how the NLM can be integrated into a multi-tiered
system of support for language
Defining the Problem
• "How we define a problem usually determines how we analyze it. It sends us in a particular direction.
• And how we analyze a problem—the direction we take —absolutely determines whether we find a solution and what the quality of that solution is.” -Morgan D. Jones
National Assessment of Educational Progress (NAEP, 2011)
Broad View of Reading
Reading
Word recognitionLinguistic Comprehension
Perfetti, 1985
Narrow View of reading
Reading Language
Kamhi, 2009
Word recognition
Linguistic Comprehension
Common Core State Standards
• “Children’s oral language competence is strongly predictive of their facility in learning to read and write…
• listening and speaking vocabulary and even mastery of syntax set boundaries as to what children can read and understand no matter how well they can decode.” (Appendix A, pg. 26)
Problem
Language Difficulty
Reading Comprehension Problems
Academic Failure
Direction - Solution
• To identify students who experience language difficulty so we can intervene and promote academic success as early as possible
• Language Difficulty– Not just a language disability– Includes English language learners– Includes cultural diversity
Multi-tiered System of SupportAs
sess
men
t AssessmentA
sses
smen
t Assessm
ent
A Paradigm Shift
• Purpose has shifted from eligibility and classification to early detection, prevention and progress monitoring.
Not QUALIFY-PLACE-SERVEBut TEST-TEACH-TEST
Purposes of Assessment
• Screening (Universal)– Helps answer “Who is at risk?”
• Diagnosis and Eligibility– Helps answer “Does this student have a disability?”
• Instructional/Intervention Planning– Helps answer “What and how should we teach?”
• Progress Monitoring– Helps answer “Is our intervention working?”and “What, if anything, needs to be adjusted?”
Curriculum Based Measurement
• Specific set of procedures for measuring student academic growth
• Designed to evaluate the effectiveness of instruction
• Primarily used for – screening – progress monitoring
• Often interpreted to mean the same thing as General Outcome Measurement
(Deno, 2003)
CBM Drives RTI
• Screening (Benchmark Assessment)– Universal benchmark screening (e.g., Fall, Winter, Spring)– Students’ scores are compared to a performance criterion– Identifies children who may require intensified services or more
frequent and targeted monitoring
• Progress monitoring– Regular sampling of performance (e.g., weekly, monthly)– Data are used to make decisions about
• Adjustments in intervention• Entry into and exit from tiers• Need for intensive “special” education
Requirements for Curriculum Based Measurement
1. Strong reliability and validity
2. Time efficient/easy to administer/score
3. Alternate forms of standardized tasks
4. Measure socially important outcomes
5. Sensitive to growth due to intervention
-Deno, 2003
Requirements for Language RTI
1. An effective, manualized language curriculum that is easy to use and sufficiently flexible to address the language needs of diverse children
2. Valid, reliable, easy to use, assessment tools with equivalent forms for frequent sampling of students’ language skills and measuring growth over time.
+
Claiming the Job
• “There is an urgent need for a simple, quick, standardized measure of language that can be administered repeatedly…in a manner akin to the continuous ‘dibbeling’ that is used to monitor progress in code learning.”
• “It behooves the profession and research community to claim the job of developing a set of very brief, standardized benchmark measures of vocabulary, grammar, and narrative…”
Ukrainetz, 2006
Narrative Language
• Narrative – an account of causally-related events in temporal order (aka. a story)
• Narrative language is associated with academic achievement.
• Narrative proficiency at age 5 is one of the best predictors of future reading comprehension (Catts, Fey, Tomblin, & Zhang, 2002; Dickinson & McCabe, 2001; Griffin, Hemphill, Camp, & Wolf, 2004.)
Narrative Language
• Narratives are socially important.• Both language comprehension and production
can be measured through narration.• Narration is a authentic, functional integration
of language• Narratives
…can be the intervention target…can be the intervention context
Reading Standards for Literature
Analysis of Narratives
• Story Grammar Analysis– Entire story is the unit of analysis– Story Grammar
• Linguistic Complexity Analysis– Words and clauses are the units of analysis– Linguistic Complexity
Story Grammar
• Story Grammar: Main parts of the story– causally and temporally related
• Reveal the plot or point of the story
• Reinforced and tested in American schools
Character Problem Feeling Action Ending
Setting End Feeling
Story Grammar
Linguistic Complexity
• The storyteller must create a context for the listener almost exclusively through the use of sophisticated, complex language.
Linguistic Complexity
• Similar to written language
– Adverbs (e.g., quickly)– Temporal subordination (e.g., When he was walking…)– Elaborated noun phrases (e.g., The huge dog that was
scary)– Mental & linguistic verbs (e.g., decided, considered)– Conjunctions (e.g., and, but, because) – Dialogue (e.g., He said, “I need a band-aide!”)
Quality Narratives
• A Quality Narrative…– includes important Story Grammar elements that create an
episode (plot)• Initiating Event (problem)• Attempt• Consequence
– includes complex language features that convey causal and temporal relations between key events (Ukrainetz, 2006).
Quality is impacted by components and the clarity of the relations between components
Narrative Language Measures (NLM)
• Three Subtests– Test of Narrative Retell (TNR)– Test of Personal Generation (TPG)– Test of Story Comprehension (TSC; Preschool only)
• Preschool, Kindergarten, First, Second, Third– 25 equivalent stories per grade level• 9 Benchmark stories (3 Fall, 3 Winter, 3 Spring)• 16 Progress Monitoring stories
Equivalent Features of the NLM
• Consistent across:– Story Grammar– Linguistic Complexity (in extreme detail)– Consistent vocabulary complexity with limited
overlapping vocabulary (across stories and across grade levels)
Technical Adequacy
• Because the TNR is a new measure, empirical investigations of its technical adequacy are ongoing.
• The following table outlines the available psychometric characteristics.
Evidence Preschool TNR Kindergarten TNR
Alternate Form Reliability: Bivariate Pearson Correlations
M = .77, p < .001 M = .75, p < .0001
Fidelity of Administration 91% 97%Real-Time Inter-rater Reliability 96% 94%Criterion-Related Evidence of Validity
Renfrew Bus Story: Strong Correlation
Frog Where Are You:No significant difference in MLU, NDW/TNW, Subordination, Story Grammar
Standardized Features of the NLM
• Standardized administration procedures – TNR– TPG– TSC (Preschool only)
• Standardized scoring guidelines– Real time– Audio file– Transcript
Standardized Administration
• Select the appropriate level• Benchmark TNR Purpose: compare children to an age or grade
criterion• Progress Monitoring TNR Purpose: show an individual child’s
language growth over time Materials and Setting
• The booklets are designed to be administered in order• The progress monitoring stories can be administered in a
different order or a story can be skipped. • Stories should never be repeated
Standardized Administration
Examiner says, “I’m going to tell you a story. Please listen carefully. When I’m done you are going to tell me the same story. Are you ready?”
Examiner reads the selected story word for word in a moderate pace with normal inflection.
Examiner says, “Thanks for listening. Now you tell me that story.”
Examiner should wait 10-15 seconds before providing encouragement.
If the child is reluctant, examiner encourages by saying “It’s okay. Just do your best.”
If the child asks for help, examiner says, “I can’t help but you can just tell the parts you remember.”
Examiner listens and makes only neutral comments such as “uh-huh” and “okay.”
Examiner may not repeat the story or any part of it, but can repeat the directions and provide encouragement as needed.
When the child appears to be finished, the examiner says, “Are you finished?”
If the child is not finished, the examiner should let the child continue.
If the child does not tell a story even after encouragement the examiner says, “That’s okay. Thanks for listening to my story.”
TNR:P with Pictures
• Lay out pictures before beginning the story.• As you read the story, point to the picture that
depicts the part of the story.
Time Efficient, Economical
Brief Administration • Three benchmark TNRs take about 5 minutes • A single TNR for progress monitoring takes less
than 2 minutes.
Reduced Scoring Time• Scoring can be done in real-time while the child is
retelling the story. • Scoring can be done by listening to an audio
recording
TNR Scoring in Real-Time
TNR Scoring in Real-Time
• Um, This kid, um Shane was getting ready for school because he was late and he couldn’t find his shirt so then he asked his mom to help him find it and she told him it was on the dryer and he got it and he was happy.
TNR Scoring in Real-Time
12 18
2
4
Some TNR Scoring Guidelines
• While listening to a child retell a story, the examiner cues in on key words or phrases provided in the scoring form.
• To be scored, the child’s utterances do not need to match the samples in the scoring rubric exactly or word for word; however they should be very close and contain the same amount of correct information.
• To earn 2 points, the utterance should be able to stand alone. The scorer should not ‐need to have shared knowledge of the story to correctly interpret the utterance.
• When children say something other than the exact words and phrases contained in the scoring form, examiners should write in what they say.
• It is helpful to underline or write words and phrases as children say them and once the administration is complete, rate the words or phrases as 2, 1, or 0 based on the general scoring guidelines.
Pgs. 7 & 9
Test of Personal Generation (TPG)
• Assesses language using a conversation elicitation technique – (adapted from McCabe & Rollins, 1994).
• Examiner elicits personal stories while playing with a young child in a very natural manner.
• Provide at least 3 opportunities.
• Most difficult narrative task we use.
TPG Variation
• Immediately following a child’s retell narrative (TNR), elicit a personal generation by asking, “Has something like that ever happened to you?”
• Samples a different dimension of their narrative skills in a very efficient manner– Content limitations– Motivation limitations
TPG Transcription
• C (Mm) my mom helped me in the bike.• E Tell me, keep going!• C Because I fall one day.• E Keep going!• C (And) and my mom say, “Just do it like this”. • C “And you go now”.• C (And) and my mom say, “Let’s do a race!” • C (And) and I won.
Benchmark Assessment (TNR)
• Benchmark testing for all students– Use student’s best/median score out of 3 administrations – Use a criterion score (level of performance) to assign level of risk
for each student (e.g., at-risk, some risk, no risk)– Use gain scores (or lack of gain) to identify students who would
benefit from more intensive intervention
Progress Monitoring
What About Diagnosis?
• Limited growth or response to instruction over time is a strong indicator of language impairment.– Chanthongthip, Petersen, Spencer, (2012)
What About Intervention Planning?
• While some tools are not capable of informing instruction, the NLM measures can directly inform a child’s language needs.
• Goals can be derived directly from the results of the NLM assessments.
• Narratives…can be the intervention target…can be the intervention context
Informing Instruction
=2 points =1 point =0 points
Informing Instruction
=2 points =1 point =0 points
Overview of Multi-tiered
Narrative Language
Intervention
Multi-tiered Language Instruction
• Multi-tiered Curriculum– Manualized– Flexible
• Key Features– Carefully structured stories– Engaging visual materials– Explicit teaching procedures– Fun and motivating
Engaging Visual Material
• Pictures for each story – 5 panels
• Story Grammar Icons– Character– Setting– Problem– Feeling
– Action– Ending– End Feeling
Character Problem Feeling Action Ending
Setting End Feeling
Story Champs - Tiers
• Large Group• Small Group• Individual
• Intensity increases as group size decreases– More opportunities to respond – More explicit and individualized targets – More specialized prompting – More contingent feedback– More frequent sessions– Increasing duration– More reliance on instructors with greater
expertise
Teaching Procedures
• Based on the effective teaching literature and principles of instruction– Frequent opportunities to respond– Explicit and individualized targets– Systematic scaffolding– Corrections– Least restrictive prompting– Curriculum-based measurement
Test of Narrative RetellM
ean
Score
s
Group x TimeF(2, 68) = 3.6, p = .03
d= .80
F(1, 68) = 4.1, p = .046, d = 0.49
F(1, 68) = 4.45, p = .039 d = 0.51
ANCOVA
Tota
l Sco
res
50-minutes Individualized Intervention: Treatment & Control Groups
Causality: Language Impairment Story Grammar: Language Impairment
Pretest Posttest0
0.5
1
1.5
2
2.5
3
3.5
4
Pretest Posttest0
0.20.40.60.8
11.21.41.61.8
2
Pretest Posttest0
0.5
1
1.5
2
2.5
3
3.5
4
Story Grammar: Typical LanguageCausality: Typical Language
0
Treatment GroupControl Group
Pretest Posttest0
0.5
1
1.5
2
LARGE GROUP
Systematic ScaffoldingSignalsStory Gestures
LARGE GROUPSystematic ScaffoldingSignalsStory GesturesQuestioningIndividual Turns
LARGE GROUP
Systematic ScaffoldingSignalsStory GesturesQuestioningIndividual TurnsLeast Restrictive PromptsCorrections
SMALL GROUPSeating/RotationPersonal Stories
Story Games
Scaffolding
CorrectionsPromptingDifferentiationVariations
INDIVIDUALPersonal Stories
Pictography
Scaffolding
Over-correctionsPromptingIndividualize Targets
Big Picture Recommendations
For more information, contact:
Doug Petersen: [email protected] Spencer: [email protected]