effective formative evaluation: the underlying logic…

Response to Intervention

www.interventioncentral.org

Formative Assessment: Specific Tools to Measure Student Academic Skills and Behavior

Jim Wrightwww.interventioncentral.org



Effective Formative Evaluation: The Underlying Logic…

1. What is the relevant academic or behavioral outcome measure to be tracked?

2. Is the focus the core curriculum or system, subgroups of underperforming learners, or individual struggling students?

3. What method(s) should be used to measure the target academic skill or behavior?

4. What goal(s) are set for improvement?

5. How does the school check up on progress toward the goal(s)?


www.interventioncentral.org 3

Use Time & Resources Efficiently By Collecting Information Only on ‘Things That Are Alterable’

“…Time should be spent thinking about things that the intervention team can influence through instruction, consultation, related services, or adjustments to the student’s program. These are things that are alterable.…Beware of statements about cognitive processes that shift the focus from the curriculum and may even encourage questionable educational practice. They can also promote writing off a student because of the rationale that the student’s insufficient performance is due to a limited and fixed potential. “ p.359

Source: Howell, K. W., Hosp, J. L., & Kurns, S. (2008). Best practices in curriculum-based evaluation. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp.349-362). Bethesda, MD: National Association of School Psychologists.



School Instructional Time: The Irreplaceable Resource

“In the average school system, there are 330 minutes in the instructional day, 1,650 minutes in the instructional week, and 56,700 minutes in the instructional year. Except in unusual circumstances, these are the only minutes we have to provide effective services for students. The number of years we have to apply these minutes is fixed. Therefore, each minute counts and schools cannot afford to support inefficient models of service delivery.” p. 177

Source: Batsche, G. M., Castillo, J. M., Dixon, D. N., & Forde, S. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 177-193).



Summative data is static information that provides a fixed ‘snapshot’ of the student’s academic performance or behaviors at a particular point in time. School records are one source of data that is often summative in nature—frequently referred to as archival data. Attendance data and office disciplinary referrals are two examples of archival records, data that is routinely collected on all students. In contrast to archival data, background information is collected specifically on the target student. Examples of background information are teacher interviews and student interest surveys, each of which can shed light on a student’s academic or behavioral strengths and weaknesses. Like archival data, background information is usually summative, providing a measurement of the student at a single point in time.



Formative assessment measures are those that can be administered or collected frequently—for example, on a weekly or even daily basis. These measures provide a flow of regularly updated information (progress monitoring) about the student’s progress in the identified area(s) of academic or behavioral concern.

Formative data provide a ‘moving picture’ of the student; the data unfold through time to tell the story of that student’s response to various classroom instructional and behavior management strategies. Examples of measures that provide formative data are Curriculum-Based Measurement probes in oral reading fluency and Daily Behavior Report Cards.



Formal Assessment Defined

“Formative assessment [in academics] refers to the gathering and use of information about students’ ongoing learning by both teachers and students to modify teaching and learning activities. …. Today…there are compelling research results indicating that the practice of formative assessment may be the most significant single factor in raising the academic achievement of all students—and especially that of lower-achieving students.” p. 7

Source: Harlen, W. (2003). Enhancing inquiry through formative assessment. San Francisco, CA: Exploratorium. Retrieved on September 17, 2008, from http://www.exploratorium.edu/ifi/resources/harlen_monograph.pdf



Formative Assessment: Essential Questions…


Problems identified for formative assessment should be:1. Important to school stakeholders.2. Measureable & observable. 3. Stated positively as ‘replacement behaviors’ or goal statements

rather than as general negative concerns (Bastche et al., 2008).

4. Based on a minimum of inference (T. Christ, 2008).

Source: Batsche, G. M., Castillo, J. M., Dixon, D. N., & Forde, S. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 177-193).Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176).



Academic or Behavioral Targets Are Stated as ‘Replacement Behaviors’

“The implementation of successful interventions begins with accurate problem identification. Traditionally, the student problem was stated as a broad, general concern (e.g., impulsive, aggressive, reading below grade level) that a teacher identified. In a competency-based approach, however, the problem identification is stated in terms of the desired replacement behaviors that will increase the student’s probability of successful adaptation to the task demands of the academic setting.” p. 178

Source: Batsche, G. M., Castillo, J. M., Dixon, D. N., & Forde, S. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 177-193).



Inference: Moving Beyond the Margins of the ‘Known’

“An inference is a tentative conclusion without direct or conclusive support from available data. All hypotheses are, by definition, inferences. It is critical that problem analysts make distinctions between what is known and what is inferred or hypothesized….Low-level inferences should be exhausted prior to the use of high-level inferences.” p. 161

Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176).



Examples of High vs. Low Inference Hypotheses

High-Inference Hypothesis. The student has an auditory processing issue that prevents success in reading. The student requires a multisensory approach to reading instruction to address reading deficits. Known

Unknown

Low-Inference Hypothesis. The student needs to build reading fluency skills to become more proficient in decoding. Known

Unknown

The results of grade-wide benchmarking in reading show that a target 2nd-grade student can read aloud at approximately half the rate of the median child in the grade.



Adopting a Low-Inference Model of Reading Skills

Source: Source: Big ideas in beginning reading. University of Oregon. Retrieved September 23, 2007, from http://reading.uoregon.edu/index.php

5 Big Ideas in Beginning Reading

1. Phonemic Awareness

2. Alphabetic Principle

3. Fluency with Text

4. Vocabulary

5. Comprehension





Apply the ‘80-15-5 ‘Rule (T. Christ, 2008) :– If fewer than 80% of students are successfully meeting academic or behavioral

goals, the formative assessment focus is on the core curriculum and general student population.

– If no more than 15% of students are not successful in meeting academic or behavioral goals, the formative assessment focus is on small-group ‘treatments’ or interventions.

– If no more than 5% of students are not successful in meeting academic or behavioral goals, the formative assessment focus is on the individual student.

Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176).



RTI Literacy: Assessment & Progress-MonitoringTo measure student ‘response to instruction/intervention’ effectively, the RTI Literacy model measures students’ reading performance and progress on schedules matched to each student’s risk profile and intervention Tier membership.

• Benchmarking/Universal Screening. All children in a grade level are assessed at least 3 times per year on a common collection of literacy assessments.

• Strategic Monitoring. Students placed in Tier 2 (supplemental) reading groups are assessed 1-2 times per month to gauge their progress with this intervention.

• Intensive Monitoring. Students who participate in an intensive, individualized Tier 3 reading intervention are assessed at least once per week.

Source: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge.



Using Local Norms in Coordination with Benchmark Data



Baylor Elementary School : Grade Norms: Correctly Read Words Per Min : Sample Size: 23 StudentsGroup Norms: Correctly Read Words Per Min: Book 4-1: Raw Data31 34 34 39 41 43 52 55 59 61 68 71 74 75 85 89 102 108 112 115 118 118 131

LOCAL NORMS EXAMPLE: Twenty-three 4th-grade students were administered oral reading fluency Curriculum-Based Measurement passages at the 4th-grade level in their school.

In their current number form, these data are not easy to interpret.

So the school converts them into a visual display—a box-plot —to show the distribution of scores and to convert the scores to percentile form.

When Billy, a struggling reader, is screened in CBM reading fluency, he shows a SIGNIFICANT skill gap when compared to his grade peers.



Baylor Elementary School : Grade Norms: Correctly Read Words Per Min : Sample Size: 23 Students January Benchmarking

Low Value=31 Hi Value=131

Median (2nd Quartile)=71

3rd Quartile=1081st Quartile=43

Billy=19

Group Norms: Correctly Read Words Per Min: Book 4-1: Raw Data31 34 34 39 41 43 52 55 59 61 68 71 74 75 85 89 102 108 112 115 118 118 131

0 20 40 60 80 100 120 140 160Correctly Read Words-Book 4-1

Group Norms: Converted to Box-Plot

Source: Tindal, G., Hansbrouck, J., &

Jones, C. (2005).Oral reading fluency: 90

years of measurement

[Technical report #33]. Eugene, OR:

University of Oregon.

National Reading Norms: 112 CRW Per Min



Team Activity: Formative Assessment and Your SchoolsAt your tables, discuss:

• What kinds of formative measures your schools tend to collect most often.

• How ‘ready’ your schools are to collect, interpret, and act on formative assessment data..





Formative assessment methods should be as direct a measure as possible of the problem or issue being evaluated. These assessment methods can:

– Consist of General Outcome Measures or Specific Sub-Skill Mastery Measures

– Include existing (‘extant’) data from the school systemCurriculum-Based Measurement (CBM) is widely used to track basic student academic skills. Daily Behavior Report Cards (DBRCs) are increasingly used as one source of formative behavioral data.

Source: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge.



Formal Tests: Only One Source of Student Assessment Information

“Tests are often overused and misunderstood in and out of the field of school psychology. When necessary, analog [i.e., test] observations can be used to test relevant hypotheses within controlled conditions. Testing is a highly standardized form of observation. ….The only reason to administer a test is to answer well-specified questions and examine well-specified hypotheses. It is best practice to identify and make explicit the most relevant questions before assessment begins. …The process of assessment should follow these questions. The questions should not follow assessment. “ p.170

Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176). Bethesda, MD: National Association of School Psychologists.



Making Use of Existing (‘Extant’) Data



Extant (Existing) Data (Chafouleas et al., 2007)

• Definition: Information that is collected by schools as a matter of course.

• Extant data comes in two forms:– Performance summaries (e.g., class grades, teacher

summary comments on report cards, state test scores).

– Student work products (e.g., research papers, math homework, PowerPoint presentation).

Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press.



Advantages of Using Extant Data (Chafouleas et al., 2007)

• Information is already existing and easy to access.

• Students will not show ‘reactive’ effects when data is collected, as the information collected is part of the normal routine of schools.

• Extant data is ‘relevant’ to school data consumers (such as classroom teachers, administrators, and members of problem-solving teams).




Drawbacks of Using Extant Data (Chafouleas et al., 2007)

• Time is required to collate and summarize the data (e.g., summarizing a week’s worth of disciplinary office referrals).

• The data may be limited and not reveal the full dimension of the student’s presenting problem(s).

• There is no guarantee that school staff are consistent and accurate in how they collect the data (e.g., grading policies can vary across classrooms; instructors may have differing expectations regarding what types of assignments are given a formal grade; standards may fluctuate across teachers for filling out disciplinary referrals).

• Little research has been done on the ‘psychometric adequacy’ of extant data sources.




Curriculum-Based Measurement: Assessing Basic Academic Skills



Curriculum-Based Measurement: Advantages as a Set of Tools to Monitor RTI/Academic Cases

• Aligns with curriculum-goals and materials• Is reliable and valid (has ‘technical adequacy’) • Is criterion-referenced: sets specific performance levels for specific

tasks• Uses standard procedures to prepare materials, administer, and

score• Samples student performance to give objective, observable ‘low-

inference’ information about student performance • Has decision rules to help educators to interpret student data and

make appropriate instructional decisions• Is efficient to implement in schools (e.g., training can be done quickly;

the measures are brief and feasible for classrooms, etc.)• Provides data that can be converted into visual displays for ease of

communicationSource: Hosp, M.K., Hosp, J. L., & Howell, K. W. (2007). The ABCs of CBM. New York: Guilford.



CBM Student Reading Samples: What Difference Does Fluency Make?

• 3rd Grade: 19 Words Per Minute





• Phonemic awareness skills• Reading fluency• Reading comprehension• Early math skills• Math computation• Math applications & concepts• Writing• Spelling

CBM techniques have been developed to assess:



CBM Math Measures: Selected Sources• AimsWeb (http://www.aimsweb.com)• Easy CBM (http://www.easycbm.com)• iSteep (http://www.isteep.com)• EdCheckup (http://www.edcheckup.com)• Intervention Central (http://www.interventioncentral.org)



Measuring General vs. Specific Academic Outcomes

• General Outcome Measures: Track the student’s increasing proficiency on general curriculum goals such as reading fluency. Example: CBM-Oral Reading Fluency (Hintz et al., 2006).

• Specific Sub-Skill Mastery Measures: Track short-term student academic progress with clear criteria for mastery (Burns & Gibbons, 2008). Example: Letter Identification.

Sources: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. Hintz, J. M., Christ, T. J., & Methe, S. A. (2006). Curriculum-based assessment. Psychology in the Schools, 43, 45-56.



Monitoring Student Academic Behaviors:Daily Behavior Report Cards



Daily Behavior Report Cards (DBRCs) Are…

brief forms containing student behavior-rating items. The teacher typically rates the student daily (or even more frequently) on the DBRC. The results can be graphed to document student response to an intervention.



Daily Behavior Report Cards Can Monitor…

• Hyperactivity• On-Task Behavior (Attention)• Work Completion• Organization Skills• Compliance With Adult Requests• Ability to Interact Appropriately With Peers



Daily Behavior

Report Card: Daily

Version

Jim Blalock May 5Mrs. Williams Rm 108



Daily Behavior Report

Card: Weekly Version

40 0 60 60 50

Jim BlalockMrs. Williams Rm 108

05 05 07 05 06 07 05 07 07 05 08 07 05 09 07



Daily Behavior Report Card: Chart





Goals are defined at the system, group, or individual student level. Goal statements:

– Are worded in measureable, observable terms,– Include a timeline for achieving those goals.– Are tied to the formative assessment methods used to monitor progress toward

the goal(s).



IEP Goal Statements for CBA/CBM



Writing CBM Goals in Student IEPs (Wright, 1992)

Source: Wright, J. (1992). Curriculum-based measurement: A manual for teachers. Retrieved on September 4, 2008, from http://www.jimwrightonline.com/pdfdocs/cbaManual.pdf



IEP Goals for CBA/CBM: READING

Reading In [number of weeks until Annual Review], when given a randomly selected passage from [level and name of reading series] for 1 minute

Student will read aloud

At [number] correctly read words with no more than [number] decoding errors.



IEP Goals for CBA/CBM: Written Expression

Written Expression

In [number of weeks until Annual Review], when given a story starter or topic sentence and 3 minutes in which to write

Student will write

A total of:[number] of words or[number] of correctly spelled words or [number] of correct word/writing sequences



IEP Goals for CBA/CBM: Spelling

Spelling In [number of weeks until Annual Review], when dictated randomly selected words from [level and name of spelling series or description of spelling word list] for 2 minutes

Student will write

[Number of correct letter sequences]



Interpreting Data: The Power of Visual Display



Sample Peer Tutoring Chart



Single-Subject (Applied) Research Designs

“Single-case designs evolved because of the need to understand patterns of individual behavior in response to independent variables, and more practically, to examine intervention effectiveness. Design use can be flexible, described as a process of response-guided experimentation…, providing a mechanism for documenting attempts to live up to legal mandates for students who are not responding to routine instructional methods.” p. 71

Source: Barnett, D. W., Daly, E. J., Jones, K. M., & Lentz, F.E. (2004). Response to intervention: Empirically based special service decisions from single-case designs of increasing and decreasing intensity. Journal of Special Education, 38, 66-79.



Jared: Intervention Phase 1: Weeks 1-6

1 131 17

1 201 24

1 271 31

2 32 7

2 102 14

2 242 28

3 33 7

3 103 14

3 173 21

3 243 28

3 314 4

4 74 11

4 144 18

XX

W 1/2271 CRW

W 1/2977 CRW

M 2/375 CRW

Th 2/1375 CRW

Th 2/2779 CRW

F 3/782 CRW



Formative Assessment: Donald: Grade 3




5. How does the school check up on progress toward the goal(s)?The school periodically checks the formative assessment data to determine whether the goal is being attained. Examples of this progress evaluation process include the following:

– System-Wide: A school-wide team meets on a monthly basis to review the frequency and type of office disciplinary referrals to judge whether those referrals have dropped below the acceptable threshold for student behavior.

– Group Level: Teachers at a grade level assembles every six weeks to review CBM data on students receiving small-group supplemental instruction to determine whether students are ready to exit (Burns & Gibbons, 2008).

– Individual Level: A building problem-solving team gathers every eight weeks to review CBM data to a student’s response to an intensive reading fluency plan.

Sources: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge.

Shinn, M. R. (1989). Curriculum-based measurement: Assessing special children. New York: Guilford.



Effective Formative Evaluation: The Underlying Logic…





5. How does the school check up on progress toward the goal(s)?



Team Activity: Data ‘Decision Points’

At your tables:

• Discuss what opportunities are available at the school, group, or individual student level to discuss data on student performance and make decisions about the effectiveness of your instructional or intervention programs.



School-Wide Case Example: Using Data to Evaluate

Appropriateness of Core Reading Program



“ ”“Risk for reading failure always involves the interaction of a particular set of child characteristics with specific characteristics of the instructional environment. Risk status is not entirely inherent in the child, but always involves a “mismatch” between child characteristics and the instruction that is provided.” (Foorman & Torgesen, 2001; p. 206).

Source: Foorman, B. R., & Torgesen, J. (2001). Critical elements of classroom and small-group instruction promote reading success in all children. Learning Disabilities Research & Practice, 16, 203-212.



“direct instruction in letter-sound correspondences practices in controlled vocabulary texts (direct code)” (Foorman & Torgesen, 2001; p. 204)

Source: Foorman, B. R., & Torgesen, J. (2001). Critical elements of classroom and small-group instruction promote reading success in all children. Learning Disabilities Research & Practice, 16, 203-212.

“Literature-based instruction emphasizes use of authentic literature for independent reading, read-alouds, and collaborative discussions. It stands in contrast to skills-based programs that are typically defined as traditional programs that use a commercially available basal reading program and follow a sequence of skills ordered in difficulty.” (Foorman & Torgesen, 2001; p. 204)

Direct / Indirect Instruction Continuum

“less direct instruction in sound-spelling patterns embedded in trade books (embedded code)” (Foorman & Torgesen, 2001; p. 204)

“implicit instruction in the alphabetic principle while reading trade books (implicit code)” (Foorman & Torgesen, 2001; p. 204)



RTI Core Literacy Instruction: ElementsUse Benchmarking/Universal Screening Data to Verify that the

Current Core Reading Program is Appropriate. The school uses benchmarking/universal screening data in literacy to verify that its current reading program can effectively meet the needs of its student population at each grade level.

• In grades K-2, if fewer than 80% of students are successful on phonemic awareness and alphabetics screenings, the core reading program at that grade level is patterned after direct instruction (Foorman & Torgesen, 2001).

• In grades K-2, if more than 80% of students are successful on phonemic awareness and alphabetics screenings, the school may choose to adopt a reading program that provides “less direct instruction in sound-spelling patterns embedded in trade books (embedded code)” (Foorman & Torgesen, 2001; p. 205).



Source: DIBELS Website. Retrieved on May 8, 2007, from https://dibels.uoregon.edu/

Comparison of Sunnyside & Baylor Schools: Winter Benchmarking: Gr 1



Nonsense Word Fluency: 34% of students fell below ‘Deficient’ level (<30 NWF)

Oral Reading Fluency: 35% of students fell below ‘Deficient’ level (<8 DORF)

Phoneme SegmentationFluency: 28% of students fell below ‘Deficient’ level(<10 PSF)

Sunnyside Central School District

District Student Population: 986

Eligible for Free/Reduced-Price Lunch: 43%

Number of Students in Grade1: 69

Winter Benchmarking: Gr 1 On all literacy screening measures, Sunnyside fell below the 80% success level:

PSF: 72% ‘emerging/ established’

NWF: 66% ‘emerging/ established’

DORF: 65% ‘some risk/ low risk’



Nonsense Word Fluency: 9% of students fell below ‘Deficient’ level (<30 NWF)

Oral Reading Fluency: 14% of students fell below ‘Deficient’ level (<8 DORF)

Phoneme SegmentationFluency: 6% of students fell below ‘Deficient’ level(<10 PSF)

Winter Benchmarking: Gr 1Baylor Unified Free School District

District Student Population: 1452

Eligible for Free/Reduced-Price Lunch: 6%

Number of Students in Grade1: 106

On all literacy screening measures, Baylor exceeded the 80% success level:

PSF: 94% ‘emerging/ established’

NWF: 91% ‘emerging/ established’

DORF: 86% ‘some risk/ low risk’



Winter Benchmarking: Gr 1:



“direct instruction in letter-sound correspondences practices in controlled vocabulary texts (direct code)” (Foorman & Torgesen, 2001; p. 204)

Source: What Works Clearinghouse. Retrieved April 15, 2009, from . http://ies.ed.gov/ncee/wwc/

Direct / Indirect Instruction Continuum

“less direct instruction in sound-spelling patterns embedded in trade books (embedded code)” (Foorman & Torgesen, 2001; p. 204)

“implicit instruction in the alphabetic principle while reading trade books (implicit code)” (Foorman & Torgesen, 2001; p. 204)

Sunnyside Elementary Core Reading Program

Baylor Elementary Core Reading Program



Individual Student Case Example: Collin: Letter

Identification



Case Example: Letter IdentificationThe Concern• In a mid-year (winter) school-wide screening for Letter Naming

Fluency, a first-grade student new to the school, Collin, was found have moderate delays when compared to peers. In his school, Collin fell at the 15th percentile compared with peers (local norms).

• Screening results, therefore, suggested that Collin has problems with Letter Identification. However, more information is needed to better understand this student academic delay.

67



Case Example: Letter IdentificationInstructional Assessment• Collin’s teacher, Ms. Tessia, sat with him and checked his letter

knowledge. She discovered that, at baselline, Collin knew 17 lower-case letters and 19 upper-case letters. (Ms. Tessia defined ‘knows a letter” as: “When shown the letter, the student can correctly give the name of the letter within 2 seconds.”)

• Based on her findings, Ms. Tessia decided that Collin was in the ‘acquisition’ phase in this letter identification skill. He needed direct-teaching activities to learn to identify all of the letters.

68



Case Example: Letter Identification

69



Case Example: Letter IdentificationIntervention• Ms. Tessia decided to use ‘incremental rehearsal’ (Burns, 2005) as an

intervention for Collin. This intervention benefits students who are still acquiring their math facts, sight words, or letters.

Students start by reviewing a series of ‘known’ cards. Then the instructor adds ‘unknown’ items to the card pile one at a time, so that the student has a high ratio of known to unknown items. This strategy promotes near-errorless learning.

• Collin received this intervention daily, for 10 minutes.

• NOTE: A paraprofessional, adult volunteer, or other non-instructional personnel can be trained to deliver this intervention.

70

Source: Burns, M. K. (2005). Using incremental rehearsal to increase fluency of single-digit multiplication facts with children identified as learning disabled in mathematics computation. Education and Treatment of Children, 28, 237-249.



East Carolina University Evidence-Based Intervention

Projecthttp://www.ecu.edu/cs-cas/psyc/

rileytillmant/EBI-Network-Homepage.cfm

Incremental Rehearsal Guidelines

71



Case Example: Letter IdentificationGoal-Setting and Data Collection• Ms. Tessia set the goals that, within 4 instructional weeks, Collin

would:– identify all upper-case and lower-case letters.– move above the 25th percentile in Letter Naming Fluency when

compared to grade-level peers.• The teacher collected two sources of data on the intervention:

– At the end of each tutoring session, the tutor logged any additional formerly unknown letters that were now ‘known’ (that the student could now accurately identify within 2 seconds).

– Each week, the teacher administered a one-minute timed Letter Naming Fluency probe and charted the number of correctly identified letters.

72



Case Example: Letter IdentificationOutcome• Ms. Tessia discovered that Collin attained the first goal (‘able to

identify all upper-case and lower-case letters’) within 2 weeks.• Collin attained the second goal (‘move above the 25th percentile

in Letter Naming Fluency when compared to grade-level peers’) within the expected four instructional weeks.

73



Individual Student Case Example: Angela: Reading

Fluency



DIBELS Case Example: Angela

• Angela is a 3rd grade student.

• Angela struggled in her classroom with reading fluency. Her teacher tried a series of classroom strategies to promote fluency for the student, including providing Angela with additional opportunities to listen to fluent text modeling from an adult and opportunities to read aloud with corrective feedback.



DIBELS Case Example: Angela

• In the mid-year schoolwide literacy screening in January, Angela read 77 words per minute on the DIBELS Oral Reading Fluency measure.

• According to DIBELS benchmark guidelines, Angela falls within the ‘strategic intervention’ range (between 67 and 92 WPM).



Source: Good, R. H., & Kaminski, R. A. (Eds.). (2002). Dynamic Indicators of Basic Early Literacy Skills (6th ed.). Eugene, OR: Institute for the Development of Educational Achievement. Available: http://dibels.uoregon.edu/.



DIBELS Case Example: Angela: Cont.• After the mid-year screening, the 3rd grade teachers, building

administrator, and reading teacher gathered for a ‘data meeting’.

• At that meeting, the group considered the screening results and discussed how to improve core literacy instruction to assist those students who fell within the ‘some risk’ and ‘at risk’ categories.

• The group next sorted students from the ‘some risk’ and ‘at risk’ categories into supplemental (Tier 2) groups, according to intervention need. Teacher knowledge of the student, classroom assessments, state test results, and other information was used to supplement the DIBELS data during this sorting process.



DIBELS Case Example: Angela: Cont.

• At the data meeting, it was decided that Angela and other students in the 3rd grade needed supplemental intervention support to increase their reading fluency, as well as to build their phonics (alphabetics) skills.

• The reading teacher agreed to start a Corrective Reading group that would meet for 4 days per week in 45 minute sessions. (The Corrective Reading program met the school’s guidelines as an ‘evidence-based’ program, based on findings from the What Works Clearinghouse website.)

• Angela and 5 other children were placed in this Corrective Reading group.



Corrective Reading: Description“Corrective Reading is designed to promote reading accuracy (decoding), fluency, and comprehension skills of students in third grade or higher who are reading below their grade level. The program has four levels that address students' decoding skills and six levels that address students' comprehension skills. All lessons in the program are sequenced and scripted. Corrective Reading can be implemented in small groups of four to five students or in a whole-class format. Corrective Reading is intended to be taught in 45-minute lessons four to five times a week. For the single study reviewed in this report, only the word-level skills components of the Corrective Reading program were implemented.

…Corrective Reading was found to have potentially positive effects on alphabetics and fluency and no discernible effects on comprehension.”

Source: What Works Clearinghouse. Retrieved on October 6, 2009 from http://ies.ed.gov/ncee/wwc/reports/beginning_reading/cr/c




• BASELINE: Before Angela began the Corrective Reading group, her reading teacher collected baseline data. The teacher used grade 3 progress-monitoring probes supplied by DIBELS. The student was administered Oral Reading Fluency probes across three separate days.

• At baseline, Angela was found to be reading 76 words per minute in grade 3 text. This became the starting point for setting a student goal for intervention.



Goal



DIBELS Case Example: Angela: Cont.• GOAL-SETTING. Because Angela would be monitored

using grade 3 ORF probes, it was decided to select an ambitious rate of progress. Using research norms, the reading teacher estimated that Angela should increase her reading rate by 1.5 additional words per week. Because the intervention would be in place for 6 instructional weeks, the teacher estimated that the student should read an additional 9 words per minute at the end of 6 weeks. Because the student’s baseline reading rate was 76 words per minute, her goal at the end of the 6 weeks is 85 words per minute. In other words, if the group intervention is successful, Angela should read at least 85 WPM at the end of the intervention period.



Table 2: Predictions for Reading Growth

by Grade




• IMPLEMENTATION OF INTERVENTION. When the Corrective Reading program began, Angela was assessed weekly (progress-monitoring) using grade 3 ORF probes from DIBELS.

• After six instructional weeks, the data team and reading teacher met to consider Angela’s progress.



Goal



DIBELS Case Example: Angela: Cont.• INTERVENTION CHECK-UP. At the end of 6 weeks, Angela

had made ‘promising’ progress but had not quite hit her intervention goal of 85 WPM.

• The school kept Angela in the Corrective Reading program, but decided to add an intervention component. A high school student was recruited and trained in Paired Reading. The tutor met with Angela 3 times per week for 25 minutes and used the Paired Reading strategy. Additionally, Angela’s parent was recruited to use Paired Reading at home for at least 2 times per week. The intervention goal was reset for 94 WPM.



Paired Reading

The student reads aloud in tandem with an accomplished reader. At a student signal, the helping reader stops reading, while the student continues on. When the student commits a reading error, the helping reader resumes reading in tandem.




• INTERVENTION CHECK-UP 2. At the end of the second 6-week intervention, the reading teacher examined the student’s monitoring data and discovered that she had met her intervention goal of 94 words per minute.



GoalGoal




• While the student had attained success, the school continued the intervention (Corrective Reading group and Paired Reading) for 3 more weeks to continue to strengthen Angela’s reading fluency. The school then discontinued the Tier 2 intervention.

• Although Angela’s teacher admitted that she was a bit anxious about the student’s ability to maintain success without the Tier 2 intervention, she was reassured that Angela would immediately be given RTI intervention support again if she were to be flagged as ‘at risk’ in a future grade-wide reading screening.





Problems identified for formative assessment should be:1. Important to school stakeholders.2. Measureable & observable. 3. Stated positively as ‘replacement behaviors’ or goal statements

rather than as general negative concerns (Bastche et al., 2008).

4. Based on a minimum of inference (T. Christ, 2008).

Source: Batsche, G. M., Castillo, J. M., Dixon, D. N., & Forde, S. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 177-193).Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176).



RIOT/ICEL Framework: Organizing Information to Better

Identify Student Behavioral & Academic Problems



RIOT/ICEL FrameworkSources of Information• Review (of records)• Interview• Observation• TestFocus of Assessment• Instruction• Curriculum• Environment• Learner

96



RIOT/ICEL Framework: Possible Uses• RTI Teams can use to better understand student

concerns, collect broader range of data• Consultants can use when working with teachers• Classroom teachers can use as one ‘self-check’

to ensure that they are making the best use of classroom data in their problem solving

• Principal can keep framework in mind when working with teachers

97



RIOT/ICEL Definition• The RIOT/ICEL matrix is an assessment guide to help

schools efficiently to decide what relevant information to collect on student academic performance and behavior—and also how to organize that information to identify probable reasons why the student is not experiencing academic or behavioral success.

• The RIOT/ICEL matrix is not itself a data collection instrument. Instead, it is an organizing framework, or heuristic, that increases schools’ confidence both in the quality of the data that they collect and the findings that emerge from the data.

98



RIOT: Sources of Information• Select Multiple Sources of Information: RIOT

(Review, Interview, Observation, Test). The top horizontal row of the RIOT/ICEL table includes four potential sources of student information: Review, Interview, Observation, and Test (RIOT). Schools should attempt to collect information from a range of sources to control for potential bias from any one source.

99



Select Multiple Sources of Information: RIOT (Review, Interview, Observation, Test)

• Review. This category consists of past or present records collected on the student. Obvious examples include report cards, office disciplinary referral data, state test results, and attendance records. Less obvious examples include student work samples, physical products of teacher interventions (e.g., a sticker chart used to reward positive student behaviors), and emails sent by a teacher to a parent detailing concerns about a student’s study and organizational skills.

100




• Interview. Interviews can be conducted face-to-face, via telephone, or even through email correspondence. Interviews can also be structured (that is, using a pre-determined series of questions) or follow an open-ended format, with questions guided by information supplied by the respondent. Interview targets can include those teachers, paraprofessionals, administrators, and support staff in the school setting who have worked with or had interactions with the student in the present or past. Prospective interview candidates can also consist of parents and other relatives of the student as well as the student himself or herself.

101




• Observation. Direct observation of the student’s academic skills, study and organizational strategies, degree of attentional focus, and general conduct can be a useful channel of information. Observations can be more structured (e.g., tallying the frequency of call-outs or calculating the percentage of on-task intervals during a class period) or less structured (e.g., observing a student and writing a running narrative of the observed events).

102




• Test. Testing can be thought of as a structured and standardized observation of the student that is intended to test certain hypotheses about why the student might be struggling and what school supports would logically benefit the student (Christ, 2008). An example of testing may be a student being administered a math computation CBM probe or an Early Math Fluency probe.

103



ICEL: Factors Impacting Student Learning• Investigate Multiple Factors Affecting Student Learning:

ICEL (Instruction, Curriculum, Environment, Learner). The leftmost vertical column of the RIO/ICEL table includes four key domains of learning to be assessed: Instruction, Curriculum, Environment, and Learner (ICEL). A common mistake that schools often make is to assume that student learning problems exist primarily in the learner and to underestimate the degree to which teacher instructional strategies, curriculum demands, and environmental influences impact the learner’s academic performance. The ICEL elements ensure that a full range of relevant explanations for student problems are examined.

104



Investigate Multiple Factors Affecting Student Learning: ICEL (Instruction, Curriculum, Environment, Learner)

• Instruction. The purpose of investigating the ‘instruction’ domain is to uncover any instructional practices that either help the student to learn more effectively or interfere with that student’s learning. More obvious instructional questions to investigate would be whether specific teaching strategies for activating prior knowledge better prepare the student to master new information or whether a student benefits optimally from the large-group lecture format that is often used in a classroom. A less obvious example of an instructional question would be whether a particular student learns better through teacher-delivered or self-directed, computer-administered instruction.

105




• Curriculum. ‘Curriculum’ represents the full set of academic skills that a student is expected to have mastered in a specific academic area at a given point in time. To adequately evaluate a student’s acquisition of academic skills, of course, the educator must (1) know the school’s curriculum (and related state academic performance standards), (2) be able to inventory the specific academic skills that the student currently possesses, and then (3) identify gaps between curriculum expectations and actual student skills. (This process of uncovering student academic skill gaps is sometimes referred to as ‘instructional’ or ‘analytic’ assessment.)

106




• Environment. The ‘environment’ includes any factors in the student’s school, community, or home surroundings that can directly enable their academic success or hinder that success. Obvious questions about environmental factors that impact learning include whether a student’s educational performance is better or worse in the presence of certain peers and whether having additional adult supervision during a study hall results in higher student work productivity. Less obvious questions about the learning environment include whether a student has a setting at home that is conducive to completing homework or whether chaotic hallway conditions are delaying that student’s transitioning between classes and therefore reducing available learning time.

107




• Learner. While the student is at the center of any questions of instruction, curriculum, and [learning] environment, the ‘learner’ domain includes those qualities of the student that represent their unique capacities and traits. More obvious examples of questions that relate to the learner include investigating whether a student has stable and high rates of inattention across different classrooms or evaluating the efficiency of a student’s study habits and test-taking skills. A less obvious example of a question that relates to the learner is whether a student harbors a low sense of self-efficacy in mathematics that is interfering with that learner’s willingness to put appropriate effort into math courses.

108



• The teacher collects several student math computation worksheet samples to document work completion and accuracy.

110

• Data Source: Review• Focus Areas: Curriculum



• The student’s parent tells the teacher that her son’s reading grades and attitude toward reading dropped suddenly in Gr 4.

111

• Data Source: Interview• Focus: Curriculum,

Learner



• An observer monitors the student’s attention on an independent writing assignment—and later analyzes the work’s quality and completeness.

112

• Data Sources: Observation, Review

• Focus Areas: Curriculum, Environment, Learner



• A student is given a timed math worksheet to complete. She is then given another timed worksheet & offered a reward if she improves.

113

• Data Source: Review, Test

• Focus Areas: Curriculum, Learner



• Comments from several past report cards describe the student as preferring to socialize rather than work during small-group activities.

114

• Data Source: Review• Focus Areas: Instruction,

Environment



• The teacher tallies the number of redirects for an off-task student during discussion. She designs a high-interest lesson, still tracks off-task behavior.

115

• Data Source: Observation, Test

• Focus Areas: Instruction



Activity: Use the RIOT/ICEL Framework

• Discussion: How can this framework be used in your school to improve the quality of the data that you collect on the student?

116



Defining Student Problem Behaviors: A Key to Identifying Effective Interventions Jim Wrightwww.interventioncentral.org



Defining Problem Student Behaviors…1. Define the problem behavior in clear, observable, measurable

terms (Batsche et al., 2008; Upah, 2008). Write a clear description of the problem behavior. Avoid vague problem identification statements such as “The student is disruptive.”

A well-written problem definition should include three parts:– Conditions. The condition(s) under which the problem is likely to

occur– Problem Description. A specific description of the problem behavior– Contextual information. Information about the frequency, intensity,

duration, or other dimension(s) of the behavior that provide a context for estimating the degree to which the behavior presents a problem in the setting(s) in which it occurs.



Defining Problem Student Behaviors…2. Develop examples and non-examples of the

problem behavior (Upah, 2008). Writing both examples and non-examples of the problem behavior helps to resolve uncertainty about when the student’s conduct should be classified as a problem behavior. Examples should include the most frequent or typical instances of the student problem behavior. Non-examples should include any behaviors that are acceptable conduct but might possibly be confused with the problem behavior.



Defining Problem Student Behaviors…3. Write a behavior hypothesis statement (Batsche et al.,

2008; Upah, 2008). The next step in problem-solving is to develop a hypothesis about why the student is engaging in an undesirable behavior or not engaging in a desired behavior. Teachers can gain information to develop a hypothesis through direct observation, student interview, review of student work products, and other sources. The behavior hypothesis statement is important because (a) it can be tested, and (b) it provides guidance on the type(s) of interventions that might benefit the student.



Defining Problem Student Behaviors…4. Select a replacement behavior (Batsche et al.,

2008). Behavioral interventions should be focused on increasing student skills and capacities, not simply on suppressing problem behaviors. By selecting a positive behavioral goal that is an appropriate replacement for the student’s original problem behavior, the teacher reframes the student concern in a manner that allows for more effective intervention planning.



Defining Problem Student Behaviors…5. Write a prediction statement (Batsche et al., 2008;

Upah, 2008). The prediction statement proposes a strategy (intervention) that is predicted to improve the problem behavior. The importance of the prediction statement is that it spells out specifically the expected outcome if the strategy is successful. The formula for writing a prediction statement is to state that if the proposed strategy (‘Specific Action’) is adopted, then the rate of problem behavior is expected to decrease or increase in the desired direction.



Formative Assessment: Culminating Team Activity

As a team:

• Discuss the formative assessment concepts, tools, and resources reviewed in this workshop.

• What are the key ‘next steps’ that your team will take to follow up on this workshop?

• What additional questions do you have on the topic of formative evaluation?

effective formative evaluation: the underlying logic…

Documents