Using Local Norms to Using Local Norms to Support RtI PracticesSupport RtI Practices

NASP NASP

20102010

AbstractThe purpose of this presentation is to show how local norms can support RtI practices in a large, urban school district.

Topics discussed will include: Rationale and creation of local norms, The potential benefits and limitations of their use How the assessment of relative risk can improve

the efficiency of resource allocation in the context of an RtI framework

RtIRtI RtI is the practice of (1) providing high quality RtI is the practice of (1) providing high quality

instruction/intervention instruction/intervention MATCHED TO MATCHED TO STUDENT NEEDSTUDENT NEED and (2) using learning rate and (2) using learning rate over time and level of performance to (3) make over time and level of performance to (3) make important educational decisions. These important educational decisions. These components of RtI are essential to the components of RtI are essential to the development of a successful RtI development of a successful RtI implementation strategy.implementation strategy.¹¹

Matching students to instruction cannot be Matching students to instruction cannot be done when you define this based on students done when you define this based on students other than the ones in front of you.other than the ones in front of you.

1 Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., 1 Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., & Tilly, D. (2005). Schrag, J., & Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Implementation. Alexandria, VA: National Association of State Directors of Special Education.Education.

RtI TriangleRtI Triangle While RtI models typically use triangles to While RtI models typically use triangles to

illustrate the “ideal” distribution of students to illustrate the “ideal” distribution of students to resources, reality often looks very different.resources, reality often looks very different.

It is common in many at-risk schools to see an It is common in many at-risk schools to see an inversion of where the resources are allocated.inversion of where the resources are allocated.

It is also common to see a hole or a gap at tier 2.It is also common to see a hole or a gap at tier 2. Often the initial distribution of students into the Often the initial distribution of students into the

tiers is based on their proficiency status.tiers is based on their proficiency status.Tier 3_____________________5%

Tier 2__________________15%

Tier 1_____________80%

50%___________Tier 3

10% ___________Tier 2

40% __________Tier 1

A Look at the Status QuoA Look at the Status Quo Impact of Lawsuit and Building CoordinationImpact of Lawsuit and Building Coordination Results have been an increase in the number of total initial referrals Results have been an increase in the number of total initial referrals Psychologist responsibilities are centered on building coordination Psychologist responsibilities are centered on building coordination

and evaluations, leaving less time for pre-referral activitiesand evaluations, leaving less time for pre-referral activities

4000

4500

5000

5500

6000

6500

2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009

School Year

To

tal

Init

ial

Ref

erra

ls

Impact on School ResourcesImpact on School Resources Most students referred (whether or not the Most students referred (whether or not the

referral is appropriate) for an evaluation referral is appropriate) for an evaluation ultimately require formal testingultimately require formal testing

Costs of Special Ed ReferralsCosts of Special Ed Referrals A full evaluation can be valued as high as A full evaluation can be valued as high as

$3000 (Mirkin & Potter, 1983)$3000 (Mirkin & Potter, 1983) Also consider the costs of special educational Also consider the costs of special educational

programming programming The shift to RtI will largely impact the role The shift to RtI will largely impact the role

of the school psychologistof the school psychologist

VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007)VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007)

Where Does This Put Us and Where Does This Put Us and Where Do We Want to GoWhere Do We Want to Go

A large urban district, with limited financial and A large urban district, with limited financial and instructional resourcesinstructional resources

Necessity to identify at-risk students and provide Necessity to identify at-risk students and provide services to improve overall district performanceservices to improve overall district performance

An increasingly inefficient evaluation process for An increasingly inefficient evaluation process for special educationspecial education

Limited and inconsistent success in Limited and inconsistent success in implementing RtI using national normsimplementing RtI using national norms

Using Local Norms may address these issues…Using Local Norms may address these issues…

What is the Goal?What is the Goal? The students’ skills have not changed just The students’ skills have not changed just

because we look at their data differentlybecause we look at their data differently But with limited resources we can channel But with limited resources we can channel

them in ways that can increase the systems them in ways that can increase the systems capacity to be effective.capacity to be effective.

Proficiency is still the destination. Using local Proficiency is still the destination. Using local data to support RtI practices lets us get their data to support RtI practices lets us get their with the car we own, not the jet we wish we with the car we own, not the jet we wish we had.had.

Reading FirstReading First

DIBELSDIBELS Training and technical supportTraining and technical support The OSPS Data SystemThe OSPS Data System The use of available national cut scores to The use of available national cut scores to

allocate local district resourcesallocate local district resourcesdibels.uoregon.edu

© University of Oregon Center on Teaching and Learning. All rights reserved.

Grade Level Spring DataGrade Level Spring Data

When using National norm data When using National norm data

Tier 3: 12 %

Tier 2: 55 %

Tier 1: 33 %

Creating Local NormsCreating Local Norms

=100*percentrank(A:A,A2,2)=100*percentrank(A:A,A2,2)

Better than sliced bread!

Grade Level Spring DataGrade Level Spring Data

When using Local norm data When using Local norm data

Tier 3: 9 %

Tier 2: 24 %

Tier 1: 67 %

Determine Your ApproachDetermine Your Approach

Before attacking the data, identify…Before attacking the data, identify… How many children can be effectively How many children can be effectively

serviced (school/district)? serviced (school/district)? Does our core service need to change?Does our core service need to change? What will be the cut-offs for which tier?What will be the cut-offs for which tier?

RtI tiers defined by percentilesRtI tiers defined by percentiles 80-15-580-15-5 20-40-6020-40-60

Pros of Local NormsPros of Local Norms

Decreases bias in decision makingDecreases bias in decision making Increases ability to match instruction Increases ability to match instruction

to student needto student need Maintains proficiency expectationsMaintains proficiency expectations Illuminates identifiable patterns of Illuminates identifiable patterns of

performance and changes over timeperformance and changes over time

Stewart, L., & Kaminski, R. (2002) Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)Stewart, L. & Silberglitt, B. (2008)

Cons of Local NormsCons of Local Norms Can be misinterpretedCan be misinterpreted Does not define acceptable Does not define acceptable

performanceperformance Norms are not diagnostic in isolationNorms are not diagnostic in isolation Must adhere to appropriate testing Must adhere to appropriate testing

standards (be careful to ensure integrity standards (be careful to ensure integrity of administration)of administration)

Stewart, L., & Kaminski, R. (2002) Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)Stewart, L. & Silberglitt, B. (2008)

Not Lowering the BarNot Lowering the Bar Proficiency status and response are two different Proficiency status and response are two different

questions.questions. Proficiency tells us if a student has reached a Proficiency tells us if a student has reached a

destination. For WI it is officially assessed in the fall destination. For WI it is officially assessed in the fall with results unavailable till 3/4with results unavailable till 3/4 thth of the school year is of the school year is over.over.

Response to local normative data tells us if a Response to local normative data tells us if a student is on the right track, and continued student is on the right track, and continued monitoring will estimate when they’ll reach their monitoring will estimate when they’ll reach their destination.destination.

A student’s response to their education is evident A student’s response to their education is evident by using local data to look at growth.by using local data to look at growth.

Further Use of Local Norms

Can be used to predict outcomes such as: Ending benchmark risk status Proficiency on state assessments Graduation rate

Can use receiver operator characteristic curves to predict binary outcomes

Creating Cut Scores for Risk on Creating Cut Scores for Risk on Binary OutcomesBinary Outcomes

Receiver Operating Receiver Operating Characteristic Curve Characteristic Curve (ROC Curve) (ROC Curve) A graphical plot of A graphical plot of

sensitivity vs. 1-sensitivity vs. 1-specificity as the specificity as the threshold point is threshold point is varied.varied.

What Does a ROC Curve Do?What Does a ROC Curve Do?

Plots sensitivity (y-axis) and 1-specificity Plots sensitivity (y-axis) and 1-specificity (x-axis) to determine accuracy of (x-axis) to determine accuracy of predictionprediction

In choosing a cut score, factors of In choosing a cut score, factors of sensitivity sensitivity andand specificity specificity need to be need to be understood. understood.

SensitivitySensitivity

Sensitivity is the proportion of true positives that Sensitivity is the proportion of true positives that were correctly classified as positive given the were correctly classified as positive given the threshold or cut score used to establish it.threshold or cut score used to establish it.

In terms of sickness it is the proportion of people In terms of sickness it is the proportion of people that actually are sick that were correctly that actually are sick that were correctly classified as being sick (strep)classified as being sick (strep)

In proficiency terms, it is the proportion of In proficiency terms, it is the proportion of students that actually are not proficient that were students that actually are not proficient that were correctly classified as at-risk.correctly classified as at-risk.

SpecificitySpecificity Specificity is the proportion of true negatives that Specificity is the proportion of true negatives that

were correctly classified as negative.were correctly classified as negative. In terms of sickness it is the proportion of people In terms of sickness it is the proportion of people

that are healthy that were correctly classified as that are healthy that were correctly classified as healthy.healthy.

In terms of proficiency it is the proportion of In terms of proficiency it is the proportion of students that are proficient that were correctly students that are proficient that were correctly classified as proficient.classified as proficient.

1-specificity is the proportion of students that are 1-specificity is the proportion of students that are incorrectly classified as being at-risk when in incorrectly classified as being at-risk when in fact they are proficient.fact they are proficient.

The ROC CurveThe ROC Curve The dashed red The dashed red

line represents line represents test B in terms of test B in terms of diagnostic utility. diagnostic utility. As the threshold As the threshold varies (cut varies (cut scores) the test scores) the test does no better does no better than random at than random at correct correct classificationclassification

Test A has more Test A has more utility and utility and

Area Under the CurveArea Under the Curve

Test Result Variable(s):Winter ORF Score

Area Std. ErrorAsymptotic

Sig.

Asymptotic 95% Confidence Interval

Lower Bound

Upper Bound

.950 .014 .000 .922 .978

Coordinates of the CurveCoordinates of the CurveTest Result Variable(s):Winter ORF Score (Predicting Spring Percentrank)

Positive if Less Than or Equal To Sensitivity 1 - Specificity

5.50 .381 .011

6.50 .476 .022

7.50 .571 .027

8.50 .714 .049

9.50 .762 .076

10.50 .810 .097

11.50 .881 .114

12.50 .905 .130

13.50 .905 .146

14.50 .929 .168

Using ROC Curves to Produce Local Cuts Based on

State Test Scores

Winter/Spring AUCWinter/Spring AUC

Lower Bound

Upper Bound

Winter ORF

.829 .013 .000 .802 .855

Spring ORF .857 .012 .000 .833 .881

Area Under the Curve

Test Result Variable(s) Area Std. Errora

Asymptotic

Sig.b

Asymptotic 95%

Winter 1Winter 1stst Grade ORF Grade ORF

Positive if Less Than or Equal

Toa Sensitivity1 -

Specificity

15.50 .643 .16916.50 .669 .18117.50 .693 .19618.50 .716 .21919.50 .736 .24420.50 .755 .26521.50 .780 .28922.50 .791 .30523.50 .801 .31724.50 .814 .339

Coordinates of the Curve

Spring 1Spring 1stst Grade ORF Grade ORF

Positive if Less Than Sensitivity 1 - Specificity

38.50 .779 .21439.50 .788 .21840.50 .792 .22641.50 .799 .23342.50 .807 .24443.50 .814 .25744.50 .829 .27145.17 .838 .28545.67 .838 .28746.50 .844 .316

Coordinates of the Curve

Test Result Variable(s):Spring ORF

DIBELS Oral Reading Fluency Risk Indicators

Local and National Norms Fall Winter Spring Grade Level/

Risk Category Local National Local National Local National

1 At Risk N/A N/A ≤ 10 ≤ 11 ≤ 24 ≤ 19 1 Some Risk N/A N/A 11 – 16 12 – 18 25 – 37 20 – 39 1 Low Risk N/A N/A > 16 > 18 >37 >39 2 At Risk ≤ 21 ≤ 28 ≤ 30 ≤ 49 ≤ 50 ≤ 69 2 Some Risk 22 – 31 29 – 44 31 – 55 50 – 70 51 – 73 70 – 89 2 Low Risk > 31 > 44 > 55 > 70 > 73 > 89 3 At Risk ≤ 44 ≤ 54 ≤ 53 ≤ 63 ≤ 67 ≤ 79 3 Some Risk 45 - 58 55 – 75 54 – 70 64 – 87 68 – 89 80 – 109 3 Low Risk > 58 > 75 > 70 > 87 > 89 > 109 4 At Risk ≤ 55 ≤ 68 ≤ 67 ≤ 82 ≤ 74 ≤ 95 4 Some Risk 56 – 67 69 – 88 68 – 82 83 – 102 75 – 97 96 – 117 4 Low Risk > 68 > 88 > 82 > 102 > 97 > 117 5 At Risk ≤ 76 ≤ 87 ≤ 76 ≤ 93 ≤ 88 ≤ 102 5 Some Risk 77 – 93 88 – 100 77 – 97 94 – 108 89 – 110 103 – 123 5 Low Risk > 93 > 100 > 97 > 108 > 110 > 123

Making Data Actionable

Use the data to determine: How instruction can be matched to student

need Whether or not the core instruction is effective Goals Progress monitoring frequency When the goal will be achieved by their

current rate of growth Whether or not interventions are effective

Potential benefits to using local Potential benefits to using local Norms within RtINorms within RtI

• Local data takes into consideration the Local data takes into consideration the instructional environment of the studentinstructional environment of the student

• Appropriate use of local data could help improve Appropriate use of local data could help improve overall instruction by making assessment more overall instruction by making assessment more meaningful meaningful

• Improved “hit rate”Improved “hit rate”

• Creating your own triangle, instead of using Creating your own triangle, instead of using someone else’s rectanglesomeone else’s rectangle

Potential Outcomes and UsesPotential Outcomes and UsesExpanded role of school psychologistsExpanded role of school psychologists

• Shift in the types of assessments towards Shift in the types of assessments towards screening, progress monitoring and interventionsscreening, progress monitoring and interventions

• This shift will result in more direct involvement in This shift will result in more direct involvement in student learningstudent learning

• Helping schools make the connections between Helping schools make the connections between assessment and instructional decision makingassessment and instructional decision making

• New roles may develop in the areas of data New roles may develop in the areas of data analysis and could include such things as whole analysis and could include such things as whole school data collection, management, and school data collection, management, and interpretationinterpretation

• Training other school personnelTraining other school personnel in the in the understanding and interpretation of local dataunderstanding and interpretation of local data

For more information:For more information:

www2.milwaukee.k12.wi.us/rti www2.milwaukee.k12.wi.us/rti Margaret Peters, Margaret Peters, kreulm@milwaukee.k12.wi.us Kelly Witz, Kelly Witz, witzka@milwaukee.k12.wi.us Robert Latterman, Robert Latterman, latterrl@milwaukee.k12.wi.us Marc Sanders, Marc Sanders, sandermc@milwaukee.k12.wi.us Steve Smith, Steve Smith, smithsl2@milwaukee.k12.wi.us

Handouts are available for download on the Handouts are available for download on the NASP websiteNASP website

ReferencesBatsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., &

Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Education.

Good, R. H., III, & Kaminski, R. A. (1996). Assessment for instructional decisions: Toward a proactive/prevention model of decision-making for early literacy skills. School Psychology Quarterly, 11(4), 326- 336.

Good, R. H., Wallin, J., Simmons, D. C., Kame’enui, E. J., & Kaminski, R. A. (2002). System-wide Percentile Ranks for DIBELS Benchmark Assessment (Technical Report 9). Eugene, OR: University of Oregon.

Hintze, J., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34(3), 372-386.

Shinn, M. (1998). Advanced applications of Curriculum-Based Measurement. New York, NY US: Guilford Press.

Stewart, L., & Kaminski, R. (2002). Best practices in developing local norms for academic problem solving. Best Practices in School Psychology IV (Vol. 1, Vol. 2) (pp. 737-752). Washington, DC US: National Association of School Psychologists.

Stewart, L., & Silberglitt, B. (2008). Best practices in developing academic local norms. Best Practices in School Psychology V (Vol. 2) (pp. 225-242). Washington, DC US: National Association of School Psychologists.

VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007) A multi-year evaluation of the effects of a Response to Intervention (RtI) model on identification of children for special education. Journal of School Psychology, 45, 225-256..