step 8: data-based decision making · 1.13 data-based decision making: tier i team reviews and uses...

“Without good data and assessment reports, we are simply throwing the spaghetti at the wall,

seeing what sticks, then trying something else.”

-Beth Baker and Char Ryan

Using data for decision making is key to using the collaborative learning cycle, which results in

effective and efficient action planning and implementation. Data are observations, facts, or

numbers when collected and organized, become information and, when used productively

in context, become knowledge. Data are merely numbers or words and, alone, have no

intrinsic meaning. It has been said data do not laugh or cry. Individuals or groups give meaning

to data by organizing, analyzing, interpreting, and using them.

Tiered Fidelity Inventory

FEATURE

Data Source Main Idea

1.12 Discipline Data: Tier I team has

instantaneous access to graphed reports

summarizing discipline data organized by

the frequency of problem behavior events

by behavior, location, time of day, and by

individual student.

Is there a centralized data system to collect

and organize behavior incident data?

Does the Tier I team have instantaneous

access to graphed reports summarizing

discipline data?

Are those data organized to review all of

the following: frequency of problem

behavior events by behavior, location, time

of day and student?

Behavioral Data

Problem Behavior

Location

Student/Grade Level

Time

Motivation

Drill Down

Teams need the

right information

in the right form

at the right time

to make effective

decisions.

1.13 Data-based Decision Making: Tier I

team reviews and uses discipline data and

academic outcome data (e.g.,

Curriculum-Based Measures, state tests) at

least monthly for decision making.

Does the team have access to discipline

data for the entire student body (school-

wide)?

Does the team have access to academic

data for the entire student body?

Are those data clearly and logically linked

to the annual action plan for Tier I?

Are those data reviewed at least monthly?

Team Initiated Problem

Solving Model

Team Meeting Minutes

Precise Problem

Statement

Goal and Timeline

Solution Plan

Fidelity & Outcome

Data

Next Steps

Multiple Data Sets

Attendance Data

Behavioral Data

Coursework Data

Teams need the

right information

in the right form

at the right time

to make effective

decisions

Step 8: Data-based Decision Making

8-1

Team Initiated Problem Solving Model Horner, R. H., Newton, J. S., Todd, A. W., Algozzine, B., Algozzine, K., Cusumano, D. L., & Preston, A. I. (2015).

Operational Definition

Team-Initiated Problem Solving (TIPS) is a problem-solving framework used during meetings

(e.g., PBIS, RTI, MTSS) focused on data-based decision making to improve student outcomes.

TIPS is applicable to varied data sources (e.g., DIBELS, AIMSweb, SWIS), content areas (e.g.,

academic, behavior), and levels of application (e.g., school, district, state).

Rationale

It is common for schools to have “problem-solving teams” focused on addressing student

academic and behavior challenges. Some teams use general problem-solving models (e.g.,

problem identification, problem analysis, plan development, and plan evaluation) to lead

them to problem resolution. Unfortunately, research documents that, although school teams

indicate they are adhering to problem-solving guidelines, they are often missing critical

components, thus decreasing the chances of improving student outcomes (Flugum &

Reschly, 1994; McDougal et al., 2009; Telzrow et al., 2009). Barriers to conducting efficient

problem solving meetings have been identified to include: limited time scheduled for

meetings, gaps in foundations (e.g., location, team members, procedures, efficiency of

meeting), an unfocused or unidentified purpose for meeting, and inadequate training and

support to implement effective and efficient problem solving (Nellis, 2012). Team-Initiated

Problem Solving (TIPS) is a framework that addresses these barriers by breaking down

problem solving into six critical steps to guide teams through a data-based decision making

process that leads to desired outcomes. TIPS also infuses critical elements of effective and

efficient meetings (e.g., consistent procedures, team member roles, meeting minutes prompt

the problem solving process). TIPS is a generic problem solving model that provides structure

to any type of meeting.

The TIPS model includes focus on meeting foundations guided by

a structured Meeting Minutes form and a six-step problem solving process.

8-3

The Problem Solving Cycle

What to Do Questions to Ask

Step 1:

Identify Problem with

Precision

What is the problem? Who? What? Where?

When? Why?

Step 2:

Identify Goal for Change

How do we want the problem to change?

What evidence do we need to show that we

have achieved our goal?

Step 3:

Identify Solution and

Create Implementation

Plan with Contextual Fit

How are we going to solve the problem?

How are we going to bring about desired

change?

Is solution appropriate for problem?

Is solution likely to produce desired change?

Step 4:

Implement Solution with

High Integrity

How will we know solution was implemented

with fidelity?

Did we implement solution with fidelity?

Step 5:

Monitor Impact of

Solution and Compare

Against Goal

Are we solving the problem?

Is desired goal being achieved?

Step 6:

Make Summative

Evaluation Decision

Has the problem been solved?

Has desired goal been achieved? What

should we do next?

8-4

TIPS Fidelity of Implementation Checklist

Items 1- 9

measures implementation status of

meeting foundations

Scoring Criteria

1. Primary and backup

individuals are assigned to

defined roles and

responsibilities of Facilitator,

Minute Taker, and Data

Analyst.

0= No primary and backup individuals are assigned to the defined roles

and responsibilities of Facilitator, Minute Taker, and Data Analyst.

1= Some primary and backup individuals are assigned to the defined

roles and responsibilities of Facilitator, Minute Taker, and Data Analyst.

2= Primary and backup individuals are assigned to the defined roles and

responsibilities of Facilitator, Minute Taker, and Data Analyst.

2. Meeting participants have

the authority to develop

and implement problem-

solving solutions.

0= Meeting participants do not have the authority to develop and

implement problem solving solutions.

1= Meeting participants have the authority to develop but not

implement problem solving solutions.

2= Meeting participants have the authority to develop and implement

problem solving solutions.

3. Meeting started on time. 0= Meeting started more than10 minutes late.

1= Meeting started less than 10 minutes late.

2= Meeting started on time.

4. Meeting ended on time, or

members agreed to

extend meeting time.

0= Meeting ended more than 10 minutes over scheduled time.

1= Meeting ended 10 minutes over scheduled time.

2= Meeting ended on time or members agreed to extend meeting time.

5. Team members attend

meetings promptly and

regularly.

0= Less than 75% of team members attend meetings promptly and

regularly.

1= Although team members (with exception of administrator) attend

meetings regularly, they are not always prompt and/or they leave

early.

2= More than 75% of team members (with exception of administrator)

attend meetings regularly, promptly and remain present until the

meeting has concluded.

6. Public agenda format was

used to define topics and

guide meeting discussion

and was available for all

participants to refer to

during the meeting.

0 = Public agenda format was not used to define topics and guide

meeting discussion.

1= Public agenda format was not used to define topics and guide

meeting discussion but agenda was available for participants to

refer to during the meeting.

2= Public agenda was used to define topics and guide meeting

discussion and was available for all participants to refer to during the

meeting.

7. Previous meeting minutes

were present and

reviewed at start of the

meeting.

0= Previous meeting minutes were not present or reviewed at start of the

meeting.

1= Previous meeting minutes were present but not reviewed at start

of the meeting.

2= Previous meeting minutes were present and reviewed at start of the

meeting.

8. Next meeting was

scheduled by the conclusion

of the meeting.

0= Next meeting was not scheduled.

1= Next meeting was referred to but not scheduled.

2= Next meeting was scheduled.

9. Meeting Minutes are

distributed to all team

members within 24 hours

of the conclusion of the

meeting.

0= Meeting Minutes are not distributed to all team members.

1= Meeting minutes are distributed to all team members but not within 24

hours of the meeting.

2= Meeting minutes are distributed to all team members within 24

hours of the meeting.

8-5

Items 10 - 18 measures the thoroughness of the

team’s problem-solving processes

Scoring Criteria

10. Team uses TIPS Meeting

Minutes form or

equivalent*.

0= Team does not use TIPS Meeting Minutes form or equivalent*.

1= Team uses part of TIPS Meeting Minutes form or equivalent*.

2= Team uses TIPS Meeting Minutes form or equivalent*.

11. Status of all previous

solutions was reviewed.

0= Previous solutions were not reviewed.

1= Status of some previous solutions was reviewed.

2= Status of all previous solutions was reviewed.

12. Quantitative data were

available and reviewed.

0= Quantitative data were not available or reviewed.

1= Quantitative data were available but not reviewed.

2= Quantitative data were reviewed.

13. At least one problem is

defined with precision (what,

where, when, by whom,

why).

0= No problem is defined.

1= At least one problem is defined but lack one or more precision

elements.

2 = At least one problem is defined with all precision elements.

14. All documented active

problems have documented

solutions.

0= Documented active problem(s) do not have documented solutions or

no active problems are documented.

1 = Some documented active problems (s) have documented solutions.

2 = All documented active problems have documented solutions.

15. Full action plan (who,

what, when) is documented

for at least one documented

solution.

0= No action plan is documented for at least one documented

solution or no solution(s) are documented.

1= Partial action plan is documented for at least one documented

solution.

2= Full action plan is documented for at least one documented

solution.

16. Problems that have

solutions defined have a goal

defined.

0= Problems that have solutions defined do not have a goal defined

or no solutions are documented.

1= Some problems that have solutions defined have a goal defined.

2= Problems that have solutions defined have a goal defined.

17. A fidelity of

implementation measure is

documented for each

solution, along with a

schedule for gathering those

data.

0 = Fidelity measure and schedule are not defined and documented

for solutions or no active problem(s)/solution(s)/goal(s) are

documented.

1= Fidelity measure and schedule are defined and documented for

some solutions.

2= Fidelity measure and schedule are defined and documented for all

solutions.

18. A student

social/academic outcome

measure is documented for

each problem, along with a

schedule for gathering those

data.

0 = Measure and regular schedule for student behavior/performance

are not documented.

1= Measure and regular schedule for student behavior /performance

are documented for some solutions.

2= Measure and regular schedule for student behavior/performance

are documented for all solutions.

8-6

Critical Features of Team Meeting Minutes

Meeting Minutes serve as documentation and guidance for decisions made during problem-solving and/or

coordination/planning team meeting includes sections and prompts to guide and prompt recording of

relevant, accurate, and succinct information across the following areas:

Meeting Demographics: Information related to meeting logistics, roles, agenda, and announcements

Critical Information to Document

Roles, Agenda Items, and Announcements (if appropriate)

Current Meeting date, time, and location including Role/Assignments

Next Meeting date, time, location, Role/Assignments (if rotating), potential agenda items

Regular team member list and/or documentation of meeting participants

Overall Systems Status Update: Information and data related to team purpose or goals regarding the fidelity

with which curriculum and practices are being implemented


Implementation Fidelity (e.g., measure used, schedule for data collection and review)

Big Picture Outcomes (e.g., measure used, schedule for data collection and review)

Problem-Solving, Action Planning, and Evaluation: Data-based decision making regarding targeted problems

reported


Problem to be addressed (e.g., group/individual social, academic, mental health problems, goal met/fade

or graduate supports)

Problem Statement that includes who, what, where, when, why, and how often

Goal or target (what will change, by how much/to what level, by when)

Solution actions and plans (what will happen, who will do it, by when)

Plan for gathering fidelity and outcome data (what, who, by when)

Evaluation of impact of solutions with current level, comparison to goal and next steps

Organization and Housekeeping Items: Tasks that are completed as part of the ongoing cycle of coordination,

development, implementation and evaluation of systems and procedures related to readiness, sustainability

and day to day operations


General announcements, systems-level action tasks, other logistical decisions

Communication actions to inform appropriate stakeholders of progress and/or decisions (e.g.,

administrator, other teams, family/community, all or specific staff members)

Meeting Assessment/Evaluation: Process to self-evaluate whether the meeting was efficient and effective in its

assigned mission or task(s).


Was meeting a good use of time?

Were tasks implemented with fidelity?

Are efforts benefitting students?

Helpful and Optional Enhancements

Provide specific prompts (what, where, who, when, why, current levels, etc.)

Provide specific areas (section) for each type of item and critical information

Add roles for time keeper, snacks for meeting, as needed

Create a general sequence for agenda items: a) Review agenda and previous Meeting Minutes, b)

Overall Systems Update (when applicable), c) Problem Solving, d) Housekeeping Tasks, and e) Evaluation

of Meeting

8-7

Meeting Minutes Guide

Date Time (begin and end) Location Facilitator Minute Taker Data Analyst

Today’s Meeting

Next Meeting

Team Members & Attendance (Place “X” to left of name if present)

Today’s Agenda Items: Agenda Items for Next Meeting

1. 4. 1.

2. 5. 2.

3. 6. 3.

Systems Overview

Overall Status Tier/Content Area Measure Used Data Collection Schedule Current Level/Rate

Problem Solving Process

Date of Initial Meeting: Date(s) of Review Meetings

Brief Problem Description (e.g., student name, group identifier, brief item description):

Precise Problem

Statement

What? When? Where? Who? Why?

How Often?

Goal and

Timeline

What? By

When?

Solution

Actions

By Who? By When?

Identify Fidelity

and Outcome Data

What? When? Who?

I

M

P

L

E

M

E

N

T

S

O

L

U

T

I

O

N

S

Did it work?

(Review current levels and compare to goal)

What fidelity data will

we collect?

What? When? Who?

Fidelity Data:

Level of Implementation

Not started

Partial implementation

Implemented with fidelity

Stopped

Notes:

Outcome Data (Current Levels):

Comparison to Goal

Worse

No Change

Improved but not to goal

Goal met

Notes:

What outcome data

will we collect?

What? When? Who?

Current Levels: Next Steps

Continue current plan

Modify plan

Discontinue plan

Other

Notes:

Notes:

Date of Initial Meeting: Date(s) of Review Meetings

Brief Problem Description (e.g., student name, group identifier, brief item description)

Precise Problem

Statement

What? When? Where? Who? Why?

How Often?

Goal and

Timeline

What? By

When?

Solution

Actions

By Who? By When?

Identify Fidelity

and Outcome Data

What? When? Who?

I

M

P

L

E

M

E

N

T

S

O

L

U

T

I

O

N

S

Did it work?

(Review current levels and compare to goal)

What fidelity data will

we collect?

What? When? Who?

Fidelity Data:

Level of Implementation

Not started

Partial implementation

Implemented with fidelity

Stopped

Notes:

Outcome Data (Current Levels):

Comparison to Goal

Worse

No Change

Improved but not to goal

Goal met

Notes:

What outcome data

will we collect?

What? When? Who?

Current Levels: Next Steps

Continue current plan

Modify plan

Discontinue plan

Other

Notes:

Notes:

Organizational/Housekeeping Task List

Item Discussion Decisions and Tasks Who? By When?

Evaluation of Team Meeting (Mark your ratings with an “X”) Our Rating

Yes So-So No

1. Was today’s meeting a good use of our time?

2. In general, did we do a good job of tracking whether we’re completing the tasks we agreed on at previous meetings?

3. In general, have we done a good job of actually completing the tasks we agreed on at previous meetings?

4. In general, are the completed tasks having the desired effects on student behavior?

8-1

0

SWIS Big 4 for October 1, 2011 through December 31, 2011

Step 1:

Identify Problem with Precision/Elementary

Data Scenario: Behavior

Precise Problem Statement

(Summary Statement)

Precision Components for

Behavior Problem Statements What problem behaviors are most common? Where are problem behaviors most likely? When are problem behaviors most likely? Who is engaged in problem behavior? Why are problem behaviors sustaining? SWIS Big 4 for October 1, 2011 through December 31, 2011

SWIS Big 4 for October 1, 2011 through December 31, 2011

8-11

Step 1:

Identify Problem with Precision/Secondary Data Scenario: Behavior

Precise Problem Statement

(Summary Statement)

Precision Components for

Behavior Problem Statements What problem behaviors are most common? Where are problem behaviors most likely? When are problem behaviors most likely? Who is engaged in problem behavior?

Why are problem behaviors sustaining?

8-12

Step 2:

Identify Goal for Change

How do we want the problem to change?

What evidence do we need to show that we

Have achieved our goal?

SMART Goals

Specific

Measurable

Achievable

Relevant

Timely Building Goals:

Problem Level Goal

Many students are leaving

garbage in the cafeteria

resulting in conflict and

ODRs.

The behavior is maintained

because students are

rushing to get to the

common area for social

time.

22 ODRs per month from the

cafeteria

Heidi (café supervisor) rates

cafeteria as “1” (low) on a 1-

5 scale of cleanliness.

Less than 5 ODRs per month

from the cafeteria.

Heidi rates cafeteria as

greater than 4 for cleanliness

two weeks in a row.

8-13 8-13

STEP: 3

Identify Solution and Create Implementation Plan with Contextual Fit How are we going to solve the problem?

How are we going to bring about desired change?

Is solution appropriate for problem?

Is solution likely to produce desired change?

SOLUTION PLAN

Precision Statement (Hypothesis):

Goal:

PREVENT What can we do to

prevent the

problem?

TEACH What can we do to

teach to solve the

problem?

REINFORCE What can we do

acknowledge

appropriate

behavior?

EXTINGUISH What can we do to

prevent the

problem behavior

from working or

being rewarded?

CORRECT What will we do to

provide corrective

feedback?

8-15

SOLUTION ELEMENTS ACTION STEPS GENERIC SOLUTION

ACTIONS

PREVENT Focus on prevention first. How

could we reduce the situations

that lead to these behaviors?

o Adjust physical environment

o Define and document

expectations and routines

o Assure consistent and clear

communication with all staff

TEACH How do we ensure that students

know what they SHOULD be

doing when these situations

arise?

o Explicit instruction linked to

school-wide expectations

o Teach what to do, how to do

it and when to do it

o Model respect

REINFORCE How do we ensure that

appropriate behavior is

acknowledged?

o Strengthen existing school-

wide rewards

o Include student preferences

o Use function-based reinforcers

EXTINGUISH How do we work to ensure that

problem behavior is NOT being

rewarded?

o Use “signal” for asking person

to “stop”

o Teach others to ignore (turn

away/look down) problem

behavior

CORRECT How will you correct errors? o Intervene early by using a

neutral, respectful tone of voice

o Label inappropriate behavior

followed by what to do

o Follow SW discipline

procedures

SAFETY Are additional safety

precautions needed?

o Separate student from others if

he/she is unable to demonstrate

self-control

Make sure adult supervision is

available

8-16

A Quick Guide for Initiating Function-Based Solution Brainstorming

When behaviors are driven by the need to gain something (attention, an item, power, etc.),

consider:

Rewarding acceptable/appropriate avenues for gaining the desired stimuli o If attention is desired – teach and reward use of a nonverbal cueing system (cue card, “I need

help” sign, raising hand, performing other task while waiting for right interval for someone to

attend to needs, using words to express need for attention rather other unacceptable avenues

e.g., hitting, biting)

o If an item is desired (gaining an object) – teach and reward behaviors that represent

acceptable ways to gain the item such as asking to borrow a pencil rather than grab a pencil

o If power or status is desired – use alternate means to focus on positive and prosocial qualities of

the student displaying the behavior and consider options such as group rewards for reaching

goals (either individual or group), promoting positive qualities of the individual (e.g., focus on

artistic qualities that are showcased in the school), have the student(s) participate in

demonstration and teaching scenarios with modeling prosocial behaviors for school-wide

instruction, videos, or morning show segments.

When behaviors are driven by the need to avoid something (attention, an item, power,

etc.), consider:

Rewarding acceptable/appropriate avenues for gaining the desired stimuli o If behaviors are driven by a need to avoid activities or tasks – provide rewards for completing

the targeted task, use a reward system to allows students to opt out of completion some

segments of tasks if they complete others (only have to complete even numbered sets of

problems, provide a choice menu for academic tasks to complete during independent seat

work).

o If behaviors signal a desire to avoid social interactions - teach and reinforce visual systems for

taking a break or using a “chill pass”

These are just a few options that you can consider during brainstorming. Use this guide to start

your thinking but remember also to think “outside the box” and use contextual supports and

resources you have in your school. Good luck and thank you for your dedication to creating

a positive and supportive learning environment for students.

RESPONDING AND SUPPORTING BEHAVIORS

Evidence-based Classroom Strategies for Teachers This technical assistance document was adapted from the PBIS Technical Brief on Classroom PBIS Strategies written by:

Brandi Simonsen, Jennifer Freeman, Steve Goodman, Barbara Mitchell, Jessica Swain-Bradway, Brigid Flannery, George Sugai,

Heather George and Bob Putman, 2015.

3.1-3.4 Data Systems

Data Collection Strategy What key strategies can I use to

collect data on student behavior

in my classroom?

Tools and Resources for Data

Collection Method How can I use this to efficiently

track student behavior in my

classroom?

Conditions and Examples For what types of behaviors will

this strategy be appropriate?

Non-Examples of Use For what types of behaviors will

this strategy be inappropriate?

3.1 Counting behaviors

Record or document how often or

how many times a behavior

occurs (frequency) within a

specified period of time; convert

to rate by dividing count by time

(minutes or hours) observed

Moving paper clips from one

pocket to the next

Keeping paper/pencil tally

Using a counter (like counter used

for golf)

App on smartphone or tablet

Behaviors that are discrete (clear

beginning and end), countable

(low enough frequency to count),

and consistent (each incident of

behavior is of similar duration)

Examples:

How often a student swears in

class

How many talk-outs versus hand

raises occur during a lesson

Behaviors that are not discrete

(unclear when behavior begins or

ends), countable (occur too

rapidly to count), or consistent

(e.g., behavior lasts for varying

amounts of time)

Non-examples:

How many times a student is off

task (likely not discrete or

consistent)

How often a student is out of seat

(likely not consistent)

3.2 Timing

Record or document how long:

(a) a behavior lasts (duration from

beginning to end), (b) it takes for

a behavior to start following and

antecedent (latency), or (c) how

much time lapses between

behaviors (inter-response time)

Timer or clock (and recording the

time with paper and pencil

App on smartphone or tablet

Use of vibrating timer (e.g.,

MotiAiders)


beginning and end) and directly

observed

Examples

How long a student spends

walking around the classroom

(duration out of seat)

How long it takes a student to

begin working after work is

assigned

How long it takes a student to start

the next problem after finishing

the last one (inter-response time)


(clear beginning and end) or

directly observed

Non-Examples

How long it takes a student to say

an inappropriate four-letter work

(duration is not the most critical

thing to measure)

How long a student is off task (if

the behavior is not discrete; that is

if the behavior does not have a

clear beginning and end)

8-1

9

3.1-3.4 Data Systems

Data Collection Strategy What key strategies can I use to

collect data on student behavior

in my classroom?

Tools and Resources for Data

Collection Method How can I use this to efficiently

track student behavior in my

classroom?

Conditions and Examples For what types of behaviors will

this strategy be appropriate?

Non-Examples of Use For what types of behaviors will

this strategy be inappropriate?

3.3 Sampling

Estimating how often a behavior

occurs by recording whether it

happened during part of an

interval (partial inter), during the

whole interval (whole interval), or

at the end of the interval

(momentary time sampling)

Shorter intervals lead to more

precise measurement

Partial interval is appropriate for

shorter more frequent behaviors;

whole interval is appropriate for

longer behaviors; and momentary

time sampling facilitates multi-

tasking (you record at the end of

an interval)

Create a table, with each box

representing a time interval (e.g.,

30 seconds), and decide how you

will estimate (partial, whole,

momentary time sampling); use a

stopwatch or app to track each

interval, and record following your

decision rule


(unclear when behavior begins or

ends), countable (occur too

rapidly to count), or inconsistent

(e.g., behavior lasts for varying

amounts of time)

Examples

An estimate of how often a

student is off-task (percentage of

intervals off task)

An estimate of how often a

student is out of seat (percentage

of intervals out of seat)




and consistent (each incident of

behavior is of similar duration)

Non-examples

How often a student swears in

class (you could count this)

How many talk outs versus hand

raises occur during a lesson (you

could count this)

3.4 Antecedent-Behavior-

Consequence (ABC) cards,

incident reports, or ODR’s

Record information about the

events that occurred before,

during or after a behavioral

incident

Paper-and-pencil notes on pre-

populated forms

Electronic data collection method

(e.g., SWIS, Google Docs, other

data based tools)




and both behavior and context

are directly observed or assessed

Examples

A tantrum (cluster of behaviors)

where staff saw what preceded

and followed

A fight among peers where the

vice principal was able to gather

information about what

happened before and after by

interviewing students


(clear beginning and end),

countable (low enough frequency

to count), and/or both behavior

and context are not directly

observed

Non-examples

How often a student swears

(count)

How long a student pauses

between assignments (measure

inter-response time)

step 8: data-based decision making · 1.13 data-based decision making: tier i team reviews and uses...

Documents