northern california pbis symposium november 18, 2013
TRANSCRIPT
![Page 1: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/1.jpg)
Using PBIS Assessment to Create School and District
PBIS Evaluation Plans
Northern California PBIS SymposiumNovember 18, 2013
Are we doing what we said we would do?
![Page 2: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/2.jpg)
Agenda
Overview of available surveys/tools on the PBIS Assessment website
Developing an Evaluation Action PlanDecisions on which tools to chooseDecisions on timelines
Assessment Example Plans
Developing Evaluation Reports
Questions/Comments
![Page 3: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/3.jpg)
Acknowledgments
School-wide Positive Behavior Support: Implementers' Blueprint and Self-Assessment
Sugai, G., Horner, R.H., Algozzine, R., et.al. (2010).
Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support
Algozzine, R., Horner, R.H, Sugai, G., et.al. (2010).
Fidelity Measures to Improve Implementation of Positive Behavioral Support
Tobin, T. J., Vincent, C. G., Horner, R. H., Rossetto Dickey, C., & May, S. A. (2012).
![Page 4: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/4.jpg)
PBIS Evaluation Blueprint
“A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.”
“Evaluation is the process of collecting and using information for decision making.
![Page 5: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/5.jpg)
Foundations
Evaluation is the process of gathering data toAnswer questionsMake decisions
Define the core questions/decisions, before selecting measures.
Provide an adequate context for interpreting the data.
Effective evaluation is cyclical (repeated)Use the evaluation process for improvement
![Page 6: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/6.jpg)
Evaluation Cycle
Core Questions
Evaluation Plan
Data Collection
Evaluation Report
![Page 7: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/7.jpg)
Core Evaluation QuestionsContext
What are/were the goals and objectives for SWPBIS implementation? (state/district capacity; school adoption; student outcomes)
Who delivered the training and technical assistance for SWPBIS implementation?
InputWhat professional development events were part of SWPBIS implementation support?
Was projected level of TA capacity provided (training/coaching)?
Who received the professional development (schools/ cohorts)?
![Page 8: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/8.jpg)
Core Evaluation QuestionsFidelity
To what extent was Tier I SWPBIS implemented with fidelity ?
To what extent were Tier II and Tier III SWPBIS implemented with fidelity?
To what extent is the Leadership Team establishing core functions?
![Page 9: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/9.jpg)
Leadership Team
Active Coordination
FundingVisibility Political
Support
Training Coaching Evaluation
Local School/District Teams/Demonstrations
BehavioralExpertise
Policy
Sugai et al., www.pbis.org
![Page 10: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/10.jpg)
Core Evaluation QuestionsImpact
To what extent is SWPBIS associated with changes in student behavioral outcomes?
To what extent is SWPBIS associated with changes in academic performance, and dropout rates?
To what extent is district/state capacity established? (local training, coaching, evaluation, behavioral expertise)
To what extent is leadership and policy structure established?
![Page 11: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/11.jpg)
Core Evaluation Questions
Replication, Sustainability, and ImprovementTo what extent do Fidelity and Impact outcomes sustain across time?
To what extent does initial SWPBIS implementation affect Implementation with later cohorts?
To what extent did SWPBIS implementation change educational/behavioral capacity/policy?
To what extent did SWPBIS implementation affect systemic educational practice?
![Page 12: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/12.jpg)
Evaluation Questions
FidelityTo what extent was SWPBS implemented as designed?
To what extent was SWPBS implemented with fidelity?
“Are we doing what we said we would do?”
![Page 13: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/13.jpg)
To What Extent Can
PBIS Applications.org
PBIS Assessment
PBIS Evaluation
Help us answer the question:“Are we doing what we said we would do?”
![Page 14: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/14.jpg)
Current Tools in PBIS Assessment
Research Tool
Annual Assessment
Tool
Progress Monitoring
Tool
UniversalSystem
School-wide Evaluation Tool
(SET)
Benchmarks of Quality (BoQ)
Early Childhood BoQ (ECBoQ)
PBIS Self-Assessment Survey (SAS)
Team Implementation
Checklist (TIC)
Secondary &TertiarySystems
Benchmarks of Advanced Tiers (BAT)
PBIS Self-Assessment Survey (SAS)
Monitoring Advanced Tiers
Tool (MATT)
Outcome Measures: School Safety Survey
![Page 15: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/15.jpg)
Current Tools in PBIS Assessment
Research Tool
Annual Assessment
Tool
Progress Monitoring
Tool
UniversalSystem
School-wide Evaluation Tool
(SET)
Benchmarks of Quality (BoQ)
Early Childhood BoQ (ECBoQ)
PBIS Self-Assessment Survey (SAS)
Team Implementation
Checklist (TIC)
Secondary &TertiarySystems
Individual Student School-wide Evaluation
Tool(ISSET)
Benchmarks of Advanced Tiers (BAT)
PBIS Self-Assessment Survey (SAS)
Monitoring Advanced Tiers
Tool (MATT)
Outcome Measures: School Safety Survey
![Page 16: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/16.jpg)
www.pbisapps.org
A Tour of
at
![Page 17: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/17.jpg)
Evaluation Action PlanningDecisions:
Evaluation ToolsWhich tools answer your district/schools’ evaluation questions?
Evaluation TimelineWhen/how often will you administer the tools?
Evaluation ReportingHow will you report outcomes to schools and to the District?How often will you report outcomes to schools and to the District?
![Page 18: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/18.jpg)
Evaluation Action Planning
Decisions about Evaluation ToolsDoes your District/School need a research tool?
(SET/ISSET)
Which annual assessment tool(s) will you use?BoQ, SAS, SSS, BAT, ECBoQ
Which progressing monitoring tool(s) will you use?TIC, MATT
Does your District/School need to evaluate Tier I or Tier II & III implementation or all three?
![Page 19: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/19.jpg)
Evaluation Action Planning
It is about…
CONTEXTYour settings => Your needs
![Page 20: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/20.jpg)
Which Surveys Meet Your
District’s Needs?
![Page 21: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/21.jpg)
Example District A
Evaluation Questions
What is our baseline, and what has been our progress through the year?
Are we improving on our implementation of PBIS?
How do our individual staff members rate our implementation?
Are our self-assessments consistent with data from outside evaluators?
Newly implementing, grant needing specific research data as well as fidelity of implementation data
![Page 22: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/22.jpg)
District A: Evaluation Plan
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SET X X X X
TIC X X X X X X X X X
Safety Survey X X X X
SAS X X X X
![Page 23: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/23.jpg)
Example District B
Evaluation Questions
In what specific PBIS areas are we strong, and in which areas do we need more training?
Are we improving on our implementation of PBIS?
How do our staff members actually rate our PBIS implementation?
How is our implementation of Tiers II and III?
Implementing PBIS for three years, have been using the TIC/SET, and have been consistently over 80%.
![Page 24: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/24.jpg)
District B: Evaluation Plan
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
BoQ X X X
SAS X X X
Safety Survey X X X
ATT X X X X X X X X X
![Page 25: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/25.jpg)
Example District C
Evaluation Questions
Which schools are not meeting the SET benchmark?
In what specific PBIS areas are we strong, and in which areas do we need more training?
How can we tailor our training and evaluation to meet the needs of various schools?
Are we improving on our implementation of PBIS?
How do our individual staff members rate our implementation?
Implementing for five years, some schools are doing very well, but others are not making progress.
![Page 26: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/26.jpg)
District C: Evaluation Plan: Cohort 1
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SASSurvey X
SET X X X
Safety Survey X X X
TIC X X X X X X X X X
Schools still not performing well
![Page 27: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/27.jpg)
District C: Evaluation Plan: Cohort 2
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SASSurvey X
BoQ X X X
Safety Survey X X X
MATT X X X X X X X X X
Schools performing well
![Page 28: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/28.jpg)
Example District D
Evaluation Questions
In what specific PBIS areas are we strong, and in which areas do we need more training?
In what academic areas are we strong, and in which areas do we need more improvement?
How can we tailor our training/evaluation to meet the needs of various schools?
How can we schedule our evaluations so that we are not overwhelming our schools?
The district wants to bring RtI Behavior and Academics together and create one evaluation plan.
![Page 29: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/29.jpg)
Assessment Instrument
Fall (August/Sept/Oct)
Winter (Dec/Jan)
Spring(April/May) Purpose
Team Implementation Checklist (TIC)
X X X Fidelity of Implementation
Self-Assessment Survey
X Fidelity of implementation
School Safety Survey
X Outcomes related to Protective and Risk Factors
SET X Fidelity of Implementation
AIMsWeb X X XScreening
Progress Monitor
SAT10 X Summative Evaluation
DIBELS X X X Screening
Progress monitoring
Nieves Flores, 2012
![Page 30: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/30.jpg)
Task: Developing Your Plan
Questions to Consider:Which surveys are your schools/districts currently using?Do they answer your evaluation questions:
Are we implementing SWPBIS with fidelity?Is there more information that we need?
Pbisapps.org > PBIS Assessment > Find Out More about Surveys
Do your surveys assess the Tiers you are implementing?Do you need a research tool?
![Page 31: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/31.jpg)
Task: Developing Your Plan
Are you “topping out” of the surveys you are currently taking?
Do you need to customize your evaluation plan for different schools, or will one evaluation plan fit for all schools?
Do you want to assess all tiers of PBIS implementation?
Is your Evaluation Plan manageable for schools?
![Page 32: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/32.jpg)
District Evaluation Planning Template
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
![Page 33: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/33.jpg)
SWPBIS School or District Evaluation Reports
![Page 34: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/34.jpg)
Basic Contents of SWPBIS Evaluation Reports
From the PBIS Evaluation BlueprintContext
Input
Impact on SWPBIS Fidelity
Impact on Student Outcomes
Implications for the Future
![Page 35: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/35.jpg)
Context
Define SWPBIS
Define goals of the specific project
Define stakeholders and evaluation questions
![Page 36: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/36.jpg)
Input
Who received what support and from whom?Leadership team
Local Capacity Building Training, Coaching, Behavioral Expertise, and Evaluation
School teams
![Page 37: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/37.jpg)
Impact on SWPBIS Fidelity
Leadership TeamSWPBIS District Leadership Team Self-Assessment
Capacity DevelopmentNumber of trainers/coaches available to support school teams/districtsBehavioral expertise available to support Tier II and Tier III implementationEvaluation capacity
School TeamsTier I Implementation (TIC, BoQ, SET, SAS)
Collectively and/or by training cohortTier II / Tier III Implementation (MATT, BAT, ISSET)
Collectively and/or by training cohort
![Page 38: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/38.jpg)
Impact on Student Outcomes
Behavioral OutcomesOffice Discipline ReferralsSuspension/ExpulsionAttendanceDrop Out/GraduationDisproportionality in ODR and suspension/expulsion
Academic outcomesCurriculum Based Measures
Oral Reading Fluency, Dibels, AimsWebStandardized Measures
State and NationalMath, reading, writing assessments
![Page 39: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/39.jpg)
Impact and Future Directions
Advances to datePlans for next three yearsAreas in need of adaptation
Narrowing of evaluation questions and concerns
![Page 40: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/40.jpg)
Sharing Evaluation Reports
State and District examples at PBIS.orgSend in your district examples, so we can all learn!
![Page 41: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/41.jpg)
Support and Resources
Portal Support SectionSearch FunctionCommunity ForumChat with us online
Account Questions:[email protected]
Technical Assistance Questions:
New phone number1-855-455-8194
![Page 42: Northern California PBIS Symposium November 18, 2013](https://reader036.vdocuments.us/reader036/viewer/2022070323/56649dc55503460f94ab8cd1/html5/thumbnails/42.jpg)
Questions & Answers
When you leave the webinar, please complete the survey.
THANK YOU!