pbmas 101 what’s the score??? go for the touchdown with esc-2

PBMAS 101What’s the Score???

Go for the Touchdown with ESC-2

PBMAS=

• Performance

• Based

• Monitoring

• Analysis

• System

PBMAS Report• Automated data system that reports on the performance of

school districts and charter schools in the 4 Program Areas below:

– Bilingual– Career and Tech– No Child Left Behind– Special Education

• http://www.tea.state.tx.us/pbm/regrpts.html

TEA’s Monitoring Systems:Historical Context

•78th TX Legislative 2003 Session

•Resulted in a realignment of TEA’s functions

•Emphasis on a coordinated approach to TEA’s monitoring to districts

PBM Overall Coordination

Performance-Based

Monitoring

Program Monitoring and Interventions

Program Areas

Historical Context Different Monitoring Components

•PBMAS

•Data Validation System

•Federal Compliance

•State Accountability System (AA, AU, etc.)

•Federal Accountability System (AYP)

Historical Context Different Monitoring Components

•Financial Integrity Rating System (FIRST)

•Financial Audits

•Complaints

•Due Process Hearings

•Governance

•Other Monitoring such as Office for Civil Rights

A Look into One of the TEA Monitoring Components:

PBMAS

PBMAS Indicators• Types of indicators generally fall into these

categories:

– Student performance indicators •(ie TAKS Passing Rate; annual dropout

rate)

– Program effectiveness indicators •(ie TAKS Participation Rate)

10 Guiding Principles of PBM

1. School District Effectiveness

2. Statutory Requirements

3. Valid Indicators of Performance

4. Maximum Inclusion

5. Individual Program Accountability6. High Standards

7. Annual Statewide Focus

8. Public Input and Accessibility

9. Systems Evolution

10. Coordination

PBMAS Components

1. Data Sources

2. Filters3. No Data Available for an Indicator

4. Rounding

5. Masking

6. Standards & Performance Levels

PBMAS Components

7. Report Only Indicators

8. Minimum Size Requirements and Special Analysis

9. Required Improvement

10.System Safeguards

Component #1Data Sources

•Student Assessment•Dropouts•Annual Graduation Data•Longitudinal Graduation Data•AEIS•PEIMS

Component #2Filters

May be necessary before

Performance Indicators are calculated

i.e. Residential Facilities increase

% of special education identified students in a program

Component #3No Data Available for an Indicator

• District can’t be evaluated for absence of data.

• Data validation analysis and reviews are conducted by TEA to ensure proper coding…

i.e. coding for migrant students: Is the district identifying and coding students appropriately?

Component #4Rounding

Calculations rounded to one decimal point

Component #5Masking

To protect student identity

Component #6Standards and Performance Levels (PL)

• PL = Level System in which a standard is applied to a performance on an indicator:• 0 = Met Standard• 1• 2• 3= farthest from meeting standard

• SA= Special Analysis• PJSA=Professional Judgment Sp Analysis

Component #6

Performance Levels (PL) Example:

TAKS Reading Standard = 70%

If a district meets or exceeds the 70% in example, it would receive a

“0” PL

Component #6

Standards and Performance Levels (PL)

2 Types of Standards:

Absolute and Relative

Component #6Standards and Performance Levels (PL)

Absolute Standards:

• 0 PL = Met Standard

• “1” PL = 0.1 - 10% below standard

• “2” PL = 10.1 -20.% below standard

• “3” PL = 20.1% or below

Component #6Standards and Performance Levels (PL)

Relative Standards:

based on the distribution of scores to the population being evaluated

ESC-2 Example of Relative Standard

Description

Indicator 12

2007

State

ID Rate

2007

ESC-2

ID Rate

2007

PL

Assigned

Calculation

Sped ID

8.5% 12.1% 2 12,844

105,828

Example of Relative StandardPL =

Not Assigned

PL = 0 PL = 1 PL = 2 PL = 3

8.5% or lower!

8.6% - 12% 12.1%-16% 16.1% or up

Changes to Standards• Changes are made after careful consideration.

Following list includes a few: – Actual improvement over time on an indicator– Overall impact on PBMAS of increased standards– External input– Whether a state or federal goal has been identified

See chart on page 8 of manual

Component #7: Report Only Indicators

Gives a district a “preview” for planning purposes.

No PL assigned for 2 years

Component #8: Minimum Size Requirements

and Special Analysis

•At least 30 students are needed for calculations.

•Can be met in current year or over 2 recent years.

•Always in the favor of the district

Component #8: Minimum Size Requirements

and Special Analysis

•If a district meets standard regardless of amount of students, then district receives a “0” PL.

Component #8: Minimum Size Requirements

and Special Analysis

•Special Analysis: for districts and charters that do NOT meet the minimum size requirements.

Component #8 (continued)

• There are two types of special analysis in PBMAS:

–Automated special analysis–Professional judgment special analysis (PJSA)

Component #8 (continued)

• GROUP SIZE DETERMINES TYPE OF ANALYSIS:– 30 or more in current year or over two

years: STANDARD ANALYSIS is applied.– SPECIAL ANALYSIS is applied in different

ways depending on the following group sizes:• 15-29• 5-14• Less than 5

Component #8 (continued)

• Special analysis does not apply to indicators for which the district has NO students in a particular student group; these districts receive a designation of No Data (ND).

• Example: Neutral ISD has no CTE students. The district receives a designation of ND – No Data for the CTE TAKS Passing Rate indicator.

Component #8 (continued)

• Special analysis (continued)

– Does not apply to indicators for which the district performance level is 0 – Met Standard, regardless of student group size.

• Example: Terrific ISD has four CTE students, and all four of these students met the TAKS passing standards. The district performance level on the CTE TAKS Passing Rate indicator is 0 – Met Standard.

Key Monitoring Concepts (continued)

• Example #1 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator– Current Year = 4/10 (PL2) – Prior Year = 4/10 (PL2) – PL = 2SA because of group size and PL

is the same for both years.

Key Monitoring Concepts (continued)

• Example #2 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator– Current Year = 4/10 (PL2)– Prior Year = 7/15 (PL1)– PL = 1SA because neither is a PL3 so

the better of two PLs is assigned.

Key Monitoring Concepts (continued)

• Example #3 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator (50% = 0)– Current Year = 1/10 = 10% (PL3)– Prior Year = 4/10 = 40% (PL2)– PL = determined by professional judgment

special analysis.

Key Monitoring Concepts (continued)

• Example #4 for Group Size 15-29: If the same performance results were for Sped Year After Exit rather than Reading, the PL=NA (Not Assigned) because professional judgment special analysis is not currently available.

Key Monitoring Concepts (continued)

• Example #5 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator – Current Year = 1/10 (PL3)– Prior Year = 8/10 (PL0) – PL = NA because PLs are more than one

level apart over two years.

Key Monitoring Concepts (continued)

• Example #1 for Group Size 5-14: District’s Performance on a TAKS

Reading Indicator – Current Year = 5/11 (PL1)– Prior Year = 2/2 (PL0) – PL = NA because size of student group.

Key Monitoring Concepts (continued)

• Example # 2 for Group Size 5-14: District’s Performance on a TAKS Reading Indicator – Current Year = 5/11 (PL1) – Prior Year = 0/2 (PL3)– PL = NE because PLs are more than one

level apart.

Key Monitoring Concepts (continued)

• Example #3 for Group Size 5-14: Current District’s Performance on a TAKS Reading Indicator– Current Year = 0/7 (PL3) – Prior Year = 2/7 (PL3) – PL = determined by professional judgment

special analysis.

Key Monitoring Concepts (continued)

• Example #4 for Group Size 5-14: If the same performance results were for Social Studies rather than Reading, the PL=NE because professional judgment special analysis is not currently available for TAKS Social Studies.

Key Monitoring Concepts (continued)

• Example #1 for Group Size Less Than 5: District’s Performance on a TAKS Reading Indicator – Current Year = 0/2 (PL3) – Prior Year = 0/2 (PL3) – PL = NE because of size of student group.

Component #9: Required Improvement

•Built-in system to help districts that demonstrate improvement from one year to the next.

•List of indicators on pages 15-16

Component #9: Required Improvement

Example: Indicator 12 Sped Identification

•Calculate actual difference from 2008- 2007:

15% - 25% = -10%

•Calculate Required Improvement for the sped identification rate: 8.5% (state) – 25%

2 (2009-2007)

•Compare the two numbers to see if there is an actual change < or = to RI. -10.0 < -8.25 = 0 PL

= -8.25

Component #10: System Safeguards

• To ensure “Integrity”

• Uses Data Validation Analyses:

1.Leaver Data

2.Student Assessment Data

3.Discipline Data

4.Other Means

System Safeguard “Other Means”:

• Accuracy of data reporting!• Implementation of intervention activities!!• Implementation of plans for improvement or

correction• Accuracy of findings made through PBM process.

Let’s Jump to INTERVENTION

LEVELSThese are different than the

PL #’s for each indicator within each program.

Intervention Levels

• Levels are assigned per program area as an overall “stage” level:– Bilingual– CTE– NCLB– Special Education

• Districts must work through intervention activities and upload documents to TEA on different deadlines.

Basic Intervention Activities

•Focused Data Analysis•Program Effectiveness Review•LEA Public Meeting•Compliance Review•Continuous Improvement

Planning

Accessing PBMIntervention Documents

• After reviewing the district PBMAS reports / Performance Levels, click on the following link for Interventions:

• http://www.tea.state.tx.us/pmi/

Who Ya Gonna Call???

• Bilingual: Velma Escamilla velma.escamilla@esc2.us

• CTE: Bob Chapman bob.chapman@esc2.us

• NCLB: Andi Kuyatt andi.kuyatt@esc2.us

• Special Ed: Barbie Tumlinson barbie.tumlinson@esc2.us