guide to demonstrating effectiveness - som - state …. lawanna shelton, education consultant,...

Guide to Demonstrating Effectiveness

School Improvement Grant Office of Education Improvement & Innovation

Michigan State Board of Education

John C. Austin President Ann Arbor

Casandra E. Ulbrich Vice President Rochester Hills

Michelle Fecteau Secretary

Detroit

Pamela Pugh Treasurer Saginaw

Lupe Ramos-Montigny NASBE Delegate

Grand Rapids

Kathleen N. Straus Detroit

Eileen Weiser Ann Arbor

Richard Zeile Detroit

Governor Rick Snyder Ex Officio

Brian J. Whiston Chairman Ex Officio

State Superintendent

Acknowledgements

The Michigan Department of Education and the Office of Education Improvement and Innovation (OEII) would like to thank and acknowledge individuals who assisted in the collaborative development of this

Toolkit at the MDE.

Under the Supervision of: Bill Witt, Unit Supervisor

School Improvement Support Unit

Developer: Dr. LaWanna Shelton, Education Consultant, School Improvement Support Unit

Project Support: Beatrice Barajas-Mills, Analyst


Graphic Design (Front and Back Cover and Artwork): Christi Lopez, Departmental Technician


MDE Review Committee: Dr. Gail Ganakas, SIG Facilitator, MDE Valencia Hughes, SIG Facilitator, MDE

Dr. Karen Lee-Lemieux, SIG Facilitator, MDE Karen Lenz-Wallington, SIG Facilitator, MDE

Tom Kowalski, SIG Facilitator, MDE Mary Smith, SIG Facilitator, MDE

Dr. Ann Trovillion-Timm, SIG Facilitator, MDE

External Review Committee: Gwendolyn Boston, SIG Coordinator, Detroit Public Schools

Inger Brooks, SIG Coordinator, Detroit Public Schools

Chatana Campbell, SIG Coordinator, Roseville Community School District

Jessica Gillard, SIG Coordinator, Lansing School District Jaclyn Gregoricka, Early Childhood Interventionist, Lansing School District

Dr. Alison Harmon, SIG Coordinator, Michigan Technical Academy

Matthew Jason, SIG Coordinator, Lansing School District

Angela Krzebietke, SIG Coordinator, Kalamazoo Public Schools

Sara Hensley, SIG Coordinator, Kalamazoo Public Schools

Robin Porter, SIG Coordinator, Ecorse Public Schools

Ken Rausch, SIG Coordinator, Lincoln Park City School District Wes Seeley, SIG Coordinator, Battle Creek Public Schools

Marylou Sitkiewicz, SIG Coordinator, Litchfield Community Schools

Cynthia White, SIG Coordinator, Ecorse Public Schools

Teressa Williams, SIG Coordinator, School District of River Rouge

Kim Ray Johnson, Principal, Lansing School District Stacie Looney, SIG Coordinator, Lansing School District

Barbara Wells-McKeown, Supervisor Grant Compliance, Detroit Public Schools

Office of Education Improvement and Innovation Linda Forward, Director

Demonstrating Effectiveness To maximize the potential of every dollar in the budget of a competitive grant, it is

imperative that expenditures yield highly effective results. The School Improvement

Grant (SIG) is a competitive grant awarded initially in three-year increments and now in

five-year increments. With the reauthorization of the Elementary and Secondary

Education Act (ESEA), Every Student Succeeds Act (ESSA), the School Improvement

Grant comes to an end. In Michigan, one more and final cohort will be selected for the

2016-2021 five-year award cycle. For the remaining cohorts and newly selected cohort,

the task of strategically determining positions, equipment, services, and professional

development can be better supported with guidelines for demonstrating their

effectiveness.

This document has been designed as a planning tool for recurring expenditures that are

requested in the budget. However, the information may be used to generate

collaborative conversations around these types of expenditures before determining how

to spend a budget. Use the key questions to create a system or tool that evaluates the

impact of personnel, programs, services, and products. Each topic is followed by a

reproducible checklist specifically tailored to address the key questions of each

expenditure.

Finally, this guide is designed to examine expenditures in three broad types. The

corresponding icons have been designed for quick visual reference:

Direct Effect on Students

Adult to Student Interaction

Indirect Effect on Students

Adult to Adult Interaction

Technology

Instructor-less Environments

Instructor-led Environments

Direct Effect on Students

Expenditures that support personnel, services, programs, and products that

have a direct effect on student success and achievement must be

examined. When expenditures fall into this category, demonstrating

effectiveness is directly connected to student outcome data. Outcome data

can be used to measure student success and achievement as well as adult

and product effectiveness. Review your Unpacking Tool and SIG Budget to

determine which expenditures fall into this category. Review the Section

Index for relevant expenditures and evaluate their effectiveness. Use the

reproducible worksheet page to document your findings.

Section Index

I. After School Program

II. Class Size Reduction

III. Field Experiences

IV. Reading Specialist

V. Math Specialist

VI. Instructional Specialist

VII. RTI (MTSS) Specialist

VIII. Para-Professional

3

Afterschool Programs

Key questions to demonstrate effectiveness of Adults: • Are the adults qualified to conduct the afterschool program?

• What training is required?

• Have the adults received the required training?

• Who monitors the adult for fidelity of implementation?

• Who reviews lesson or program plans?

• How often are the adults observed?

• Who evaluates the adults?

Key questions to demonstrate effectiveness of Program/Strategy: • Was the targeted audience achieved?

• How were students selected?

• What baseline data were collected and analyzed?

• What post program data were collected and analyzed?

• How can the afterschool program be credited with student success?

• What role did attendance play?

• What were the assessments provided by the afterschool program?

5

Afterschool Programs Checklist & Worksheet

Review the following documents to support key questions:

Vendor Contracts

Job Descriptions

Résumés

Professional Development Registration

Professional Development Description

Professional Development Sign-in Sheets

List of students who were targeted

Daily Student Attendance Reports

Assessments used for Baseline, Progress Monitoring, and Exit

Can we demonstrate effectiveness to justify continued implementation of the current Afterschool Program?

Yes No List evidence of effectiveness List what is missing or ineffective

Reproducible

6

Class Size Reduction

Demonstrating effectiveness of Adults: Determining effectiveness of the teacher in class size reduction is the responsibility of

the evaluating administrator.

Key questions to demonstrate effectiveness of Program/Strategy: • What criteria was used to establish student selection?

• Based on the established student criteria, were students placed appropriately?

• What baseline data were collected and analyzed to determine a need for a class-size

reduction class?

• What post program data were collected and analyzed to determine continuation of

the class size reduction class?

• What was the overall percentage of student attendance in this class size reduction

class?

• What progress monitoring tool was used?

• What will be the maximum teacher-to-student ratio for this class size reduction?

• What research-based program, activity or initiative will be used to close the

achievement gap/accelerate learning to meet the need?

• stick figure

7

Class Size Reduction Checklist & Worksheet


Class list of students who were targeted

Assessments used to determine CSR need


Progress Monitoring Tool

Teacher-to-Student Ratio Data

Evidence that teacher evaluation is “Effective” or “Highly Effective”

Other:

Other:

Can we demonstrate effectiveness to justify continued implementation of Class Size Reduction?


Reproducible

8

Field Experiences

Key questions to demonstrate effectiveness of Adults: • Who monitors the fidelity of implementing the experience?

• Who reviews the lessons, protocols, and documentation needed to provide approval

for the experience?

• What other supports are needed for the experience to be successful?

• How is adult feedback collected, measured, and used after the experience is

complete?

Key questions to demonstrate effectiveness of Program/Strategy: • How does the field experience support student achievement, success, or attendance?

• Why was the experience selected for the targeted audience?


• What post-data were collected and analyzed?

• What evidence demonstrates the impact of the experience on student achievement,

success, or attendance?

9

Field Experience Checklist & Worksheet


Class list of students who were targeted

Student Attendance during Field Experience

Assessments used to measure learning

Lesson Plans

Field Experience Literature (for example, what does the museum have to offer and

was it used)

Other:

Other:

Can we demonstrate effectiveness to justify continued Support of Field Experiences?


Reproducible

10

Reading Specialist

Key questions to demonstrate effectiveness of Adults: • How are the adult’s actions affecting student performance?

• How are the adults qualified to conduct the math intervention?







Key questions to demonstrate effectiveness of Program/Strategy: • How were students selected?

• Was the targeted audience achieved?

• How many students participated in the program?

• What was the positive impact on students?



• What were the results of program data that demonstrate academic growth?

• How can the Reading Interventionist program be credited with student success?

• What role does attendance play?

• What were the assessments provided by the Reading Interventionist program?

• What strategies/interventions were effective?

• Do the interventionists have the resources/materials needed?

11

Reading Specialist Checklist & Worksheet


Vendor Contracts

Job Description

Résumé







Documentation that demonstrates collaboration between specialist and teachers

Other:

Can we demonstrate effectiveness to justify continued use of a

Reading Specialist?


Reproducible

12

Math Specialist


• How are the adults qualified to conduct the math intervention?







Key questions to demonstrate effectiveness of Program/Strategy: • Was the targeted audience achieved?


• What baseline data was collected and analyzed?

• What post program data was collected and analyzed?

• How can the Math Specialist be credited with student success?


• What are the assessments provided by the Math Specialist?



13

Math Specialist Checklist & Worksheet


Vendor Contracts

Job Description

Résumé








Other:

Can we demonstrate effectiveness to justify continued use of a Math Specialist?


Reproducible

14

Instructional Specialist


• How are the adults qualified to conduct the instructional intervention?







Key questions to demonstrate effectiveness of Program/Strategy:

• Was the targeted audience achieved?




• How can the Instructional Specialist be credited with student success?


• What are the assessments provided by the Instructional Specialist?



• stick figure

15

Instructional Specialist Checklist & Worksheet


Vendor Contracts

Job Description

Résumé








Other:

Can we demonstrate effectiveness to justify continued use of an Instructional Specialist?


Reproducible

16

RTI (MTSS) Specialist

Key questions to demonstrate effectiveness of Adults: • Are the interventionists qualified?


• Have the interventionists received the required training?

• Who monitors the fidelity of implementation of the intervention?

• Who reviews the interventionist's documentation?

• Who evaluates the interventionists?

• How often are the interventionists observed?

• Are the interventionists using the resources/materials with fidelity?

Key questions to demonstrate effectiveness of Program/Strategy:

• Was the targeted audience identified?



• What post intervention data were collected and analyzed? (timeline)

• How can the intervention be credited with student success?


• What assessments were used to determine the effectiveness of the intervention?


17

RTI (MTSS) Specialist Checklist & Worksheet


Vendor Contracts

Job Description

Résumé








Can we demonstrate effectiveness to justify continued use of a RTI (MTSS)?


Reproducible

18

Para-Professionals (Instructional Assistants, Tutors)

Key questions to demonstrate effectiveness of Adults: • What training is required?

• Have the Para-Pros received the required training?

• Who monitors the work?

• What do lead teachers/teachers do to prepare Para-Pros?

• How often do lead teachers/teachers meet, collaborate, and plan with the Para-Pros?

• Who evaluates performance?

Key questions to demonstrate effectiveness of Practice/Strategy:

• Was the target audience achieved?


• What is the measure used to determine student success/achievement?

19

Para-Professional Checklist & Worksheet


Job Description

Schedules/Calendar of Planning with Teacher

Documentation that demonstrates collaboration between Para-Pros and Teachers

List of Students or Records that document who is being serviced, when applicable

Description of the Push-in or Pull-out Model

Pull-out Model: Visual inspection of Pull-out location

Pull-out Model: Amount of time dedicated to transition

Other:

Can we demonstrate effectiveness to justify continued use of a Para-Professionals?


Reproducible

20

Indirect Effects on Students

Expenditures that support personnel, services, programs, and products that

have an indirect effect on student success and achievement must be

examined. When expenditures fall into this category, demonstrating

effectiveness is directly connected to program, process, and perception

data. These data points can be used to measure adult and product

effectiveness. Review your Unpacking Tool and SIG Budget to determine

which expenditures fall into this category. Review the Section Index for

relevant expenditures and evaluate their effectiveness. Use the reproducible

worksheet page to document your findings.

Section Index

I. Professional Learning

II. Instructional Coach

III. Math Coach

IV. Reading/Literacy Coach

21

Professional Learning

Key questions to demonstrate effectiveness of Presenter and Workshop:

• Was the target audience for the workshop achieved?

• Did the presenter enact the agreed upon agenda/topics?

• What did the Professional Learning evaluations indicate?

o Did the presenter begin and end on time?

o Did the presenter engage all participants?

o Did the presenter provide and/or refer to research-based or research-supported

content and resources?

o Was the presenter knowledgeable?

• Were participants receptive to the professional learning?

Key questions to demonstrate effectiveness of Direct Impact on Teacher Learning:

• How was the Learn-Try-Evaluate cycle of job-embedded professional learning

demonstrated?

• What comprehensive data or evidence collected during implementation walk-throughs

support the effectiveness of the professional learning?

Key questions to demonstrate effectiveness of Indirect Impact on Student Learning:

• Can student achievement be directly linked to the profession learning of the teacher,

increase in numeracy or literacy skills, etc.?

• Can student success be directly linked to the professional learning of the teacher, i.e.

fewer discipline issues, increased attendance, etc.?

23

Professional Learning Checklist & Worksheet


Vendor Contracts

Workshop Evaluations

Participation/Attendance Sign-in Sheets

Implementation Walk-through Data

Teacher Survey

Student Survey or Focus Group Notes

Lesson Plans

Can we demonstrate effectiveness to justify continued Training in this particular Professional Learning?


Reproducible

24

Instructional Coach

Key questions to demonstrate effectiveness of Adults: • How is the adult qualified?

• What professional learning is required?

• Who monitors the adult?

• What is the monitoring tool?

• What is the frequency of monitoring?

• Who evaluates the adult?

• What is the evaluation tool?

• How is impact measured?

• What is the goal of the position?


• What tool is used to measure impact on teacher learning?

• How is teacher feedback collected, measured, and used?

• What are the expectations for teachers pre and post coaching?

• What are the goals for teachers pre and post coaching?

• What process is used to select and work with coaches?

25

Instructional Coach Checklist & Worksheet


Vendor Contracts (if position is provided through an ESP)

Job Description

Résumé

Endorsements

Work Logs

Observation Reports

Walk-Through Schedules

Coaching Schedules/Calendars

Can we demonstrate effectiveness to justify continued use of an Instructional Coach?


Reproducible

26

Math Coach
















27

Math Coach Checklist & Worksheet



Job Description

Résumé

Endorsements

Work Logs

Observation Reports



Can we demonstrate effectiveness to justify continued use of a Math Coach?


Reproducible

28

Reading/Literacy Coach
















29

Reading/Literacy Checklist & Worksheet



Job Description

Résumé

Endorsements

Work Logs

Observation Reports



Can we demonstrate effectiveness to justify continued use of a Reading/Literacy Coach?


Reproducible

30

Demonstrating Effectiveness of Technology Instructor-led and Instructor-less Environments

Technology expenditures typically include both hardware and software.

Additionally, these expenditures most often are one-time purchases.

However, when technology expenditures are not one-time purchases,

schools will be asked to demonstrate effectiveness of both hardware and

software before being approved for continuing (i.e. licenses) and/or

additional purchases (i.e. phase II of Interactive Whiteboard initiative).

Review the Section Index for relevant expenditures and evaluate their

effectiveness. Use the reproducible worksheet page to document your

findings.

Section Index

I. Desktop Computers

II. Laptop Computers

III. Tablets

IV. Interactive White Boards

V. Interactive Document Cameras

VI. Annual Software Licenses or Updates

31

Desktop Computers for Student Use

Key questions to demonstrate effectiveness of Instructor-less Environments: • Do students have access to individual computers or will students be expected to

share, i.e. extending a computer lab, adding computers to classrooms?

• How often are the computers used by students?

• When are the computers used?

• What are the computers used for?

• What protocols or guidelines are in place for students to transition to the instructor-

less environment?

• What is the assignment protocol?

Key questions to demonstrate effectiveness of Direct Impact on Student Learning:

• Is there a protocol in place to ensure students have a working machine accessible?

• Do students have restricted access to internet?

• What do students say about using the computers?

• What type of data is collected to demonstrate the impact of the use of technology?

Key questions to demonstrate viability of the hardware:

• Can students easily manipulate the computers?

• Are the computers age appropriate?

• Are peripherals needed to use the computer effectively, i.e. earphones, speakers?

• Does the school and/or district have the supporting infrastructure?

• Is there a technology plan in place and being used (i.e. repair, replacement)

• What technical support is in place to assist in minimizing interruptions to learning?

• Is make or model of the computer problematic, i.e. too many glitches, hard boot

required too often, etc.?

33

Desktop Computer Checklist & Worksheet


Computer Lab Schedules


Teacher Survey


Lesson Plans

Inventory reports

Help Desk Tickets or Logs

Student Login Information

Computer Usage Reports

Copy of Technology Plan

Can we demonstrate effectiveness to justify additional purchase of Desktop Computers?


Reproducible

34

Laptops for Student Use

Key questions to demonstrate effectiveness of Instructor-less Environments:

• Is this a 1-to-1 initiative?

• How often are the laptops used by students?

• When are the laptops used?

• What are the laptops used for?

• What protocols or guidelines are in place for students to transition to the instructor-

less environment?

• What is the check out or assignment protocol?


• Is there a protocol in place to ensure students have a working machine accessible?

• What do students say about using the laptops?




• Can students easily manipulate the laptops?

• Are the laptops too heavy or too bulky for students?

• Are the laptops age appropriate?

• Are peripherals needed to use the laptops effectively, i.e. earphones, speakers?




• Is make or model of the laptop problematic, i.e. too many glitches, hard boot

required too often, etc.?

35

Laptop Computer Checklist & Worksheet


Computer Lab Schedules


Teacher Survey


Parent Feedback (if 1:1 initiative allows for home use)

Parent Training Sign-In Sheets (if 1:1 initiative allows for home use)

Lesson Plans

Inventory reports


Student Login Information or Laptop Assignment Information


Can we demonstrate effectiveness to justify additional purchase of Laptop Computers?


Reproducible

36

Tablets for Student Use


• Is this a 1-to-1 initiative?

• Do the tablets go home with students?

• What protocols or training have been developed for parents if tablet goes home with

students?

Key questions to demonstrate effectiveness of Direct Impact on Student Outcomes:

• What do students say about using the tablets?




• Can students easily manipulate the tablets?

• Are the tablets too heavy for students?

• Are the tablets too fragile for students?

• Are the tablets age appropriate?

• Are peripherals needed to use the tablets effectively, i.e. earphones, speakers?




• Is make or model of the tablet problematic, i.e. too many glitches, hard boot required

too often, etc.?

37

Tablet Checklist & Worksheet



Teacher Survey


Parent Feedback (if 1:1 initiative allows for home use)

Parent Training Sign-In Sheets (if 1:1 initiative allows for home use)

Lesson Plans

Inventory reports


Student Login Information or Tablet Assignment Information


Can we demonstrate effectiveness to justify additional purchase of Tablets?


Reproducible

38

Interactive Whiteboards for Classroom Use

Key questions to demonstrate effectiveness of the Interactive Whiteboard:

• How often is the board used in an interactive mode?

• How often is the board expected to be used in interactive mode?

• What is the board used for?

• Do students manipulate and interact with the board?

Key questions to demonstrate effectiveness of Direct Impact on Teaching:

• What training is needed by teachers to use the board effectively?

• How has the board enhanced teaching?

• Do teachers have access to the software needed to use the board in interactive

mode?


• What do students say about using the board?

• What do students say about learning through this medium?


• Do teachers have easy access to the board?

• Where is the board located in the classroom?

• Are computers compatible with the board?




• Is make or model of the interactive whiteboard problematic, i.e. too many glitches,

hard boot required too often, etc.?

39

Interactive Whiteboard Checklist & Worksheet



Training Agendas and Sign-in Sheets

Teacher Survey


Lesson Plans



Can we demonstrate effectiveness to justify additional purchase of Interactive Whiteboards?


Reproducible

40

Interactive Document Cameras for Classroom Use

Key questions to demonstrate effectiveness of the Interactive Document Camera:

• How often is the camera used?

• How often is the camera expected to be used?

• What is the camera used for?

• Do students interact with the camera?

Key questions to demonstrate effectiveness of Direct Impact on Teaching:

• What training is needed by teachers to use the camera effectively?

• How has the camera enhanced teaching?

• Do teachers have access to the software needed to use the camera effectively?


• What do students say about using the camera?

• What do students say about learning through this medium?


• Do teachers have easy access to the camera?

• Is the camera easy to manipulate, i.e. not too heavy, bulky:

• Where is the camera located in the classroom?

• Are computers compatible with the camera?




• Is make or model of the document camera problematic, i.e. too many glitches, hard

boot required too often, etc.?

41

Interactive Document Camera Checklist & Worksheet




Teacher Survey


Lesson Plans



Can we demonstrate effectiveness to justify additional purchase of Interactive Document Cameras?


Reproducible

42

Annual Software Licenses or Updates


• Was the target audience achieved?

• Were target goals achieved?

• Is software program designed to be implemented without an adult?

• Is software program self-explanatory, i.e. provides tutorials, exemplars, etc.?

• Is there a minimum or maximum amount of time students are to be engaged in

activity or lesson?

Key questions to demonstrate effectiveness of Instructor-Led Environments:

• Is the teacher/instructor trained in the use of the software program?

• What training is needed?

• How much time is dedicated to set-up and preparation?

• What is the teacher’s role?


• How do students receive immediate feedback?

• How do students track their own progress?

• How do students transition to an instructor-less environment?

• What pre and post assessments are used?

• How many similar software programs do students have access to?

Key questions to demonstrate software compatibility:

• Can software be used on multiple platforms, i.e. PC, Mac, tablets, etc.?

• Is software stand-alone or web-based?

• If web-based, can software be used with multiple browsers, i.e. Safari, Chrome, etc.?

43

Annual Software Licenses/Updates Checklist & Worksheet




Teacher Survey


Pre and Post Assessment Data

Achievement Data

Growth Data

Lesson Plans



Can we demonstrate effectiveness to justify renewal of software licenses or software updates?


Reproducible

44

STUDENT

TE

AC

HER PA

RE

NT

guide to demonstrating effectiveness - som - state …. lawanna shelton, education consultant,...

Documents