Bringing Evidence to Bear: What Facilitates Adoption of Research-based Evidence in Practice and Policy-

making in Schools

Large Group PanelOSEP Project Director’s Conference 2014

Washington, DC

Session Goals

• Discuss organizational and other factors that facilitate take-up of new knowledge and facilitate implementation of innovative programs and policies

• Discuss program features that lends itself to early adoption and uptake of programs in the school environment

• Adopt a systems perspective in intervention development for improved adoption

Panel

Presenters• Dr. Vivian Tseng, Vice President, William T. Grant Foundation,

New York, NY• Dr. David Test, Professor, Department of Special Education

and Child Development, Co-Director, National Secondary Transition Technical Assistance Center, UNC Charlotte, NC

• Dr. Arun Karpur, Research Director, New York State PROMISE, Research Faculty, Cornell University, New York, NY

Moderator: • Dr. David Guardino, Research to Practice Division, U.S.

Department of Education, Office of Special Education Programs

Format of LGP

• Presentation by individual presenters• Q&A session moderated by Dr. Guardino• Discussion with the attendees and the speakers

Promoting the Use of Research Evidence

Vivian Tseng, Ph.D.

Supporting Research to inform policy and practice to Improve the Lives of Young People

Evidence-based Programs

Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities (2006)

Research to Practice

Improve Quality of Research Improve Communication, Dissemination, Marketing Demand Use of Research Evidence Incentivize Evidence-based Programs

Research and Practice

Improve Research and Its Use Develop Shared Commitments Build Relationships and Trust

Organizations & Systems

Intermediaries

What’s an NSTTAC?National Secondary Transition

Technical Assistance Center

The Transition to College and Career TA&D Center for secondary transition funded by USDOE, OSEP

(1/1/06-12/31/14)

Co-Directors: David W. Test (UNC Charlotte)

Paula Kohler (Western Michigan University)

Project Coordinator: Cather Fowler (UNC Charlotte)

NSTTAC’s Purpose• Disseminate information to State Education Agencies, Local

Education Agencies, schools, and other stakeholders to improve the:

• implementation and scaling up of evidence-based practices to develop appropriate measurable postsecondary goals and implement transition services

• implementation of policies, procedures, and practices that facilitate and increase the participation of students with disabilities in programs and initiatives that are designed to ensure college- and career-readiness

• achievement of compliance with IDEA’s transition requirements

• Support efforts to ensure that all students with disabilities are prepared for college (or other postsecondary education and training) and the workforce.

Model for Extending Transition Research

Effective Transition Practices

Increase Capacity to Implement Effective Transition Practices

Facilitate Implementation of Effective Transition

Practices

Data-Based Decision Making

Professional Development

Policy Analysis and

Change

Technical Assistance

Broad Definitions(Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Twyman, 2008)

• Are based on rigorous research designs • Have demonstrated a record of success for

improving student outcomes• Have undergone systematic review process

using quality indicators to evaluate level of evidence

Evidence-Based Practices

• Are based on rigorous research designs • Have demonstrated a record of success for

improving student outcomes

Research-Based Practices

• Are based on research• Have demonstrated limited success• Have used a ‘weak’ research design

Promising Practices

• Are not based on research• Have no data to support effectiveness• Based on anecdotal evidence and/or

professional judgment

Unestablished Practices

What We Have Done• Reviewed experimental research to identify

evidence-based practices in secondary transition

• Identified 63 evidence-based practices

Taxonomy Category

Evidence-Based Practices

Research to Practice Lesson

Plan Starters

Student Focused Planning

6 9

Student Development 56 98

Family Involvement 1 0

Program Structure 9 9

Student Development

Teaching Grocery Shopping Skills

• Using Computer Assisted Instruction

• Using Community Based Instruction

• Using Response Prompting

• Using a System of Least to Most Prompts

Teaching Home Maintenance Skills

• Using Response Prompting

• Using Video Modeling

Teaching Laundry Tasks Using Response Prompting

Teaching Leisure Skills • Using Response Prompting

• Using Constant Time Delay

Teaching Safety Skills • Using Community Based Instruction

• Using Progressive Time Delay

• Using a System of Least to Most Prompts

Based on High-Quality Research, the Field of Secondary Transition now has Evidence-based:

Practices

Micro Level Specific

interventions

Predictors!

Macro Level• Systems• Programs• “Generic” practices

In-School Predictors by Post-School Outcome Area

Predictors/Outcomes Education Employment Independent Living

•Career Awareness X X

•Occupational Courses X X

•Paid Employment/Work Experience X X X

•Vocational Education X X•Work Study X•Community Experiences X

•Exit Exam Requirements/High School Diploma Status

X

•Inclusion in General Education X X X

•Program of Study X

Continuation of In-School Predictors by Post-School Outcome Area

Predictors/Outcomes Education Employment Independent Living

•Self-Advocacy/Self-Determination X X

•Self-Care/Independent Living X X X•Social Skills X X•Interagency Collaboration X X•Parent Expectations X X•Parental Involvement X•Student Support X X X•Transition Program X X

Levels of TA

• Universal/general– Website– Annual Capacity Building Institute

• Targeted/specialized– State specific topics/issues

• Intensive/sustained– State & LEA specific topic/issues

Data-Driven Decision Making Tools (Some Samples)• Predictor Implementation Self-Assessment• Data Use Toolkit (NPSO – Indicator 14)• State Toolkit for Examining Post-School

Success (STEPSS; Indicators 1, 2,13, & 14)• NDPC-SD Data Tools lite (Indicators 1 & 2)• NSTTAC Planning Tool (multiple data

sources,including from other tools)

• Short Data Probe (Indicator focused)• District Initiative Inventory (qualitative,

broad focus)

Assessing Evidence-Based Programs and Practices Need

Need in school, district, state• Academic & socially significant Issues• Parent & community perceptions of need• Data indicating need

Fit

Fit with current Initiatives •School, district , state priorities• Organizational structures• Community values

ResourceAvailability

Resources and supports for;• Curricula & Classroom• Technology supports (IT dept.)• Staffing• Training• Data Systems• Coaching & Supervision• Administration & system

Evidence

Evidence• Outcomes – Is it worth it?• Fidelity data• Cost – effectiveness data • Number of studies• Population similarities• Diverse cultural groups• Efficacy or effectiveness

Intervention Readiness for Replication

Readiness• Qualified purveyor • Expert or TA available• Mature sites to observe • Several replications• How well is it operationalized?• Are Imp Drivers operationalized?

Capacity to Implement

Capacity• Staff meet minimum qualifications• Able to sustain Imp Drivers

• Financially• Structurally

• Buy-in process operationalized• Practitioners• Families• Agency

EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4.

High Medium Low

Need

Fit

Resources Availability

Evidence

Readiness for Replication

Capacity to Implement

Total Score:

© National Implementation Research Network 2009

Adapted from work by Laurel J. Kiser, Michelle Zabel,

Albert A. Zachik, and Joan Smith at the University of Maryland

Organizational & Program Factors

• Commitment to Adopting EBPs• Readiness• Flexibility• Communication• Time

Utilizing Continuous Quality Improvement for Promoting

Adoption and Uptake of KnowledgeNYS PROMISE Approach

Arun Karpur, MD, MPHResearch DirectorNYS PROMISEEmployment and Disability Institute, Cornell University

NYS PromiseNYS PROMISE will utilize an indigenous model that naturally equips and engages schools, local disability service providers, independent living centers, one stop centers, literacy zones, regional parent training centers, work incentive planners, regional transition specialists, and other community transition stakeholders to achieve higher postsecondary employment, education and economic outcomes for SSI youth by...

NYS PROMISE InterventionIn

crea

sed

Acce

ss to

Sta

te a

nd N

ation

al T

A on

Tra

nsiti

on B

est P

racti

ces

Services under outcomes-based payment model

Improved access to key high quality

services leading to intended outcomes

PROMISE-sponsored case management & service coordination

Improved collaboration

between LEAs and local service providers

CQI Feedback Loops : Project M

IS , Fidelity Assessment, and TA

PROMISE-sponsored parent training,

information & family coaching

Improved self-determination, expectations,

engagement, attitudes toward work and financial literacy

Higher post-secondary employment, education and economic outcomes for

youth on SSI

Improved quality of transition to adulthood programs in LEAs for

youth in treatment group

Continuous Quality Improvement

Quality Improvement

Continuous Quality Improvement

Quality Planning Tools & Processes• Outreach and Recruitment Guide• PROMISE Recruitment Tracking System• Intervention Guide for Practitioners and Evaluators• Training for using project MIS• Training for outcomes-based payment • Organizational capacity assessment for implementation• Brokering relationships between project partners• Formation of regional teams of stakeholders• NYS PROMISE State Steering Committee and Coordinating

Council

Quality Control Tools & Processes

• Project MIS tool [NYESS]– tracking program participation and services received

• Online training and technical assistance tracking (OTAT) system

• Program fidelity assessment – qualitative and quantitative indicators

• Youth and Parent Voices – survey and qualitative study

• Stakeholder Voices – collaboration survey, bi-annual learning communities, and steering committee meetings

Quality Improvement Tools and Processes• Quarterly automated NYESS reports for RDS, Providers

and Parent Centers• TA translating data into action plans at local, regional and

state-level• Bi-annual comprehensive project reports at Learning

Community identifying (a) what was done well, (b)what worked and (c) what needs to be done

• Sharing data with steering committee at state-level• Engaging stakeholders in concept mapping activity to

synthesize information• Annual progress report

Access to National and State TA

• Access to national and state TA on transition through partnerships with NYS PDSC, NSTTAC training events, and other research and TA entities

• Annual DCDT conference attendance• Webinars on lessons learned from NYS PROMISE

implementation• Reports from the National Evaluator on NYS PROMISE