2010 ECO Conference

How Will I Know?

How Will You Know?

Implementation Fidelity Surveys:

2 States’ Experiences

2010 ECO Conference

How Will I Know?

A Tale of Two StatesOhio 619• 6th Largest State• 612 School Districts• 88 Counties

– Education Service Centers

– County Bds of DD

• 16 State Support Teams-regionalized TA/PD

• World’s largest basket

Minnesota C & 619

• Xx School Districts• 8? Counties• Regionalized prof.

development & TA• Worlds largest ball

of twine

2010 ECO Conference

How Will I Know?

A Tale of Two StatesOhio 619• Has used COSF

since 2007• Locals report to

state 2x per year (fall & spring)

• State does conversion of COSF scores to progress categories

Minnesota C & 619• Started using COSF

on April 1, 2006• Ongoing data

collection locally• Reported to state

annually• Moving to “real

time” collection in future

2010 ECO Conference

How Will I Know?

A Tale of Two States

Ohio 619• Two institutes…• Teacher reference

sheet• Data collection

protocols• Data checks in

the state data system– Funding impact

Minnesota C & 619• Early emphasis on

data quality as precursor to date use.

• Pattern checking tool• COSF data linked to

demographic & service variables plus Part C family outcome data

2010 ECO Conference

How Will I Know?

Ohio Assessment Policies

State statute requires programs to document child progress using research-based indicators prescribed by the Department and report results annually

Data/Reporting Videoconferences in September– Policies provided– Reporting dates calendar

2010 ECO Conference

How Will I Know?

Ohio Assessment Policies

Multiple Sources Required Two Sources are consistent statewide

for funded programs Preschool Special Education

Early Childhood Entitlement GrantGet It! Got It! Go! & Ages/Stages Questionnaire: Social-Emotional (completed with parent)

Early Childhood Outcomes Summary Form– Preschool Special Education: 7 point scale

• Completed with parent and team providing services– 9 point scale with ECO Center for

nondisabled peers

2010 ECO Conference

How Will I Know?

Minnesota Assessment Policies

Multiple sources of information inform COSF ratings for Part C & Preschool Special Education entry/exit ratings

One source must be a tool appropriate for ongoing assessment cross-walked by ECO

State Pre-K program has new statutory mandate to conduct ongoing assessment

Procedural updates provided annually. E-module release planned for mid-August

2010 ECO Conference

How Will I Know?

How Will You Know?

Child Outcome Summary Form?

Decision Tree ?

Publisher’s Online Assessment?

Team Process?

Parent Engagement?Data Entry

Data Entry

Protocol?

Protocol?Vigilant Administrative

Oversight?

VALID DATA?

Summary Statements?Local Reportin

g?

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

Fidelity Survey

• What do we want to know?• Who will we need to ask?• How will we ask them?• How will we analyze the data• How will we turn analysis into action?• How will we know that the action has

addressed the identified issues?

2010 ECO Conference

How Will I Know?

What Did We Want to Know?

• Are all contributors to COSF data and Child Outcome Measurement System implementing process with fidelity?

• If not….who?• If not…which component or

components?• If not…why not?• If not…what will it take?

2010 ECO Conference

How Will I Know?

What Do We Need to Know?

• Minnesota used its previously developed Child Outcome Summary Process: Quality Review Form as the basis for survey questions.

• Outstanding assistance from NECTAC & ECO in formulating questions/responses/branching

• Electronic survey distributed to local providers and administrators of Early Childhood Special Education (C & 619).

2010 ECO Conference

How Will I Know?

What Do We Need to Know?

• Ohio has ongoing contract with ECO Center

• ECO assisted in survey development in partnership with Ohio EC Quality Network

• Survey sent to Preschool SpED Supervisors

• ECO conducted Data Quality and Analysis Institutes

2010 ECO Conference

How Will I Know?

Key Questions: Respondent Demographics

• Professional license(s) held• Length of time in ECSE• ECSE “role” during 2008-09 school year

– Service coordinator– Direct provider of ECSE or related service– Regular Early Childhood Educator– Administrator

• Age Level Served (e.g., 0-2; 3-5; 0-5; etc…)

2010 ECO Conference

How Will I Know?

Key Questions: Respondent COSF Experience

• How many COSF’s did the respondent contribute to in 2008-09?

• Adherence to Procedures– Timely– Completed during team meeting– Parent input– Multiple Sources of Data

2010 ECO Conference

How Will I Know?

MN Fidelity Survey—November 2009

• Zoomerang• Sent from ECO through ECSE List

Serve• 617 valid respondents

– Administrators (39)– ECSE Teachers (392)– Speech/Language (164)– OT/PT/SW (146)– Other (45)

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

Challenges to Face to Face Meeting

• TIME (76%)• Schedules of team members (66%)• Very small team/only 1 provider

(33%)• Don’t value the process (25%)

– Too subjective– Too redundant

• Other (7%)

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

Identified Challenges to Using Decision Tree

• Time• Can’t always find it when needed• “What’s a Decision Tree?”• “I’m a related service

provider and I never received training.”

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

Identified Challenges to using Crosswalks

Consistently• Time• More training is needed• “I’m not sure what a crosswalk is.”• “We don’t always know where to

find the crosswalks.”• Not all tools have been

crosswalked.

2010 ECO Conference

How Will I Know?

Actionable Items

• Reduce duplication of documentation

• Increase use of Decision Tree• Create defensible procedure for

“virtual teaming”• Improve access to “just in time”

professional development

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

COSF’s 7-Point Scale

1 2 3 4 5 6 7

Age Expected (AE) FunctioningAE with Qualitative ConcernImmediate Foundational SkillsFoundational Skills

2010 ECO Conference

How Will I Know?

2010 ECO Conference

How Will I Know?

Articulate!

• E-Module on Child Outcomes Measurement System– Overview– Tabs for specific roles/functions

within COMS– State and local pattern-checking

• Scheduled for release mid-August

2010 ECO Conference

How Will I Know?

ECO Implementation Survey

• Purpose of the Survey: Identify challenging practices for local districts and personnel– Areas with approximately 20% or more of the

respondents reporting somewhat, very or extremely challenging

• Implementation of ECOSF in 2007-2008

• Preparation since August 2007

• Survey May 2010

2010 ECO Conference

How Will I Know?

ODE

• Public Preschool• Preschool Special Education• Even Start• Kindergarten Oversight• Program Licensing• Ready School Initiative• Head Start Collaboration Office

2010 ECO Conference

How Will I Know?

Lessons Learned

• Multiple sources of information– Required sources

• Get It! Got It! Go! (outcome 2)• Ages and Stages Questionnaire: Social-

Emotional (outcomes 1 and 3)

– Record types of sources• Family• Observations• Assessments

– Ongoing Assessment System

2010 ECO Conference

How Will I Know?

Inputs• Family information on

child functioning – Received in team

meeting – Collected separately – Incorporated into

assessment(s)– Not included

• Varied assessments– Curriculum based– Criterion referenced– Standardized– Checklists– Work sampling– Portfolios– Interview

• Varied Observations– Varied social settings

• Individual• Small group• Large group

– Different interactions/dynamics

– Different degrees of choice

– Different levels of competence

– Different times of day• Transitions• Circle time Entry• Exit• Bus• Free choice

– Child preferences

2010 ECO Conference

How Will I Know?

Synthesizing Results from Multiple Sources

2010 ECO Conference

How Will I Know?

Lessons Learned

• Comparable to same-aged peers• Use of Decision Tree• Fidelity of Process• Data Issues

2010 ECO Conference

How Will I Know?

Understanding & Applying Knowledge of Functioning of

Non-disabled Peers

2010 ECO Conference

How Will I Know?

Use of the Decision Tree

2010 ECO Conference

How Will I Know?

Observe Process for Fidelity

2010 ECO Conference

How Will I Know?

Review to Ensure Fidelity

2010 ECO Conference

How Will I Know?

Accuracy Issues

2010 ECO Conference

How Will I Know?

Lessons Learned

• Core team input– Preschool special education

teacher– General education teacher– Parent– Related services personnel

• Involving parent

2010 ECO Conference

How Will I Know?

Reasons for Completion without all Members

Contributing

2010 ECO Conference

How Will I Know?

Involving Parents

2010 ECO Conference

How Will I Know?

Regionally BasedLocal Needs

• Professional Development• Technical Assistance• Assessment Training• Observation and Mentoring• ELLCO Observations and TA• Regional Leadership Meetings

2010 ECO Conference

How Will I Know?

Regionally BasedLocal Needs

• Teacher Leader Project• Preschool Literacy Core

Curriculum (CORE)• SIRI: Pre K/K State Institutes for

Reading Instruction• ELLCO Observations and TA• English Language Learners

2010 ECO Conference

How Will I Know?

PD Connection

IVDL/Higher Education FacultyELLS• Preschool Literacy Core Curriculum (CORE)• SIRI: Pre K/K State Institutes for Reading

Instruction• Teacher Leaders• Ohio Early Learning Content Standards

– Integrated Curriculum– The IEP and the Early Childhood Curriculum– Mathematics– Science– Social Studies

2010 ECO Conference

How Will I Know?

• Ohio Resource Center for Mathematics, Science ad Reading

• Resources for early childhood– Early Childhood Bookshelf– Standards-aligned and Peer Reviewed

Lessons– Early Childhood Building Blocks- Best

Practice Articles

http://www.ohiorc.org/

2010 ECO Conference

How Will I Know?

So What Now?• Webinars

– Data Accuracy and Analysis• Newbies• Advanced

– Summary Form• Source Citation Form

– Newbies– Refresher

• Evidence Summary Form– Newbies

2010 ECO Conference

How Will I Know?

So What Now?• Continue state

research efforts– ECO analysis of

GGG, ASQSE and COSF

– ELLCO– KRA-L– Local readiness

projects

2010 ECO Conference

How Will I Know?

Get It, Got It, Go! (GGG) FY09

• 8,395 ECE children (79%) had pre- and post measures

• 11,196 PSE children (70%) had pre-and post measures

• Children served in ECE and PSE classrooms experienced statistically significant growth from fall to spring

2010 ECO Conference

How Will I Know?

OEL&SR Program Participant Performance on KRA-L

Fall 2007 KRA-L scores

• ECE participants had significantly higher KRA-L scores than all other K students.

• PSE participants who are nondisabled in K have higher KRA-L scores than all other K students who are nondisabled.

• PSE students who are SWD in K are comparable to other K students who are SWD.

2010 ECO Conference

How Will I Know?

More….

• New Web page• Outcomes brochures • Outcomes reference card for families• Outcomes reference sheet for

teachers and related services personnel

2010 ECO Conference

How Will I Know?

Teacher Reference: PROCESS Implementation

2010 ECO Conference

How Will I Know?