iowa’s ssip phase iii year 1 progress report part c · ffy 15 iowa’s ssip phase iii year 1...

FFY 15

Iowa’s SSIP Phase III Year 1

Progress Report – Part C

Submitted April 3, 2017

Iowa Department of Education

[Intentionally left blank.]

i

Iowa Part C State Systemic Improvement Plan Phase III Year 1

Table of Contents

INTRODUCTION

Acronyms .................................................................................................................................................... 1

Structure of Early ACCESS, Iowa's IDEA Part C Early Intervention System ................................................. 2

Stakeholder Involvement ....................................................................................................................................... 2

SECTION A: Summary of Phase III .......................................................................................................................... 5

SECTION B: Progress in Implementing the SSIP .................................................................................................... 8

Table 1 Progress on Evidence-Based Practices Activity 1 of 2 from Logic Model .......................... 9

Table 2 Progress on Evidence-Based Practices Activity 2 of 2 from Logic Model ........................ 13

Table 3 Progress on Implementation Strategies Activity 1 of 4 from Logic Model ...................... 14




Table 7 Progress on High Quality System Activity 1 of 3 from Logic Model ................................. 18



Figure 1 Cohort 4 Training Cycle ................................................................................................... 10

Figure 2 Cohort 4 Example of Expert Coaching Timeline for Quads of Service Providers............ 10

Figure 3 Example of Cohort 4 Expert Coaching Session Cycle Responsibilities of

Quad Members ................................................................................................................. 11

Figure 4 Cohort 4 Roles and Responsibilities of the Service Provider Participants ...................... 11

Figure 5 Community of practice Schedule of Webinars to Support Shift to Evidence-Based

Practices ............................................................................................................................ 12

Figure 6 Dedicated Website for Iowa Distance Mentoring Model for Early ACCESS ................... 12

Figure 7 Core Principles of Adult Learning.................................................................................... 13

Figure 8 List of Implementation Team Recorded Webinars from IA DMM Website ................... 15

Figure 9 Example of Adjusting Written Agreements Based on Feedback

and Changing Needs ......................................................................................................... 16

Figure 10 Iowa Family Support Network Early ACCESS Webpage with Sections Identified for

Different Stakeholders ...................................................................................................... 17

SECTION C: Data on Implementation and Outcomes ......................................................................................... 20

Table 10 Selected Elements from Direct Service Provider Level

Evaluation Plan L2Q5, L2Q5.1 ........................................................................................... 21

Table 11 IA DMM Cohort 4 First Session, Day 1 ........................................................................... 24

Table 12 IA DMM Cohort 4 First Session, Day 2 ........................................................................... 24

Table 13 IA DMM Cohort 4 Second Session, Day 1 ...................................................................... 24

IOWA Part C SSIP Phase III Year 1

ii

Table 14 IA DMM Cohort 4 Second Session, Day 2 ...................................................................... 24


Evaluation Plan L2Q6a-d ................................................................................................... 25


Evaluation Plan L2Q6.1, L2Q6.2 ........................................................................................ 27

Table 17 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.3 ............ 28

Table 18 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q7 ............... 29

Table 19 ROLE Areas, Subareas, and Number of Items ................................................................ 29

Table 20 Individual AEA/LEA Mean Scores for ROLE Culture and Leadership.............................. 31

Table 21 Individual AEA/LEA Mean Scores for ROLE Systems & Structures, Communication,

Teams and Evaluation ....................................................................................................... 32

Table 22 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q8 ............... 33

Table 23 Selected Elements from Family Participant Level Evaluation Plan L1Q1....................... 36

Table 24 Early Intervention Parenting Self-Efficacy Scale for Cohorts 1 through 3, Fall 2015 ..... 37

Table 25 Selected Elements from Family Participant Level Evaluation Plan L1Q2a-b, L1Q3 ....... 38

Table 26 Selected Elements from Family Participant Level Evaluation Plan L1Q4....................... 38

Table 27 Selected Elements from Implementing Agency Level Evaluation Plan L3Q9-Q11 ........ 41

Table 28 Levels of Collaboration Definitions and Relationship Characteristics ........................... 42

Table 29 Selected Elements from Implementing Agency Level

Evaluation Plan L3Q12, L3Q12.1 ....................................................................................... 48

Table 30 Regional Implementation Team Report Questions & Examples of Data Collected ....... 49

Table 31 Selected Elements from Implementing Agency Level

Evaluation Plan L3Q13, L3Q14 .......................................................................................... 50

Table 32 Selected Elements from State Level Evaluation Plan L4Q15, L4Q16 ............................. 51

Table 33 Selected Elements from State Level Evaluation Plan L4Q17, L4Q17.1 .......................... 52

Table 34 Selected Elements from State Level Evaluation Plan L4Q18a-b .................................... 53

Table 35 Selected Elements from State Level Evaluation Plan L4Q19 ......................................... 55



Figure 11 Learning Based Assessment of Training—IA DMM Cohort 4 First Session Day 1 ........ 22

Figure 12 Learning Based Assessment of Training—IA DMM Cohort 4 First Session Day 2 ........ 22

Figure 13 IA DMM Cohort 4 Second Session

(6 months after 1st Face-to-Face Training), Day 1 ............................................................ 23

Figure 14 IA DMM Cohort 4 Second Session

(6 months after 1st Face-to-Face Training), Day 2 ............................................................ 23

Figure 15 Cohort 1-4 Observation Results, FGRBI SS-OO-PP-RR Key Indicators, Coaching

Strategies, Routines Used ....................................................................................................... 26

Figure 16 Number of Participants by Cohort and Percentage of Disciplines

Trained Cohort 1-4 ............................................................................................................ 27

Figure 17 Examples of Interview Quotes from Cohorts 1 and 2 ................................................... 34

Figure 18 Examples of Interview Quotes from Cohorts 1 and 2 ................................................... 35


iii

Figure 19 Caregiver Key Indicators Form ...................................................................................... 40

Figure 20 Caregiver Key Indicators Observations Cohort 4 Video 1 and Video 2 ........................ 40

Figure 21 AEA "X" Network Maps for 2014 and 2015 .................................................................. 43

Figure 22 AEA "X" 2014 and 2015 Ties Valued as Coalition or Higher ......................................... 43

Figure 23 AEA "Y" Network Maps for 2014 and 2015 .................................................................. 44

Figure 24 AEA "Y" 2014 and 2015 Ties Valued as Coalition or Higher ......................................... 44

Figure 25 AEA "Z" Network Maps for 2014 and 2015 .................................................................. 45

Figure 26 AEA "Z" 2014 and 2015 Ties Valued as Coalition or Higher.......................................... 45

Figure 27 All AEAs Network Maps, 2014 ...................................................................................... 47

Figure 28 All AEAs Network Maps, 2015 ...................................................................................... 47

Figure 29 ECTA System Framework Self-Assessment Governance Results .................................. 54

Figure 30 ECTA System Framework Self-Assessment Personnel/Workforce Results .................. 54

SECTION D: Data Quality Issues ........................................................................................................................... 57

SECTION E: Progress Toward Achieving Intended Improvements ..................................................................... 58

Table 38 Evidence-Based Practices Short, Intermediate, Long-term Outcomes.......................... 58

Table 39 Implementation Strategies Short, Intermediate, Long-term Outcomes ....................... 61

Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes

Governance ....................................................................................................................... 65

Table 41 High Quality Infrastructure Short, Intermediate, Long-term Outcomes

Personnel/Workforce ....................................................................................................... 66

Table 42 Licensed Area Education Agency and District Staff that Could Retire in Next Five Years

(Based on Rule of 88 or Retirement at Age 65. ................................................................ 67

Table 43 High Quality Infrastructure Short, Intermediate, Long-term Outcomes Accountability &

Quality, Quality Standards, Data System, Finance ........................................................... 68

SECTION F: Plans for Next Year ............................................................................................................................ 68

References ............................................................................................................................................................ 69

Appendix A

Logic Models: Instructional Practices, Implementation Strategies, High Quality Part C System ............. 70

Appendix B

System Level Evaluation Plan for Early ACCESS ........................................................................................ 73

1

Acronyms

AEA ..................... Area Education Agency

AI ........................ Active Implementation

CPT ..................... Core Planning Team for personnel/workforce development (interagency team of consultants for birth to age 5 services)

DaSy ................... Center for IDEA Early Childhood Data Systems

EA ....................... Early ACCESS

EAGL ................... Early ACCESS Leadership Group (Liaisons from all Area Education Agencies, Departments of Education, Public Health, and Human Services, Child Health Specialty Clinics, Special Education Directors, Iowa School for the Deaf, Iowa Educational Services for the Blind and Visually Impaired, Autism)

EC CSPD .............. Early Childhood Comprehensive System for Personnel Development

ECSE .................... Early Childhood Special Education

ECTA ................... Early Childhood Technical Assistance Center

FGRBI .................. Family-Guided Routines-Based Intervention

FSU ..................... Florida State University

GV ....................... Governance (component of the ECTA System Framework)

ICEA .................... Iowa Council for Early ACCESS (Iowa's interagency coordinating council)

IA DMM .............. Iowa Distance Mentoring Model of Professional Development for Early ACCESS

MOA ................... Memorandum of Agreement between Departments of Education, Public Health, and Human Services, and Child Health Specialty Clinics, University of Iowa

OT ....................... Occupational Therapist

PD ....................... Professional Development

PDSA ................... Plan Do Study Act

PN ....................... Personnel/Workforce (component of the ECTA System Framework)

PSP ...................... Primary Service Provider

PT........................ Physical Therapist

RIT ...................... Regional Implementation Teams (at each AEA and Des Moines Public Schools)

ROLE ................... Readiness of Organizational Learning and Evaluation

SC ........................ Service Coordinator

SiMR ................... State-identified Measurable Result

SIT ....................... State Implementation Team (early childhood interagency team)

SLP ...................... Speech Language Pathologist

SS-OO-PP-RR ...... Setting the Stage, Observation and Opportunities to practice, Problem solving and Planning, Reflection and Review

SSIP ..................... State Systemic Improvement Plan

SWT .................... State Work Team (Departments of Education, Public Health, and Human Services, and Child Health Specialty Clinics)


2

Structure of Early ACCESS, Iowa's IDEA Part C Early Intervention System

Iowa Administrative Rules for Early ACCESS establish that four signatory agencies commit to implementation

and evaluation of the IDEA Part C early intervention system. “Signatory agency” means the:

Iowa Department of Education (lead agency),

Iowa Department of Public Health,

Iowa Department of Human Services, and

Child Health Specialty Clinics of the University of Iowa.

There are 9 regional grantees, called Area Education Agencies (AEA), which guarantee services are delivered to

all parts of the state.

STAKEHOLDER INVOLVEMENT IN EARLY ACCESS Who's Who in Early ACCESS: Stakeholder Groups, SSIP Phase I, II and III The following groups routinely engage in Early ACCESS activities. Long-standing relationships between and within these groups allows for smooth transitions when membership changes due to retirements, new hires, changing roles or jobs, or expiring terms for membership. Each group has a role in supporting the successful implementation of Early ACCESS in Iowa. This includes engaging with implementation strategies, continuous improvement and evaluation. In addition to permanent long-standing groups, task teams form for the purpose of dealing with specific, time-bound activities. During the current reporting period, 13 different task teams (e.g. service coordinator competency training revision, eligibility criteria, Every Student Succeeds Act, Autism Navigator Implementation Plan) formed and included diverse groups of stakeholders that volunteered to work on different aspects of the early intervention system. There is always an electronic format for joining so that anyone from across Iowa can participate. It is not unusual that a service provider "attend" a task team meeting from a car between home visits or that a family member be at home with the sounds of children and pets in the background.


3

There is a prevailing belief that having the voice of people who are impacted most by the work must be included in helping to define problems, develop and carry out solutions, and define success. This belief translate into action through the task team and stakeholder group work. The success of the Early ACCESS system depends on the knowledge and skills the following permanent, stakeholder groups: Early ACCESS State Work Team (SWT) (6 members)

Staff from Iowa Department of Education, Iowa Department of Public Health, Department of Human Services,

and Child Health Specialty Clinic.

Two full-day meetings per month; two-day summer retreat; contact between meetings to get work done. This

is the core team that is responsible for the day-to-day operations of Early ACCESS and supporting the

work of the SSIP.

Early ACCESS Leadership Group (EAGL) (30-34 members)

Area Education Agencies (liaisons/providers/service coordinators, special education director as liaison to all

directors, mid-level management)

Des Moines Public Schools Iowa School for the Deaf

Department of Education Department of Public Health

Department of Human Services Child Health Specialty Clinics

Autism Consultant Iowa Educational Services for the Blind and Visually Impaired

Iowa Council for Early ACCESS (ICEA) (18-25 members)

Parent Leadership and Members

Multi-Agency Stakeholders

Department of Education Department of Public Health

Department of Human Services Child Health Specialty Clinics

Signatory Agency Leadership Team (4 members)

Administrators from Iowa Department of Education, Iowa Department of Public Health, Department of Human

Services, and Child Health Specialty Clinic who are bound by a written Memorandum of Agreement (MOA) to

ensure the Early ACCESS system functions successfully. MOA is executed every 5 years.

Regional Implementation Teams (RIT) (10 teams, 5-13 members per team, 80 individuals statewide)

Personnel Preparation/Higher Education

AEA Administrators, Supervisors, Coordinators, Liaisons

Service Providers (SLP, ECSE Teachers, OT, PT, Deaf/Hard of Hearing Teachers, Social Workers, Autism

Consultant)

Media Specialists

Service Coordinators

Out-of-State PD Trainer


4

State Implementation Team (SIT) (10-15 members)

State Work Team members (Staff from Iowa Department of Education, Iowa Department of Public Health,

Department of Human Services, and Child Health Specialty Clinic)

Area Education Agencies (Special Education Directors, EA Liaisons)

Early Childhood Iowa (Early Head Start, Head Start; Child Care; Maternal Infant Early Childhood Home

Visitation; Iowa Association for the Education of Young Children)

Child Care Resource and Referral

Personnel Preparation/Higher Education

Iowa Distance Mentoring Model for Early ACCESS Project Administration Team (5-7 members)

Florida State University, College of Communication and Information, School of Communication Science and

Disorders

Iowa Department of Education

Evaluation Team (5-7 members)

Department of Education

Florida State University IA DMM project staff

External evaluator

IFSP/IEP Data System Core Team (3-5 members)


Area Education Agency

IFSP Data System Team (3-5 members)


Department of Public Health

Private contractors

Early Childhood Comprehensive System for Personnel Development (EC CSPD) (6-8 members)

Department of Education, Early ACCESS (IDEA Part C)

Department of Education, ECSE, (IDEA Part B 619)

Department of Human Services

Iowa Association for the Education of Young Children

National Center for Systemic Improvement Cross State Learning Collaborative (Family Outcomes)

Multiple, various state and US Territory membership

Division of Early Childhood, Early Intervention Special Interest Group

Multiple, various state, organization and individual membership

Infant Toddler Coordinator Association

IDEA Part C coordinators from states and US Territories


5

SECTION A: Summary of Phase III

Theory of Action, including the State-identified Measurable Result (SiMR)

The Early ACCESS Theory of Action was originally developed in 2012 and updated in 2014 when submitted to OSEP. Iowa is focusing on increasing the percentage of families that report early intervention has helped them help their child develop and learn (the Part C SiMR) and corresponding information in the Theory of Action is highlighted by the light blue box.

Improvement Strategies, Logic Models including the SiMR

Iowa's Theory of Action has three action strands: practice, professional development, and infrastructure that lead to three coherent improvement strategies.

Improvement Strategy 1: New Instructional Practices

Shifting instructional practice away from teaching the child to using family-guided routines-based interventions(FGRBI) where the focus is on coaching the caregiver to support the child.


6

Improvement Strategy 2: New Implementation Strategies

Incorporate implementation science frameworks in order to develop the capacity to make effective, statewide, and sustained use of evidence‐based practices.

Improvement Strategy 3: New High Quality System

Use the ECTA System Framework self-assessment processes in order to develop a high-quality Early ACCESS system that encourages, supports, and requires implementation of evidence-based practices.

A logic model was created for each of the 3 improvement strategies. The logic models (see Appendix A) were

used to help build the Systems Level Evaluation Plan for Early ACCESS (see Appendix B).

Specific Evidence-Based Practices Implemented To Date

Family-Guided Routines-Based Interventions (FGRBI), caregiver coaching and Active Implementation (AI)

Frameworks to sustain evidence-based practices are being implemented as part of the Iowa Distance

Mentoring Model (IA DMM) of Professional Development for Early ACCESS. Through coaching caregivers in

family-guided routines-based interventions, there will be an increase in the competence and confidence of

caregivers which will lead to an increase in the percentage of families reporting that early intervention helped

them to help their children develop and learn (Iowa's SiMR). Using AI Frameworks will ensure implementing

with high fidelity, scaling up and sustainability. As of the current reporting period, 4 cohorts of early

intervention service providers have been trained in FGRBI and caregiver coaching.


7

Brief Overview of the Year’s Evaluation Activities, Measures, and Outcomes

The majority of evaluation measures were completed as scheduled. Family level pre-interview surveys and

interviews did not occur as scheduled. Solicitation for participants did take place during the current reporting

period and took longer than expected. Regional and state level measures changed from Active

Implementation (AI) Hub quizzes to more contextual measures and data was collected. The Levels of

Collaboration Scale for state level evaluation was dropped. This is a very time intensive and expensive survey

to analyze. Results from the scale are used to support 3 short-term outcomes that have other, better

measures already being used to show progress. Evidence from multiple evaluation measures shows progress

is being made on short-term outcomes.

Highlights of Changes to Implementation and Improvement Strategies

Improvement strategies remain the same as planned and submitted in the SSIP Phase II. Implementation of IA

DMM involves a continuous improvement process driven by stakeholder input and data. Shifts in the makeup

of cohort groups happen based on feedback from the proceeding cohort.

Original plans included training pairs of service providers from each agency per cohort. Stakeholders asked for

more providers to be included per cohort. Training was redesigned to accommodate groups of 4 providers or

quads for cohort 4. In June 2016 at a Joint Implementation Team Meeting (regional and state teams), in

preparation for cohort 5, data related to cohort 4 structure changes was reviewed. Based on the review,

cohort 5 structure would be changed to groups of 3 service providers or trios. In addition, cohort 5 is to

include a veteran IA DMM trained provider to begin training as an internal coach to build agency's capacity to

have expert coaches that will focus on fidelity.

One of the biggest changes was giving up, not just the toy bag, but my ability to bring the latest most greatest interesting and exciting toys into the home to entice children to get involved and to give that up and realize that the toys I need are in the home and that those are the things that will be there every day for children to use to learn. So that's been a challenge but it's been a positive change. Also through that process learning that my role really is to empower parents to teach their children every day during their routines and activities. What I do in the few minutes I am there really doesn't make as much difference as what the parents are doing everyday with their child.


8

SECTION B: Progress in Implementing the SSIP

The focus of Section B is on progress in implementing activities in each of the 3 improvement strategies

chosen to improve outcomes for children and families served in Early ACCESS:

Improvement Strategy 1: New Instructional Practices

Shifting instructional practice away from teaching the child to using family-guided routines-based

interventions (FGRBI) where the focus is on coaching the caregiver to support the child.

Improvement Strategy 2: New Implementation Strategies

Incorporate implementation science frameworks in order to develop the capacity to make effective,

statewide, and sustained use of evidence‐based practices.

Improvement Strategy 3: New High Quality System

Use the ECTA System Framework self-assessment processes in order to develop a high-quality Early

ACCESS system that encourages, supports, and requires implementation of evidence-based practices.

Tables are used to share progress in each of the activities identified in the logic models (Appendix A) for Early

ACCESS (submitted in Phase II) and include:

activities taken directly from the logic model,

evidence or outputs of the activity,

if implemented as intended including description of changes in an activity based on data collected, and

successes and challenges to implementation.

NOTE: Data generated from evidence listed for an activity are shared in Section C.


9

Progress on Instructional Practices Table 1. Progress on Evidence-Based Practices Activity 1 of 2 from Logic Model

Act

ivit

y

Design and implement professional development plan to build and expand capacity of providers in coaching caregivers and family-guided routines-based intervention (FGRBI)

Evid

en

ce

Timelines and training cycles (link?)

Training attendance.

Pre-then-Post retrospective surveys collected at

end of each face-to-face training

Journey reflection documents

IA DMM website visits and materials used

# of Weekly emails read and used consistently

# of online modules

Facebook usage

# videos submitted

External coaching sessions completed

# of self assessments completed

# of webinars and attendance

Online Community of Practice usage

Survey results for importance of training

features

Participant feedback video recordings

Observational data on routines used, SSOOPPRR

home visit components, key indicators to FGRBI

including coaching strategies.

Survey results for each cohort on next steps,

what to keep and what to adjust.

Imp

lem

en

ted

as

inte

nd

ed?

[ye

s/n

o]

Tim

elin

e

Yes; As of 6/30/16, completed 4 cohorts and prepared for cohort 5 as scheduled. However, structures of the cohorts changed over time. Cohorts 1-3 (trained in FFY 2013-2014) had pairs of providers from each region. This changed to having quads or 4 providers or Quads from each region. Changes were made based on feedback from stakeholders asking for more people to be trained per cohort. After that change occurred, cohort 4 Quad data was reviewed to inform cohort 5 plans and included: (1) participation and completion rates of Quads; (2) journey reflections; (3) Quad survey results; (4) frequency counts of participant contacts; (5) Google analytics; (6) FSU feedback and (7) input from regional and state implementation teams at Joint Implementation Team Meeting. Based on all these pieces of information, the makeup of the training groups changed to trios. In addition, cohort 5 is to included a veteran IA DMM trained provider to begin training as an internal coach to build agency's capacity to have expert coaches that will focus on fidelity. Implementation of this change takes place during the next reporting period and will be address in SSIP Phase III Year 2 report.

Succ

ess

es

&

Ch

alle

nge

s

Contracting with experts in FGRBI and caregiver coaching who are both flexible and organized.

Well planned timelines and training cycles.

Written roles and responsibilities.

Multiple learning opportunities designed to support change (e.g. face-to-face training, webinars, distance expert coaching sessions, community of practice, online modules).

Using core principles of adult learning in the training while teaching integration of adult learning principles in family centered practices.

Having a core team of people responsible for the overall plan is both a success and challenge so that the work is bigger than any one person should they leave.

Making professional development part of service providers' everyday work is a challenge that is strong even into year 4 of the work.

Less technology challenges over time.


10

Figure 1-7 are illustrations of the successful elements of the Iowa Distance Mentoring Model of Professional Development for Early ACCESS. Figure 1. Cohort 4 Training Cycle Timeline

Figure 2. Cohort 4 Example of Expert Coaching Timeline for Quads of Service Providers

IA DMM Cohort 4 Training Cycle Timeline


11

Figure 3. Example of Cohort 4 Expert Coaching Session Cycle Responsibilities of Quad Members

Figure 4. Cohort 4 Roles and Responsibilities of the Service Provider Participants

IA DMM Cohort 4 Session Cycle –Post Follow Up Workshop Example

Jane – March 8 Suzi – April 12

Carol – May 10 Amy – June 7


12

Figure 5. Community of Practice Schedule of Webinars to Support Shift to Evidence-Based Practices Figure 6. Dedicated Website for Iowa Distance Mentoring Model for Early ACCESS (http://dmm.cci.fsu.edu/IADMM/index.html)

http://dmm.cci.fsu.edu/IADMM/index.html


13

Figure 7. Core Principles of Adult Learning

Table 2. Progress on Evidence-Based Practices Activity 2 of 2 from Logic Model

Act

ivit

y

Develop and implement plan for tracking other related professional development (PD) that occurs within individual Area Education Agencies and signatory agencies that supports coaching caregivers and FGRBI

Evid

en

ce List of PD funded and offered through regional

Area Education Agencies

Notes from statewide stakeholder meetings (EAGL)

Lists of data collection tools and methods for measuring FGRBI and coaching caregivers.

Imp

lem

ente

d a

s

inte

nd

ed

? [y

es/n

o]

Tim

elin

e

Yes. As of 6/30/16 have met regularly with the Early ACCESS Leadership Group where information is gathered on the following: writing functional, participatory child and family outcomes; routines-based interviewing; Primary Service Provider (PSP) teaming practices; and, data collection tools and methods used at the agency level to track progress in FGRBI and coaching caregivers.

Succ

ess

es

&

Ch

alle

nge

s

Using already existing stakeholder groups and meetings.

Using Google Docs to collect and share information.

Discussing in large groups supports cross-agency learning.

Nuances from different experts in coaching strategies can confuse some service providers.

Each agency is independently governed and each is at different stages of practice shift which is

challenging.


14

Progress on Implementation Strategies

Table 3. Progress on Implementation Strategies Activity 1 of 4 from Logic Model

Act

ivit

y

Establish collaborative planning & implementation teams & networks

Evid

en

ce Regional Implementation Teams (RIT) rosters

RIT reports and updates

State Implementation Team (SIT) rosters

SIT meeting agendas and notes

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, teams created as scheduled.

Succ

ess

es

&

Ch

alle

nge

s

In the beginning (2013) teams were established quickly in order to meet timelines.

Now, membership based on roles, responsibilities and needs.

Some regional teams include early childhood partners outside of the Area Education Agency system.

SIT serves more as an advisory and high-level problem-solving and reflection team. The State Work

Team functions as what Active Implementation Frameworks describe as a state implementation team,

meeting often and doing the daily work of supporting the system change.

State Implementation Team meeting, small group discussion


15

Table 4. Progress on Implementation Strategies Activity 2 of 4 from Logic Model A

ctiv

ity

Design and implement plan to build capacity of state and regional teams in Active Implementation Frameworks

Evid

en

ce

Evidence-Based Practices and Professional Development (PD) Implementation Plan

Written PD vision

Written agreements between administrators, providers, and training organization

Meeting agenda items, notes, and completed modules related to active implementation frameworks by State Work Team, EA Leadership Group, Iowa Council for EA, SIT, RIT

Recorded webinars

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, implemented as planned and is ongoing.

Succ

ess

es

&

Ch

alle

nge

s

Webinars specifically for implementation teams.

Written action plans for regional teams.

Use Google Docs to routinely update progress on RIT activities.

Face-to-face EAGL meetings used as RIT events where team members across the state who are not part of EAGL join electronically (via Zoom) to participate while other EAGL members who typically are uninvolved with RIT work participate as well.

Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed.

State Work Team functioning as AI Hub definition of state implementation team.

Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.

Figure 8. List of Implementation Team Recorded Webinars from IA DMM Website


16


ctiv

ity

Assess readiness in the Area Education Agencies

Needs assessment

Assess current EI policies, practices, etc.

Assess and acquire resources needed to do the work

Evid

en

ce

Needs assessment results from practices survey

Needs assessment results from technology

survey

Administrator agreements include commitment

of resources to do the work

Notes from review of current procedures,

policies, practices

Feedback surveys showing PD is meeting the

needs of the participants

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, completed as scheduled.

Succ

ess

es

&

Ch

alle

nge

s

Adjusted agreements based on feedback from previous year and needs of the upcoming year (see

Figure 9).

June Joint Implementation Team meetings, share data and what to keep and what to change for next

cohort. Each cohort contributing to the continuous improvement of the PD

Getting timely consents from families and providers has always been a challenge. Providers get the

needed consents after several reminders from Florida State University.

Figure 9. Example of Adjusting Written Agreements Based on Feedback and Changing Needs


17


ctiv

ity

Maintain communication with stakeholders, PD participants, EA leadership team, and council

Evid

en

ce

Written communication plans for internal and

external communications

List of individualized one-day trainings at each

AEA including FGRBI 101 complete by Florida

State University

Written documents explaining FGRBI for various

stakeholders

Video recordings posted on public website

Lists of presentations and PowerPoints used

Task team rosters of volunteers helping with

various aspects of the work

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, implemented as planned and is ongoing.

Succ

ess

es

&

Ch

alle

nge

s

History of long-standing, regular meetings of statewide stakeholders.

Use of Iowa Family Support Network website for public access to information

(http://www.iafamilysupportnetwork.org/) (see Figure 10).

Good participation in task teams when work needs to be done.

Unsure of level of communication and engagement with families so a task team has been created to

select a statewide self-assessment and develop plan around engaging families in all levels of the early

intervention system (next reporting year activity).

Figure 10. Iowa Family Support Network Early ACCESS Webpage with Sections Identified for Different Stakeholders

http://www.iafamilysupportnetwork.org/


18

Progress on High Quality System Table 7. Progress on High Quality System Activity 1 of 3 from Logic Model

Act

ivit

y

Complete ECTA self-assessment for Governance (GV) component; engage stakeholders to develop and implement GV improvement plan

Evid

en

ce

Completed self-assessment with priorities identified and work plans that address priorities

Updated self-assessment document

Early Childhood Comprehensive System of Personnel Development plan (EC CSPD)

Memorandum of Agreement (MOA) Action Plan between Iowa Department of Education, Iowa Department of Public Health, Department of Human Services and Child Health Specialty Clinics of the University of Iowa

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, implemented as planned.

Succ

ess

es

&

Ch

alle

nge

s

Self-assessment reviewed at Early ACCESS state work team meetings and updated annually using the Early Childhood Technical Assistance (ECTA) Center resources.

Multi-state agency shared responsibility to complete MOA Action Plan.

Revisions to Endorsement 100 (Birth-3rd Grade general & special education) approved.

Challenge to balance infrastructure development and shift to evidence-based practices which must happen simultaneously.

Table 8. Progress on High Quality System Activity 2 of 3 from Logic Model

Act

ivit

y

Complete ECTA self-assessment for Personnel/Workforce (PN) component; engage stakeholders to develop and implement PN improvement plan

Evid

en

ce Completed self-assessment with priorities

identified and work plans that address priorities

Updated self-assessment document Early Childhood Comprehensive System of

Personnel Development plan (EC CSPD)

Early Childhood Comprehensive System of Personnel Development team roster (includes IDEA Part C, Part B 619, public and private early childhood agencies)

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

Yes, implemented as scheduled and ongoing.

Succ

ess

es

&

Ch

alle

nge

s

EC CSPD is for statewide cross-sector early childhood system not just Early ACCESS (IDEA Part C) and Early Childhood Special Education (IDEA Part B 619).

Self-assessment reviewed specifically for items directly related to Early ACCESS (IDEA Part C) during state work team meetings and updated as needed using the Early Childhood Technical Assistance (ECTA) Center resources.

Small groups used to address different parts of the EC CSPD with cross-agency core team responsible for overall plan.

Challenge to balance infrastructure development and shift to evidence-based practices which must happen simultaneously.


19

Table 9. Progress on High Quality System Activity 3 of 3 from Logic Model A

ctiv

ity

Review and prioritize remaining ECTA System Framework component for future completion:

Accountability & Quality (AC)

Quality Standards (QS)

Data System (DS)

Finance (FN)

Completed self-assessment on data system (DS) component

Evid

en

ce

Imp

lem

ente

d

as in

ten

ded

? [y

es/n

o]

Tim

elin

e

N/A not scheduled to occur during this reporting period.

Succ

ess

es

&

Ch

alle

nge

s

We are trying to focus on routines in the visits and how to use these skills in all the daily routines that they are doing and different places that they go.


20

SECTION C: Data on Implementation and Outcomes

This section will focus on data from implemented activities and connect them to the short, intermediate and

long-term outcomes. First is information explaining the relationship of the Theory of Action, improvement

strategies, logic models and the evaluation plan. It is important to understand that the evaluation questions,

measures and methods of data collection are traced back to the outcomes of the improvement activities. The

remainder of this section is organized around the evaluation questions and measures from the Systems Level

Evaluation Plan for Early ACCESS (submitted in Phase II).

Based on the following evaluation of improvement practices, improvement strategies and infrastructure

development, there are no modification in the outcomes, SiMR or SSIP overall.

Progress on Systems Level Evaluation Plan for Early ACCESS

Connecting the Dots: Theory of Action, Improvement Strategies, Logic Models and Evaluation

The Theory of Action identifies 3 parts of the early intervention system (practice, professional development

and infrastructure) where actions are required that will move the Early ACCESS system to its desired future.

From each of these parts or "strands of action" Iowa identified 3 improvement strategies to use in order to

ensure eligible children and families have improved outcomes. A logic model was created for each

improvement strategy and provides a visual framework for describing the relationship between resources or

inputs, activities, and results or outcomes. Each short, intermediate and long-term outcome on the logic

models is measured using the tools and methods described in the evaluation plan. There are direct

connections between the Theory of Action, improvement strategies, logic models and the items on the

evaluation plan.

How Are We Doing on Evaluating Activities to Achieve Outcomes?

Iowa's improvement activities are designed to impact multiple levels of the Early ACCESS system: Direct

Service Provider Level ; Family Level ; Implementing Agency Level , and State Level . In order to gauge if

activities are having the desired impacts, the evaluation plan is designed to measure all four of these levels.


21

Tables are color coded by level and include:

evaluation questions,

measures,

data collection methods include who is responsible for the task,

if implemented as intended, and

success and challenges to the evaluation work.

Following each table, when appropriate, is a description of baseline data, sampling procedures, planned data

comparisons, data management/analysis procedures for assessing progress toward achieving intended

improvements, and results of data collection.

D i r e c t S e r v i c e P r o v i d e r L e v e l o f E v a l u a t i o n

Early intervention service providers are the direct recipients of the professional development (PD) activities

and it is important to evaluate the training effectiveness. The following information demonstrates if the

training is meeting the needs of the providers, what knowledge and skills it has taught, and what change it has

brought in the provider's performance (see Tables 10 through 22).

Table 10. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q5, L2Q5.1

Evaluation Question How do service providers' perceptions of their abilities to implement FGRBI change? (L2Q5)

How do service providers' satisfaction and confidence in service provision change? (L2Q5.1)

Measure(s) Post-then-Pre Evaluation Survey

Data Collection Methods Paper survey completed by each participant at the end each face-to-face PD session Administered by Florida State University

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S2, S10

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes

Successes & Challenges: Simple process. Time allowed complete the form at the end of training each day.

Using a "post-then-pre" or "retrospective pre test" after each face-to-face training allows data to be gathered

one time each day (as compared to a "pre test" done at the beginning of the day and a "post test" done at the

end of the day) which gives a valid picture of the trainings effects (see Figure 11-14). Analysis of results offers

trainers information about provider needs and improves each future training element (i.e. webinars, emails,

online resources, expert coaching sessions, community of practice). The goal is increase net learning which

will reduce the learning lag. The first face-to-face two-day training session occurs shortly after participants are

selected and pre-training videos and forms are complete. The second face-to-face two-day training session

occurs around 6 months later.


22

RESULTS

Definitions: Pre-Existing Learning Pre-test Score Net Learning Post-test score minus pre-test score. This is the score attributable to the training. Learning Lag Total minus post-test score

Figure 11. Learning Based Assessment of Training – IA DMM Cohort 4 First Session, Day 1

IA DMM First Face-to-Face Training Cohort 4 - Day 1

Figure 12. Learning Based Assessment of Training – IA DMM Cohort 4 First Session, Day 2

IA DMM First Face-to-Face Training Cohort 4 - Day 2

I can describe rationale for useof caregiver-implementedintervention in everyday

routines and activities

I can identify strategies forbuilding family capacity using

adult learning

I can describe strategies forembedding functional child

and family outcomes ineveryday routines and

activities using evidence basedinterventions

I can describe general andspecific caregiver strategies

Learning Lag 17.40% 20.00% 18.00% 19.60%

Net Learning 16.40% 20.80% 17.40% 16.20%

Pre-Existing Learning 66.20% 59.20% 64.60% 64.20%

66.20%59.20% 64.60% 64.20%

16.40%20.80%

17.40% 16.20%

17.40% 20.00% 18.00% 19.60%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I can list SS-OO-PP-RRcomponents

I can practice use of aframework for coaching

caregivers that promotessuccess for the triad

I can describe generaland specific caregiver

coaching strategies

I can define qualityindicators that support

implementation of familycentered services in the

natural environment

I can develop an actionplan for next steps with

peer partner

Learning Lag 37.10% 0.80% 17.20% 18.00% 16.00%

Net Learning 17.10% 45.80% 21.40% 20.80% 23.20%

Pre-Existing Learning 45.80% 53.40% 61.40% 61.20% 60.80%

45.80%53.40%

61.40% 61.20% 60.80%

17.10%

45.80%21.40% 20.80% 23.20%

37.10%

0.80%

17.20% 18.00% 16.00%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

This supports comments obtained in provider videos stating this training has offered them a framework for their work.


23

Figure 13. IA DMM Cohort 4 Second Session (6 months after 1st Face-to-Face Training), Day 1

IA DMM Second Face-to-Face Training Cohort 4 - Day 1

Figure 14. IA DMM Cohort 4 Second Session (6 months after 1st Face-to-Face Training), Day 2

IA DMM Second Face-to-Face Training Cohort 4 - Day 2

I can describe 3changes in my

knowledge aboutFGRBI

I can describe 3changes in my

knowledge of thecoaching used in IA-

DMM

I can list four stepsfor the problem-

solving process touse with families

I can describe 5 SS-OO-PP-RR strategies

I can describestrateges for

coaching in earlycare and education

Learning Lag 16.80% 13.20% 14.40% 15.20% 27.20%

Net Learning 16.80% 20.00% 34.80% 23.20% 13.60%


66.40% 66.80%

50.80%61.60% 59.20%

16.80% 20.00%

34.80%23.20%

13.60%

16.80% 13.20% 14.40% 15.20%27.20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I can describelearning

opportunities foradult learners.

I can list two newways to use video to

support learning.

I can describegeneral and specificcaregiver coaching

strategies

I can define qualityindicators that

supportimplementation of

family centeredservices in the

natural environment

I can develop anaction plan for next

steps with peerpartner

Learning Lag 14.40% 11.60% 17.60% 16.40% 12.00%

Net Learning 24.40% 26.00% 22.80% 19.20% 21.20%


61.20% 62.40% 59.60% 64.40% 66.80%

24.40% 26.00%22.80%

19.20%21.20%

14.40% 11.60%17.60% 16.40% 12.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

strategies


24

Tables 11-14 are additional survey data collected after each face-to-face training, used for continuous

improvement purposes.

Table 11. IA DMM Cohort 4 First Session, Day 1

Question Mean/

Average

Very Much (5 pts)

Much (4 pts)

A Fair Amount (3 pts)

A Little (2 pts)

Not at All (1 pt)

1. The facilitation was of high quality. (n=51) 4.21 18 26 7 0 0

2. The information was relevant. (n=52) 4.63 36 16 0 0 0

3. The conversations held were critical for my future work. (n=51)

4.49 29 18 4 0 0

4. Overall, this training was very beneficial. (n=52) 4.27 24 20 7 0 1

Table 12. IA DMM Cohort 4 First Session, Day 2

Question Mean/

Average

Very Much (5 pts)

Much (4 pts)


A Little (2 pts)

Not at All

(1 pt)

1. The facilitation was of high quality. (n=49) 4.33 19 27 3 0 0

2. The information was relevant. (n=51) 4.71 38 11 2 0 0

3. The conversations held were critical for my future work. (n=49)

4.57 33 12 3 1 0

4. Overall, this training was very beneficial. (n=48) 4.50 29 15 3 1 0

Table 13. IA DMM Cohort 4 Second Session, Day 1

Question Mean/

Average

Very Much (5 pts)

Much (4 pts)


A Little (2 pts)

Not at All (1 pt)

1. The facilitation was of high quality. 4.56 30 18 2 0 0

2. The information was relevant. 4.70 35 15 0 0 0

3. The conversations held were critical for my future work.

4.56 34 10 6 0 0

4. Overall, this training was very beneficial. 4.52 31 15 3 1 0

Table 14. IA DMM Cohort 4 Second Session, Day 2

Question Mean/

Average

Very Much (5 pts)

Much (4 pts)


A Little (2 pts)

Not at All (1 pt)

1. The facilitation was of high quality. 4.48 26 21 2 0 0

2. The information was relevant. 4.71 37 10 2 0 0

3. The conversations held were critical for my future work.

4.57 31 16 1 1 0

4. Overall, this training was very beneficial. 4.57 32 14 2 1 0

Three open-ended questions wrap up the evaluations for the face-to-face trainings. Below is each question

and examples of top categories that emerged from responses given by providers.

Best Feature of the IA DMM Professional Development Session was….

Watching example videos and talking through the videos

Discussion and reflection with quads/team members/large group

Learning SS-OO-PP-RR

Adult learning strategies


25

Any Suggestions for Improvement?

More on outcomes

Room arrangement

More time to brainstorm, discuss problems

Easier uploading/technology options

Other Comments and Reactions I Wish to Offer

Location was convenient

Sidebar conversations were distracting

Showing video examples of providers at the beginning stages of coaching in FGRBI to compliment the videos of

seasoned providers

Table 15. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6a-d

Evaluation Question How do service providers shift in their abilities to implement family guided routines based interventions? (L2Q6a-d)

Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts.

Measure(s) Family Routine Categories; Coaching Strategies; FGRBI Key Indicators; SS-OO-PP-RR Framework;

Data Collection Methods IA DMM Session Summary Form Administered by Florida State University

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S10, I1, L1


Successes & Challenges: Video recordings from "door to door" (from the beginning of the early intervention visit through the end) is an improvement to having providers sending selected parts of a recorded early intervention visit. Technology issues have been a challenge.

This systematic change process in early intervention practices over time is key to measuring the impact of the

IA DMM professional development work. These observation data are directly related to meeting the goals of

implementing family-guided routines-based intervention and coaching caregivers.

Service providers are required to submit a self-assessment form with each video that is shared with the expert

coaches at Florida State University (FSU). The self-assessment includes sections to record: routines with 4

categories (i.e. play, caregiving, pre-academic, community and family); specific coaching strategies used;

Family-Guided Routines-Based Key Indicators (12 items); and SS-OO-PP-RR Framework or Setting the Stage-

Observation and Opportunities to practice; Problem solving and Planning-Reflection and Review. In addition,

the expert coaches at FSU complete the IA DMM Session Summary Form which is identical to the self-

assessment each provider completes. Data from the IA DMM Session Summary Forms are used to first, show

baseline from video 1 submitted prior to the beginning of the training, then to show change over the 9 month

training period as videos are submitted and expert coaching sessions occur.


26

FSU uses the information as part of the continuous improvement process for the individual participants.

Cohort level data is shared annually in June at the Joint Implementation Team Meeting (includes state and

regional implementation team members) and at regularly scheduled stakeholder meetings.

RESULTS Figure 15. Cohort 1-4 Observation Results, FGRBI SS-OO-PP-RR Key Indicators, Coaching Strategies, Routines Used

As expected, each cohort comes to the training with higher baseline scores. Early ACCESS regional liaisons

and/or supervisors attend training as observers. This helps them gain information to support trainees in the

shift to using evidence-based practices. They also share the information with others who are not yet trained

which piques interest and begins to build the capacity of future trainees. Each new cohort has some influence

from the proceeding cohorts. Overall, trend lines are upwards which indicates improvement.

0

2

4

6

8

10

12

14

Cohort 1 Cohort 2 Cohort 3 Cohort 4

Me

an N

um

be

r o

f In

dic

ato

rs

FGRBI SS-OO-PP-RR Key Indicators

0

10

20

30

40

50

60

70

80

90

100


Me

an P

erc

en

tage

of

Ke

y In

dic

ato

rs U

sed

FGRBI SS-OO-PP-RR Key Indicators

0

1

2

3

4

5

6

7

8


Me

an N

um

be

r o

f Sp

eci

fic

Co

ach

ing

Stra

tegi

es

Caregiver Coaching Strategies

0

0.5

1

1.5

2

2.5

3

3.5

4


Me

an N

um

be

r o

f R

ou

tin

es

Pe

r Se

ssio

n

Routines Use


27

A pilot study to check if each cohort continues to implement new practices with fidelity takes place during the

next reporting year. Permanent fidelity checks are built into the system by (1) requiring random samples of

providers to submit a video recorded home visit for assessment, and (2) training internal coaches to replace

the long distance mentors (Florida State University) with local staff. Internal coaches will provide continued

support to providers and ensure fidelity. Details will be submitted in the SSIP Phase III Year 2 report.

Table 16. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.1, L2Q6.2

Evaluation Question Are there more service providers participating as internal coaches for their peers? (L2Q6.1)

Are there more service providers participating as coaches to parents/caregivers?

(L2Q6.2)

Measure(s) Training completion count (# and %)

Data Collection Methods Online survey collected from EAGL at least annually + Participant lists from FSU Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S3, I2


Successes & Challenges: Easy to collect.

Knowing how many people have been trained helps determine future number of trainings required to meet

the needs of the state. Online surveys gather the information. Training rosters provide numbers of trained

providers and their discipline.

RESULTS

Regions vary widely in the percent of providers trained due to their different sizes. Survey results indicate that

the majority of Area Education Agencies (AEA) have less than 50% left to train. This information is used to

determine PD priorities, schedule future cohort training sessions and determine Early ACCESS fiscal needs for

PD. Figure 16 shows how many early interventionists have been trained and the different disciplines that have

participated.

Figure 16. Number of Participants by Cohort and Percentage of Disciplines Trained Cohort 1-4

ECSE59%

SLP18%

PT11%

OT9%

SC2%

Other1%

Cohort 1, 31

Cohort 2, 29

Cohort 3, 28

Cohort 4, 55

# of Participants by Cohort % Disciplines Cohorts 1-4


28

Table 17. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.3

Evaluation Question Are service providers able to understand and integrate multiple evidence-based approaches for supporting FGRBI and coaching caregivers?

(L2Q6.3)

Measure(s) Anecdotes from regional liaisons

Data Collection Methods Notes from Early ACCESS Leadership Group meetings & Regional Implementation Team updates Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S6, I5


Successes & Challenges: Information on other trainings are routinely gathered at naturally occurring statewide meetings of stakeholders. AEAs are investing in regional training on caregiver coaching through Rush and Shelden that compliments the IA DMM work. Some AEAs invest in Routines Based Interviews from Robin McWilliam that compliments the IA DMM work. Trainings on coaching from other experts in the early intervention field can confuse some providers.

RESULTS

The Iowa Distance Mentoring Model (IA DMM) of Professional Development (PD) for Early ACCESS operates

along side other PD efforts that are funded by the individual agencies providing services. However, IA DMM is

the only statewide effort that trains to use evidence based practices and uses active implementation

strategies to scale up and sustain the use of family-guided routines-based interventions through coaching

caregivers. For this reason, it is essential that our stakeholders work together to integrate what they learn

through local efforts and the statewide system change.

When agencies report struggles with integration, Florida State University staff bring together other experts via

webinars in order to strengthen everyone's understanding of how different approaches work together and not

against each other (i.e. Shelden & Rush, Robin McWilliam). In addition, other states that are doing similar

work join Iowa webinars. These efforts provide some of the evidence available to show progress in this area.

The goal is to have all early interventionists in Iowa using family-guided routines-based interventions and

caregiver coaching practices to improve outcomes for children and families regardless of which experts

provide training.


29

Table 18. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q7

Evaluation Question How do service providers assess the “readiness” of their agency for organizational change? (L2Q7)

Measure(s) Readiness for Organizational Learning and Evaluation Instrument (ROLE); Preskill & Torre (2000)

Data Collection Methods Online survey collected from each participant after their first face-to-face training Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S10, I8, L5


Successes & Challenges: Online process is simple. Good return rates for cohorts 1-4 (93%, 74%, 77%, 76%). Outside evaluation administering and analyzing results. Some providers answered survey based on their work unit with an organization and others on the entire organization, limiting use of results. Data are from only a few people, the Early ACCESS service providers participating in IA DMM not an entire agency or department so caution should be used when interpreting the results.

The Readiness for Organizational Learning and Evaluation or ROLE (Preskill & Torres, 2000) was designed to

help determine the level of readiness for implementing organizational learning, evaluation practices, and

supporting processes. This survey is completed by all cohort service providers after the first face-to-face

training.

The ROLE consists of 75 items grouped into six major areas: (1) Culture, (2) Leadership, (3) Systems and

Structures, (4) Communication, (5) Teams, and (6) Evaluation. Four of these areas – Culture, Leadership,

Systems and Structures, and Communication of Information – are further divided into subareas (see Table 19).

Table 19. ROLE Areas, Subareas, and Number of Items

Culture (27 items)

Leadership (12 items)

Systems & Structures (12 items)

Communication (8 items)

Teams (8 items)

Evaluation (8 items)

Collaboration & Problem Solving

12 items

No Subarea Open & Accessible Work Environment

4 items

Availability – 3 items

No Subarea No Subarea

Risk Taking 5 items

Rewards & Recognition 5 items

Dissemination 5 items

Participatory Decision Making

10 items

Relationship of Work to Organizational Goals

3 items

Service providers are asked to respond to items on a scale of 1 to 5, with 1 meaning “Strongly Disagree,” and 5

meaning “Strongly Agree.” There are five additional items/questions found on this instrument including the

following: a) three yes/no items about teaming practices, and b) two multiple choice items about demographic


30

information (job category and length of employment). This report contains the responses from those IA

DMM participants (cohort 1: 2013-cohort 4: 2015) who completed the ROLE instrument.

RESULTS

Preskill & Torres (1999, 2004) have developed a scoring sheet for analyzing the individual items and areas and

suggest that results for all respondents within a department or organization be combined and reported in

summary form. Average scores of 3.5 or higher were considered passing. The authors interpret average scores

below 3.5 as follows: “If an organization were to score low in one or more of the dimensions, this would

indicate that the department or organization isn’t prepared to engage in other kinds of organizational learning

practices” (Preskill & Torres, 1999, p. 11). Scores that fall below a 3.5 are highlighted in red for further review

and discussion by the members of the Regional Implementation Teams (RITs).

RITs explore if the items obtaining scores at or above an overall mean of 3.5 (in white and considered passing)

are indeed representative of their agency. Likewise, they explore if the items obtaining scores below an

overall mean of 3.5 (in red and considered needing attention) are representative of their agency.

The Joint Implementation Team Meeting held June 15, 2016 was used to share data to inform future plans as

well as discuss and develop Regional Implementation Team Action Plans. Stakeholders from the agencies

providing services, state departments, other early childhood organizations, the external evaluator and Florida

State University engaged in discussions and planning based on the data. Conversations continued at Early

ACCESS Leadership Group meetings.

The data below represent the average or mean scores by agency for Culture and Leadership (see Table 20) and

Systems & Structures, Communication, Teams and Evaluation (see Table 21). In addition to the tables on

average scores, analysis of the individual 75 items are used to drill down to specific results of each question.

This helps the agencies target specific items for review which can lead to change with their specific agency.

That information is not provided in this report since the table with the mean scores in each area provide the

overall results.

This program has given me confidence to go into a home visit and be partners with the parents in regards to helping them coach their child in these skills.


31

Table 20. Individual AEA/LEA Mean Scores for ROLE Culture and Leadership

AEA/LEA

Culture Leadership

Col

labo

ratio

n

& P

robl

em

Sol

ving

Ris

k T

akin

g

Par

ticip

ator

y

Dec

isio

n

Mak

ing

Total Mean Score

Total Mean Score

A 3.92 3.18 3.37 3.49 3.65

B 4.07 3.88 3.85 3.93 3.72

C 4.09 3.57 3.94 3.87 3.83

D 4.30 3.78 3.89 3.99 3.81

E 3.69 3.15 3.38 3.41 3.45

F 4.13 3.76 3.64 3.84 3.73

G 3.88 3.65 3.63 3.72 3.66

H 4.04 3.76 3.67 3.82 3.69

I 4.22 3.83 3.97 4.01 3.93

J 3.95 3.54 3.84 3.78 3.78

Total # of Mean Scores below

3.5 0 2 2 2 1

Overall there are high scores in the areas of Culture and Leadership. For two agencies, scores fell slightly

below the 3.5 overall mean score. Each has the detailed item results to address concerns.

Note: Agency name/number are de-identified to assure anonymity.


32

Table 21. Individual Agency Mean Scores for ROLE Systems & Structures, Communication, Teams and Evaluation

AEA/LEA

Systems & Structure Communication Teams Evaluation

Ope

n &

Acc

essi

ble

Wor

k E

nviro

nmen

t

Rew

ards

&

Rec

ogni

tion

Sys

tem

s &

Pra

ctic

es

Rel

atio

nshi

p of

Wor

k to

Org

aniz

atio

nal

Goa

ls

Total Mean Score

Ava

ilabi

lity

Dis

sem

inat

ion

Total Mean Score

Total Mean Score

Total Mean Score

A 3.11 3.00 3.70 3.29 3.33 3.73 3.53 3.66 3.59

B 3.44 3.35 4.09 3.63 3.75 3.73 3.74 3.80 3.50

C 3.38 3.06 4.29 3.58 3.81 3.80 3.81 4.04 3.50

D 3.04 3.15 3.92 3.37 3.75 3.88 3.82 4.04 3.50

E 3.46 2.80 3.74 3.33 3.23 3.51 3.37 3.53 3.60

F 3.28 3.12 4.33 3.58 3.93 4.02 3.99 4.11 3.45

G 3.16 3.25 4.14 3.51 3.47 3.65 3.56 3.75 3.54

H 3.20 2.90 3.87 3.32 3.40 3.74 3.57 3.96 3.49

I 3.46 3.29 4.33 3.69 3.82 4.05 3.94 4.11 3.53

J 3.36 3.64 4.09 3.54 3.36 3.64 3.50 3.81 3.42

Total # of Mean Scores below

3.5 10 9 0 4 5 0 1 0 3

There is a clear pattern of low scoring in the Open and Accessible Work Environment and Reward and

Recognition Systems and Practices subareas for all agencies. There are also pockets of other issues as

indicated by the red highlights in Communication and Evaluation. Each agency has the detailed item results to

address their specific concerns.


33

Table 22. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q8

Evaluation Question What was the most significant change as reported by providers? (L2Q8)

Measure(s) Reflection interview using the following questions: 1) what is working 2) what is tough 3) what I would do differently next time 4) what has been the most significant change so far

Data Collection Methods Video recording of participant reflection interviews Administered by Larry Edelman completed at the second face-to-face training (about 6 months into the training cycle)

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S2, L1


Successes & Challenges: Larry Edelman, early childhood and technology expert, working through Florida State University, has been critical to the success of capturing high quality video reflections of early intervention service providers. Interview data supports results found in other measures. Richer information collected that surveys cannot convey. Reviewing, transcribing and analyzing are very time intensive.

Reflective videos are recorded at the second face-to-face training session which is about 2/3 of the way

through the 9 month training cycle. Peer partners go the recording session as a group and are asked

prescribed questions (see Measures in above table). To date, cohorts 1-3 are under analysis and cohort 4 is

being transcribed.

The process used for analyzing the interviews includes the following steps. First, the three videos were typed

verbatim by one transcriber. Second, each transcription was reviewed by one evaluator. Key responses by the

participants were then highlighted or underlined. Third, the first cohort’s transcriptions were reviewed to

determine if any of the Division for Early Childhood’s Recommended Practices (2014) related to the area of

family were included in the responses. The highlighted transcription was reviewed to determine if any of the

practice characteristics identified on the four performance checklists developed by the Early Childhood

Technical Assistance (ECTA) Center were reported by the participants.

The four performance checklists included the following: 1) Family-Centered Practice Checklist, 2) Informed

Family Decision-Making Practice Checklist, 3) Family Engagement Practices Checklist, and 4) Family Capacity-

Building Practices Checklist. Each practice checklist contained as few as six or as many as nine practice

characteristics. Responses appeared to fall within a few of the characteristics from the Family-Centered or

Family Capacity Building Practices Checklist. At this time, the external evaluator has not conducted this type

of review with the second and third video tapes.


34

The fifth step was to summarize themes that emerged within each of the video tapes. This preliminary report

provides the results from video tapes 1 and 2 and information is provided in a bullet format where main

themes are captured with a top line or underline and supporting quotes shared that help illustrate those

themes. A final step would be to aggregate the data across all cohort summaries.

RESULTS

Responses obtained from cohort 1 were brief yet primarily positive. Themes that emerged included: (1)

increased confidence in their ability to partner with parents and other caregivers; (2) increased intentionality

when on visits to address components of the IA DMM model (using routines, asking questions, etc.); and 3)

their need to continue learning and practicing. A few did identify issues including the need for visits to

families to be more frequent than what is currently in place.

Responses from cohort 2 were lengthier. Themes that emerged included: (1) now having a new framework or

a focus to their visit; (2) they have increased the amount of time they are listening and observing families

while encouraging the families or care providers to do most of the communication, problem solving, and/or

decision-making; (3) they specifically discussed the shift away from toy bags compared to Cohort 1; and (4)

they have experienced issues with the technology and/or the time it takes to do the requirements of the

professional development model.

Some of the providers in cohort 2 identified issues or challenges when using this model with particular

families, especially families who were involved with Early ACCESS prior to the training, families who are in

crisis, families where the parents have developmental delays, and/or families and caregivers who have very

young infants that are also medically fragile. Another concern was teaming with other service providers who

have not been trained to coach the caregivers in family-guided routines-based early interventions. A sample

of service provider quotes are in Figures 17 and 18.

Figure 17. Examples of Interview Quotes from Cohorts 1 and 2

I think that DMM has given me a structure. It's helped me organize my own thinking and how I am going to approach the family. It's helped me realize I want the family to be the problem solvers. I want to guide the family to help them realize what's important to them. That they want to change, and then help them, guide them to make choices that they can try and see if they work to make their daily routines better.

The biggest change has been changing the outcome. How I write the outcomes to a more participation based outcome so we look at kind of a big picture thing about what we wanted to accomplish. What they want. What we want the child to participate in as a team and then back track a little bit and pick up the specific skills that they need to work on to be able to accomplish that participation.

I think the biggest change for me is to really look at the families’ routine…and help the family problem solve what will work in their routine…as far as working with their child on strategies or some of the interventions just trying to get their child to learn. And with a particular family that I have had it's just working on one thing at a time, but four or five times within the day within that routine.


35

Figure 18. Examples of Interview Quotes from Cohorts 1 and 2

I think the most significant change I've made since starting with DMM is that focused shift from my agenda, to coming in and asking families what their concerns are with their child. What they would like to work on with their child. Not just once, but like, coming in the next visit and saying, how's that going? That reflection. And then what would you like to work on today? What would you like to see happen this week? Kind of taking me out of it. I used to be such a planner and kind of here's the plan for the day. And that collaboration between myself and the families, I think that was probably the biggest change I’ve made with DMM is that collaboration piece with parents.

It's most difficult I think working with children at a child care center or children in the home who have an in-home nurse when the parent is not present to participate. Especially when sometimes the nursing is very inconsistent. I had a family that had a different in-home nurse every visit I went to, so it was much harder to engage them into daily routines and activities when every day they were just trying to get through the demands of their nursing job and just get to know the child for that day and meet their medical needs. So I find that very challenging.

Open-ended questions probably has been the most difficult for me. I tend to want to give information. I want to take care of families. I want to give them all those resources and supports that they need and get them to, give that to them immediately and so kind of stepping back and observing and letting them talk and share and working on those open ended questions has probably been the most difficult for me just to kind of sit back and listen. It's been a roller coaster. I’ve had days where I've felt pretty good about it… It's been a good experience. I think that I've seen some great changes in some of my families and I think probably the best part is just seeing the empowerment that I think this service is giving them. What I've had to kind of accept is that some of

(us) have (had) the training and some of us haven't and recognizing that It’s not going to be this overnight change for some people and recognizing that we just need to give it time for people, for those who it is a big change, to give them time to make that change. And recognizing those that have been doing (some )aspects of the DMM, that it's not a huge change and to allow time, to be patient with those for whom it is a big change.

The biggest change I've made in terms of looking at family focus in my first interview with the family is to ask the parents or the caregiver, what is their style of learning so I can tailor my materials, my response to their style of learning. Not assuming that they are a visual learner like I am. And that has been helping for providing pictures, providing information and parents have said that's been helpful.

I think the challenging part as an OT is the medically fragile children. The referral sources (are) looking at traditional based therapy, but being in a rural area where there are not hospital based therapists for these children to go to…that can be limiting for families who have limited transportation, that we are the only ones in the home. They don't have the option to go to the hospital for medically based or clinically based therapy services. The communication that needs to happen (from) medical providers and what they will describe to the families that will happen (in Early ACCESS) doesn't always have the same face.


36

F a m i l y L e v e l o f E v a l u a t i o n

Table 23. Selected Elements from Family Participant Level Evaluation Plan L1Q1

Evaluation Question How confident and competent do IA DMM family participants feel about helping their child develop and learn? (L1Q1)

Measure(s) Early Intervention Parenting Self-Efficacy Scale (EIPSES); Guimond, Wilcox & Lamorey (2008)

Data Collection Methods Survey online or via mail done not sooner than 9 months from initial entry into IA DMM

Administered by external evaluator and Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) I3, I4, L2

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, Fall 2015 completed cohorts 1-3 (C1-3) as scheduled

Successes & Challenges: C1-3 return rate = 44% (40/91) Reached families via email/online and paper surveys Difficulty getting accurate email addresses Period of time between participating in IA DMM and completing survey too long for majority of families Need more timely analysis and sharing results back with stakeholders including parents

The development of the EIPSES was based on existing concerns that few researchers have explored the role of

parenting self efficacy in families of children with disabilities (Gowen, Johnson- Martin, Goldman, &

Appelbaum, 1989; Hastings & Brown, 2002; Pit-ten Cate et al., 2002; Scheel & Rieckmann, 1998). Few existing

instruments allow professionals to focus on understanding whether parents feel competent and confident in

their skills, knowledge, and the ability to make a difference in the lives of their children, especially when their

infant or toddler also participates in early intervention services because of risk or developmental delay. A scale

with this emphasis is important because involvement with early intervention programs may provide

challenges to parents who are raising children with or at risk for developmental delays.

The EIPSES was developed to address the need for viable measures of change in families who are receiving

early intervention. The measure was developed with the intent to quantify parent perspectives about their

ability to facilitate positive child outcomes within the context of early intervention programs and via

interactions with early intervention practitioners.

This EIPSES survey was completed at the end of the cohort 3 training in Fall 2015 and included families from

cohorts 1-3 who participated in 2013 through 2015 (see Table 24). The survey was not administered prior to

training so there is no baseline or comparison data. This was by design as Early ACCESS state staff had to

balance all of the evaluation components with available resources and consider the paperwork already

required by the family at the beginning of the training and IFSP process. This was intended to act as a

feasibility study to make sure the survey and process was administered properly and the information would be


37

useful. Eventually, the survey will be completed earlier in the training process. The data from the EIPSES will

be used in identifying strengths and weaknesses regarding caregivers’ perceived abilities in promoting the

development of their children who are receiving early intervention services, information that might then be

used as targeted changes in the families’ intervention programs. This survey will be combined with direct

observations and interviews that will help assess effectiveness or impact of the professional development

efforts on the family.

RESULTS Table 24. Early Intervention Parenting Self-Efficacy Scale for Cohorts 1 through 3, Fall 2015

Early Intervention Parenting Self-Efficacy Scale (EIPSES) 1 to 7 Scale

N = 40 Cohorts 1-3

Range Average/Mean (SD)

Minimum Maximum

Parent Confidence (10 items)

2.40 5.30 3.38 (.69)

Parent Competence (4 items)

4.25 7.00 5.89 (.69)

Parent Self-Efficacy (16 items)

3.44 5.44 4.10 (.47)

Parents report feeling competent in their abilities to promote their child’s development (average score 5.89)

but less confident in their abilities to exert control over their children’s early intervention outcomes (average

score 3.38). In other words, caregivers feel they are able to do the interventions but are less sure that the

interventions are going to produce the outcomes wanted for their child.

Definitions Parent Confidence: The extent to which parents believe child outcomes are a function of environmental influences or constraints, such as family background or the availability of early intervention or community support. Parent Competence: The degree to which caregivers perceive themselves as being personally effective and capable in parenting their child.

I think the biggest change so far in this process has been really involving families more. Observing families more. I spent the last twenty years in this job doing and having families watch me and taking in my toy bag and I'm really trying to not take in toys and utilize more family options and inviting the families to sit down and play and observing them in their natural routines.


38

Table 25. Selected Elements from Family Participant Level Evaluation Plan L1Q2a-b, L1Q3

Evaluation Question How confident and competent do IA DMM family participants feel about helping their child develop and learn? (L1Q2a-b)

How confident do family participants feel about working with their child throughout the day? Has coaching changed how effective families feel about helping their child? (L2Q3)

Measure(s) Pre-Interview Survey and Family Interview

Data Collection Methods For a sample of families in DMM, surveys online or via mail done after the EIPSES has been completed. For a sample of families in DMM, after the end of the cohort DMM training cycle.

Administered by Florida State University

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S4, S5, L2

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, cohorts 1-3 scheduled for Spring 2016. Solicited participation during this time but evaluation not completed until Fall 2016 (next reporting period).

Successes & Challenges: Families did not respond to invitations to participate in interview. Asked multiple times and ways. Only one family signed consent and participated from cohorts 1-4. Adjusted interview schedule to align more closely with ending of IA DMM training so providers can personally encourage families to participate. Results will be submitted in next reporting period.

Table 26. Selected Elements from Family Participant Level Evaluation Plan L1Q4

Evaluation Question Are families demonstrating increased participation and proficiency in helping their child develop and learn? (L1Q4)

Measure(s) Caregiver Key Indicators

Data Collection Methods Video coding for a sample of families in DMM

Administered by Florida State University

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S5, I3, I4, L2

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, developed tool during current reporting period for use Summer 2016 (next reporting period)

Successes & Challenges: Instrument had to be developed. Key Indicators developed and piloted.

The primary aim of Part C services is to support the family’s confidence and competence in promoting their

child’s developmental needs. While supporting family capacity is a fundamental goal for services for children

under three, it remains a difficult construct to define and measure. There are limited options for measuring

family capacity building practices in the context of intervention sessions that matches the model that Early

ACCESS providers are being coached to use (FGRBI; Woods, Kashinath, & Goldstein, 2004).


39

In order to meet this need, Florida State University developed a tool to describe and quantify the caregiver’s

role in home-based intervention sessions. This tool, called Caregiver Key Indicators (see Figure 19), is intended

to be used in conjunction with the providers’ fidelity measure, and in some ways mirrors the provider fidelity

items. Instead of measuring provider fidelity, though, this tool examines behaviors from the vantage point of

how the caregiver participates in the intervention sessions. The Caregiver Key Indicators uses video

observation of full-length home visits in order to assess whether caregiver-child interactions are primary in the

session, whether the parent participates and practices embedding strategies in routines, whether the

caregiver engages in problem-solving and reflection on the intervention with the provider, and whether the

caregiver helps contribute to an action plan with the provider for the time between visits. The twelve-item

tool offers an overall percentage of indicators that are either present, partially present, or not present.

This tool is a means for providers and program administrators to see the ways in which they are engaging the

family rather than a way to evaluate a family’s participation or to fault them for not participating. Providers

and caregiver’s have a bidirectional relationship, and a family’s engagement and growth in capacity is linked to

what the provider does during each session. Some providers may create opportunities for the caregiver to

practice strategies in routines, but the family may not have a clear role in planning and problem-solving.

Likewise, providers may offer caregivers opportunities to share information and discuss developmental

challenges, but they may struggle to create practice opportunities for the caregivers to try strategies in every

day routines during the session. It is expected that there will be a relationship between the provider’s fidelity

scores and the scores on the Caregiver Key Indicators.

This tool is used to measure participant videos before service providers receive professional development

(PD), during the PD sequence, and after to measure change as a result of the provider’s training. In doing so,

we aim to measure not only what the provider is doing, but how the providers impact the family’s

engagement and participation in intervention sessions.


40

Figure 19. Caregiver Key Indicators Form

YES PARTIAL NO 1. Caregiver-child interaction is primary Yes if more than 50 % of time; Partial if 25-50 % of time, No if 24 % or less.

2. Caregiver participates in 2-3 typical/preferred routines from different routines categories? If they only do one type of routine but talk about others, that is scored as partial.

3. Caregiver uses family’s typical materials? 4. Caregiver embeds the child’s functional targets 3-5x in routines? If 3-5 times but only in one routine, score as partial. If 3-5 times in 2 or more routines, then yes. If 2 trials in one routine and three in another, then yes. If less than three times in only one routine, then no.

5. Caregiver practices intervention strategies 3-5x in routines? If 3-5 times but only in one routine, score as partial. If 3-5 times in 2 or more routines, then yes. If 2 trials in one routine and three in another, then yes. If less than three times in only one routine, then no.

6. Caregiver responds to feedback with comments or more practice at least one time per session?

7. Caregiver requests or responds to information and ideas at least one time per session? 8. Caregiver problem solves with interventionist on what is and is not working at least one time

per session?

9. Caregiver reflects on current session at least one time? 10. Caregiver summarizes action plan for intervention between visits?

RESULTS Figure 20 shows the percentage of weighted items observed on video 1 which is the baseline recording done

prior to the service provider's training. This is compared to the percentage of weighted items observed after

training took place. Data shows how the providers impact the family’s engagement and participation in

intervention sessions.

Figure 20. Caregiver Key Indicators Observations Cohort 4 Video 1 and Video 2

DRAFT NOT FOR DISTRIBUTION

0

10

20

30

40

50

60

70

80

90

100

Video1(Range0-91.67) Video2(Range25-91.67)

Percentageofweighteditems

C4CaregiverKeyIndicatorsData

Video 1 (Range 0-91.67) Video 2 (Range 25-91.67)

Per

cen

tage

of

wei

ghte

d it

em

s

C4 Caregiver Key Indicators Data


41

I m p l e m e n t i n g A g e n c y L e v e l o f E v a l u a t i o n

Table 27. Selected Elements from Implementing Agency Level Evaluation Plan L3Q9-Q11

Evaluation Question How do partnerships on the regional implementation teams change over the project period? (L3Q9)

What types of network/ collaboration changes occur? (L3Q10)

How and when do they begin to identify "others" who need to be part of their implementation team? (L3Q11)

Measure(s) Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006).

Data Collection Methods Online survey collected from each regional implementation team member annually Collaboration maps from survey results Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S7, L7


Successes & Challenges: Online process worked. Good return rates for each round of data collection (79%, 74%, 63%, 71%) Complex analysis and graphing process. Limited use by implementing agencies.

An increased level of collaboration among agencies providing early intervention services is an implied

objective as it is necessary in order for a statewide system to achieve and sustain long-term outcomes of

improving the results for infants and toddlers served in Early ACCESS. The Level of Collaboration Scale is

designed to measure levels of collaboration among organizations and individuals which is viewed as essential

for success. In addition, a format for visually displaying the results is obtained from using the scale. Even

though analysis and data displays are complex, the "picture" of collaboration for the Regional Implementation

Teams show interesting degrees of relationships.

Data were collected from members of 9 Area Education Agencies (AEAs) and Des Moines Public School District

during 2014 and 2015. Each agency's Regional Implementation Team (RIT) member was asked to participate in

a Levels of Collaboration Survey (Frey et al., 2006), where they were provided with a list of potential

collaborators from their respective AEA. Survey participants were asked to rate the relationship characteristic

of each potential collaborator on a scale of 1 to 5, where 1 corresponds to Networking and 5 corresponds to

Collaboration. For those individuals with whom respondents had no collaboration, they could select 0, “No

Interaction At All”. The Levels of Collaboration Survey, in addition to the definitions for each Relationship

Characteristic (Network to Collaboration), is in Table 28.


42

Table 28. Levels of Collaboration Definitions and Relationship Characteristics Five Levels of Collaboration and Their Characteristics

Networking

1 Cooperation

2 Coordination

3 Coalition

4 Collaboration

5

Relationship Characteristics

Aware of organization

Loosely defined roles

Little communication

All decisions are made independently

Provide information to each other

Somewhat defined roles

Formal communication

All decisions are made independently

Share information and resources

Defined roles

Frequent communication

Some shared decision making

Share ideas

Share resources

Frequent and prioritized communication

All members have a vote in decision making

Members belong to one system

Frequent communication is characterized by mutual trust

Consensus is reached on all decisions

Data was sent to The Center for Education Integrating Science, Mathematics, and Computing (CEISMC)

(pronounced like "seismic"), a partnership uniting the Georgia Institute of Technology with educational

groups, schools, corporations, and opinion leaders throughout the state of Georgia. (For more information go

to https://www.ceismc.gatech.edu/). The survey data was re-entered for compatibility with social network

analysis (SNA) software. Specifically, UCINet (Borgatti et al., 2002) was chosen for its ease of use in network

analysis and accompanying network visualization software.

Due to the high turnover of AEA members between 2014 and 2015, it was determined that the most reliable

way to analyze RIT networks was by each year separately. Changes in the strength of ties were explored

between the two years. The RIT networks of each AEA were analyzed by exploring changes in tie strength and

network connectedness within individual AEA Regional Implementation Team, as well as by exploring network

patterns observed within and across individual AEA RITs.

RESULTS

Three (out of 10) individual agency findings are included in this report as examples that will provide enough

information to build understanding of the results. The results for the other 6 AEAs and Des Moines Public

Schools are available in the full report from CEISMC.

AEA "X"

Between 2014 and 2015, AEA X consisted of 13 unique members in the Regional Implementation Team. In

2014, AEA X consisted of 6 members, three of whom left the team following 2014. These 6 members were

connected by a total of 24 ties. In 2015, there were 7 members of AEA X connected by 32 ties, three of whom

were new to the team during that year. Three AEA X members were present in both 2014 and 2015. Figure 21

illustrates the network maps for AEA X during the years 2014 and 2015.

https://www.ceismc.gatech.edu/


43

Figure 21. AEA "X" Network Maps for 2014 and 2015*

*Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey

During 2014, 20 ties, or 83.3% of ties were valued “Coordination (3)” or higher, and 66.7% of ties were valued

as “Coalition (4)” or higher. During 2015, 75% of ties were values as Coordination or higher, and again, 75%

were rated as Coalition or higher (see Figure 22). Overall, AEA X was a very well connected in both 2014 and

2015 by both measures of interest here, including number of ties and tie strength. Interestingly, in 2015, new

members of the AEA are all strongly connected to other members. Overall, few ties in the 2015 network can

be considered weak, indicating high levels of knowledge and resource distribution across the network.

Figure 22. AEA "X" 2014 and 2015 Ties Valued as Coalition or Higher*

2014 2015

AEA "X"

AEA "X" 4+ 2014 2015


44

AEA "Y"

Between 2014 and 2015, AEA Y consisted of 16 unique team members. In 2014, AEA Y consisted of 11

members, three of whom left the team following 2014. These 11 members were connected by a total of 82

ties. In 2015, there were 13 members of AEA Y Regional Implementation Team connected by 96 ties, 5 of

whom were new to the group during that year. Eight AEA Y members were present in both 2014 and 2015.

Figure 23 illustrates the RIT network maps for AEA Y during the years 2014 and 2015.

Figure 23. AEA "Y" Network Maps for 2014 and 2015* *Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey

During 2014, 48.8% of ties were valued “Coordination (3)” or higher, and 22% of ties were valued as “Coalition

(4)” or higher. During 2015, 47.9% of ties were values as Coordination or higher, and again, 43.8% were rated

as Coalition or higher (see Figure 24).

Figure 24. AEA "Y" 2014 and 2015 Ties Valued as Coalition or Higher*

AEA "Y" 4+ 2014 2015

AEA "Y" 2014 2015


45

AEA "Z"

Between 2014 and 2015, AEA Z consisted of 13 unique Regional Implementation Team (RIT) members. In

2014, AEA Z consisted of 9 members, including five who left the group following 2014. These 9 members were

connected by a total of 60 ties. In 2015, there were 8 members of AEA Z team connected by 36 ties, including

four people who were new to the group during that year. Four AEA Z members were present in both 2014 and

2015. Figure 25 illustrates the RIT network maps for AEA Z during the years 2014 and 2015.

Figure 25. AEA "Z" Network Maps for 2014 and 2015*

*Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey

During 2014, 20% of ties were valued “Coordination (3)” or higher and as “Coalition (4)” or higher. During

2015, 22.2% of ties were valued as Coordination or higher; however, no ties were rated as Coalition or higher.

As previously presented, there are much less ties in the overall network during 2015 than during 2014. In

addition, of the existing ties, relatively few were rated by survey respondents as strong ties. As illustrated in

Figure 26 , there are slightly more strong ties created among recurring members of the AEA during 2015.

However, the ties are still weaker than those in 2014, where 20% of ties were valued by respondents as 4 or

higher.

Figure 26. AEA "Z" 2014 and 2015 Ties Valued as Coalition or Higher*

AEA "Z" 2014 2015

AEA "Z" 4+ 2014 2015


46

Summative Patterns

Figures 27 and 28 present all agencies together, during 2014 and 2015, respectively, in an effort to examine

patterns that may occur throughout the AEA network. This summary examines network patterns, but is not

intended to be interpreted as a comparison of individual AEA networks to each other. Figures 27 and 28 are

meant to visualize these patterns in single diagrams, rather than as a comparison of individual agencies.

It is apparent throughout the majority of AEAs that the recurring network actors (orange nodes in the previous

figures) tend to have the strongest ties. As such, those networks with the least amount of turnover tend to

have the strongest overall networks, in terms of strength of ties. This should be expected, as recurring actors

have been contributing members of the AEA for a longer period of time; therefore, they have likely been

afforded a greater opportunity to become familiar with each other, and further collaborate and share ideas.

With few exceptions, this same pattern is not seen among members who were in the AEA for a single year,

represented by the pink nodes in the previous figures. In most cases, these AEA members have weaker

connections to all other nodes within the network. This pattern is important, since according to the Levels of

Collaboration Survey, strong ties (defined here as “Coordination” or higher) indicate AEA members who are

sharing information and resources communicate frequently and share in decision-making. Therefore, a

network in which a greater number of actors are connected by strong ties signifies an AEA in which there is a

greater flow of important information and resources to the whole network.

In addition, recurring members of AEA tend to be more central in the AEA networks. In most cases, those AEA

members most central to the network are again recurring members, or those orange nodes that represent

AEA members present in both 2014 and 2015. In less connected networks, those central actors hold the

greatest ability to move information and resources throughout the network. It is important to note that

network patterns indicate that those with the greatest ability to move resources and information throughout

the majority of AEA networks are those recurring members, which may signify the importance of limiting

member turnover. However, this may also be attributable to characteristics not available as a part of this

study, such as actor job role.

I am guessing the biggest change for me as a speech language pathologist is to leave my toy bag at home. That was kind of a big change. It's always easy to bring novel toys into a home and get kids engaged very easily. So now, you have to work with that a little bit more, but having to use their own toys and the things that they have in their house makes a whole lot more sense and actually you can get kids just as easily engaged that way. So I think that's probably the biggest change for me


47

Figure 27. All AEAs Network Maps, 2014*

Figure 28. All AEAs Network Maps, 2015*

*Key: The key for the diagram is as follows: Colors – Each Agency has same color for 2014 and 2015 Shapes: 2014 Only (Circle), Recurring Members (Square) Tie Thickness: Strength of Tie (Thicker = Greater Strength)


48

After receiving the results of the Levels of Collaboration Survey, volunteer AEA managers reviewed the data

and results based on a series of questions created by the external evaluator. The questions were about use of

this data. It was the opinion of AEA managers that this data is not useful at the Regional Implementation

Team (RIT) level unless further evidence on the importance of collaboration to successful practice change is

provided. The evaluation team discussed stakeholder feedback and decided one more round of surveys will

be collected in the fall of 2016 to analyze change in the implementation teams across time. This decision is

based on the fact that data from RIT Reports and anecdotal information gathered at Early ACCESS Leadership

Team meetings shows progress in team functioning.

Table 29. Selected Elements from Implementing Agency Level Evaluation Plan L3Q12, L3Q12.1

Evaluation Question How do implementation team members shift in their understanding of Active Implementation Frameworks? (L3Q12)

How do implementation team members shift in their use of Active Implementation Frameworks? (L3Q12.1)

Measure(s) Regional Imp Team (RIT) Reports State Imp Team (SIT) meeting notes State Work Team agendas & notes EAGL agendas & notes

Data Collection Methods RIT report updates collected via Google Docs SIT meeting notes State Work Team meeting notes collected via Google Docs EAGL meeting notes collected via Google Docs Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S9, S12, S17, I6, I7, I14, I15, L4, L8-L10

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, changed measure from Active Implementation (AI) Hub quizzes to more contextual measures and did collect data as scheduled.

Successes & Challenges: Making use of regularly occurring stakeholder meetings. Dedicated one day of the two-day statewide Early ACCESS Leadership Group (EAGL) meetings to FGRBI, coaching caregivers and implementation work. Florida State University participation via technology at EAGL meetings. Use Google Docs to routinely update progress on RIT activities. Face-to-face EAGL meetings used as RIT events where team members across the state who are not part of EAGL join electronically (via Zoom) to participate while other EAGL members who typically are uninvolved with RIT work participate as well. Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed. Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.

Regional Implementation Team Reports are updated via Google Docs 5 times per year at the EAGL meetings.

Table 30 shows the questions that are asked/updated at each meeting by all Area Education Agencies and Des

Moines Public Schools. All agencies record on the same document. This supports communication between


49

agencies as they can see what each other has reported. Discussions and actions happen based on the topics in

the reports and is recorded in the Early ACCESS Leadership Group meeting and State Work Team meeting

notes, also completed in Google Docs for easy accessibility by everyone.

RESULTS Table 30 includes examples of information reported and then used to continually improve the work. Table 30: Regional Implementation Team Report Questions & Examples of Data Collected

Who has attended the meetings Mid level management staff, past IA DMM cohort participants, internal coaches, AEA administrators, regional liaisons, instructor of early childhood college courses, community partner health providers, people who are ECSE teachers, occupational therapists, physical therapists, speech language pathologists

How many meetings have you held since August 2016? When are your future meetings? Range from one time per month to quarterly.

Highlights from the meetings

PD planning for cross-discipline meetings, outcome writing, internal coaching plans, initial PSP conversations.

Sharing videos and discussing. Sharing ideas to use with coaching teams. Updates on IA DMM resources available. Setting expectations for next year. Discussions on how we combine coaching, RBI and PSP to move forward.

Participated in April webinars. Met on 5/12 to further plan for FGRBI rollout beginning in 16-17 school year and work on Action Plan.

Cross disciplinary PD planning (new PD implementation beginning fall 2016).

In April and May, we have met to discuss the action plan for sustaining and scaling up DMM, organizing into teams to help support PSP, and having a plan to support the new staff who will be coming on July 1st.

Discussion focused on what PD will be needed for the coming year, particularly as we try to bring along staff who have not been part of a DMM cohort. Areas identified include outcomes, support for service coordinators, family assessments, RBI, and peer coaching.

Barriers or problems addressed or working on at regional level

How to best train new staff, maintaining implementation and working towards fidelity, administration change, PSP implementation.

Support from administrators for those providers that are hard to move forward in personal growth.

Pulling together all 0-3 staff to ensure everyone who has not participated in a Cohort will receive adequate training and support. Planning to have combined 0-3 provider meetings starting in 16-17.

Training staff who spend a small portion of their time in EA - they follow a more traditional model rather than coaching because of lack of training.

Not everyone (particularly SLPs, OTs, PTs, etc.) is putting FGRBI and caregiver coaching strategies into practice. Still having difficulty bringing along staff who have not been DMM participants. We are working on supporting our Regional Directors (middle management) in understanding FGRBI and caregiver coaching. Teams are having difficulty finding sufficient common time to collaborate and meet together.

Help or support needed from the State-level Implementation Team (SIT)

Sustaining FGRBI/rolling out to all AEA 9 providers, moving towards PSP (including training, coaching, learning from other regions).

Keeping the momentum and continuing support to move towards fidelity.

Working with the inter-rater reliability with the SS-OO-PP-RR and ensuring implementation teams have the opportunity to learn about it in depth as well (aka the 32 page manual).

Seeing the types of implementation and results data each AEA is collecting on top of what the state collects for the IA DMM work.

Focusing on service coordinator needs (family assessment, family outcomes, coaching in regard to resources, etc.).


50

Information gathered at each EAGL meeting is used to support the entire process of shifting the early

intervention system from child-focused services to a family focus. A database is being created to better

capture the information from these reports and meetings and tie them directly to actions taken. Relying on

Google Docs has become cumbersome with massive amounts of information to manage. Information

gathered helps continuous improvement efforts and plans for the future which are all in line with the State

Systemic Improvement Plan. Nothing indicates that the plan should change. The work is hard; vigilance is

never-ending.

Table 31. Selected Elements from Implementing Agency Level Evaluation Plan L3Q13, L3Q14

Evaluation Question How did systems change to accommodate this initiative? (L3Q13)

What are the barriers that impact agencies/service providers in supporting this model; what policies might impact usage of this model? (L3Q14)

Measure(s) Focus groups and interviews

Data Collection Methods Focus group and phone interviews with the regional implementation teams Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S11, S16, I6-I8, I13, I14, L5, L6, L8-L11

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) N/A not scheduled to occur during this reporting period.

Successes & Challenges:

Focus groups and interviews were originally scheduled as task for the external evaluator to take place Fall

2016, falling within the next reporting period. A change has occurred. The focus groups/interviews will be

administered by the Iowa Department of Education beginning Fall 2017 and results shared in the appropriate

reporting year.

It's a work in progress. I am learning more information. My main job is working with…the 3-5 teachers in daycare centers and right now I am working with families in home. So I feel like I am learning two-fold. I do have experience with families in home, but I have more experience with the centers and I'm working at this DMM model, putting it into practice with teachers as well...especially the coaching aspect of it and the problem solving aspect of it. So the teachers have more ownership of what they are working on with the child. In home, it definitely is kind of a challenge for me. So I am continuing to work on it.


51

State Level Evaluation

Table 32. Selected Elements from State Level Evaluation Plan L4Q15, L4Q16

Evaluation Question How did systems change to accommodate this initiative? (L4Q15)

How did early childhood systems change to accommodate this initiative across the state? (L4Q16)

Measure(s) Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006).

Data Collection Methods Online survey collected from state-level implementation team members Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S7, S16

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, this survey have been removed from the evaluation plan

Successes & Challenges: This is a very time intensive and expensive survey to administer and analyze. Results of the state level evaluation would have been used to support 3 short-term outcomes that have other, better measures already being used to show progress. System change is better measured by the Early Childhood Technical Assistance (ECTA) Center self-assessments, documents from team meetings and interviews. For these reasons, this item has been removed from the Systems Level Evaluation Plan for Early ACCESS. (It is still being used at the regional implementation level).


52

Table 33. Selected Elements from State Level Evaluation Plan L4Q17, L4Q17.1

Evaluation Question How do state implementation team members shift in their understanding of Active Implementation Frameworks? (L4Q17)

How do implementation team members shift in their use of Active Implementation Frameworks? (L4Q17.1)

Measure(s) Regional (RIT) and State Implementation Team (SIT) Reports State Work Team agendas & notes EAGL agendas & notes

Data Collection Methods RIT and SIT report updates collected via Google Docs State Work Team meeting notes collected via Google Docs EAGL meeting notes collected via Google Docs Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S9, S12, S17, I6, I7, I14, I15, L4, L8-L10

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, changed measure from Active Implementation (AI) Hub quizzes to more contextual measures and did collect data as scheduled..

Successes & Challenges: Making use of regularly occurring stakeholder meetings as well as dedicated implementation team meetings. Dedicated time for regional teams to update reports and specifically state what issues need to move to State Implementation Team. Use Google Docs to routinely update progress on regional and state activities. Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed. Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.

Regional Implementation Team (RIT) Reports are updated via Google Docs 5 times per year at the EAGL

meetings. One question on the RIT Reports asks the regional teams what help or support is needed from the

State-level Implementation Team (SIT). Information from this report is fed directly to the State Work Team

(SWT) (staff from the Iowa Departments of Education, Public Health, Human Services and Child Health

Specialty Clinics) to take 2 possible actions: (1) address the issue at the SWT level, or (3) move the issue to the

SIT level. Even when addressed at the SWT level, information is shared with the SIT at their regularly

scheduled meetings.

The State Work Team meets face-to-face twice a month with regular and frequent contact between meetings.

The SWT is responsible for overall functions of the Early ACCESS system and are all members of the State

Implementation Team. The State Implementation Team includes a larger group of early childhood

stakeholders and meets as needed (fall, winter, spring of this reporting period). A database is currently under

creation to better manage the large volume of written data collected during these meetings.


53

RESULTS

Evidence that State Implementation Team is increasing their knowledge of and use of active implementation

frameworks include during the current reporting period:

Development and use of written communication plans.

Rapid cycle problem solving using the Plan Do Study Act (PDSA) cycle used to address barriers that

impact statewide service provision

o Service coordination issues

o Eligibility criteria

Reviews of Regional Implementation Team Reports

Review of State Systemic Improvement Plan

Table 34. Selected Elements from State Level Evaluation Plan L4Q18a-b

Evaluation Question How did systems change to accommodate this initiative?

Governance (GV)

Personnel/workforce (PN)

Other components (L4Q18a-b)

Measure(s) ECTA Self-Assessments Implementation Teams, State Work Team, EAGL, ICEA meeting activities

Data Collection Methods Excel self-assessment forms collected annually for GV, PN Agenda and note documents from each group Administered by Iowa Department of Education

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S11, S13-S19, I9-I16, L5-L11


Successes & Challenges: ECTA self-assessment form is very useful. Intensive TA from the Early Childhood Personnel Center very useful in self-assessment and plan development for Personnel/Workforce component.

DaSy and ECTA Frameworks Self-Assessment Comparison Tool is an Excel-based instrument that provides a

structure for states to record the current status of the state system and set priorities for improvement. It is a

companion to the ECTA and DaSy System Frameworks and was developed with extensive input from Part C

and Section 619 staff from partner states (http://ectacenter.org/sysframe/selfassessment.asp). Stakeholders,

including state staff, completed the self-assessments for the Governance (GV) and Personnel/Workforce (PN)

components completed during the current reporting year. The ECTA Center staff helped with the process.

The PN component was completed for the birth through age 5 early childhood system including Early ACCESS.

GV was completed specific to Early ACCESS.

http://ectacenter.org/sysframe/selfassessment.asp


54

RESULTS Figure 29. ECTA System Framework Self-Assessment Governance Results

Initial or baseline results indicate: 4 Quality Indicators have half or more of the quality elements implemented (score 6-7, green bar). 3 Quality Indicators have at least some elements in place and a few fully implements (score 3-5, yellow bar) 1 Quality Indicator has most of the elements not yet planned or in place (score 2, red bar) 0 Quality Indicators have no elements yet to be planned or in place.

Figure 30. ECTA System Framework Self-Assessment Personnel/Workforce Results

Initial or baseline results indicate: 0 Quality Indicators have half or more of the quality elements implemented (score 6-7, green bar). 8 Quality Indicators have at least some elements in place and a few fully implements (score 3-5, yellow bar) 3 Quality Indicator has most of the elements not yet planned or in place (score 2, red bar) 1 Quality Indicators have no elements yet to be planned or in place.

Updating the self-assessment is a fall activity. The first update is Fall 2016 and will be reported in the next

SSIP. The ECTA/DaSy Self-Assessment Comparison Tool (released 6/29/16) will be used to assess progress

toward achieving intended improvements to different aspects of the Early ACCESS infrastructure.


55

Table 35. Selected Elements from State Level Evaluation Plan L4Q19

Evaluation Question What is working? What are the positive impacts? What, if any, barriers? What would you do differently? (L4Q19)

Measure(s) Interviews

Data Collection Methods Phone interview with DE Early ACCESS state coordinator and CSPD consultant Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) S16, S17, I10-I14, L4, L6, L10, L11

Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, data collection has taken place; analysis scheduled to occur during next reporting period.

Successes & Challenges: Phone interviews simple to do and can be recorded. Qualitative data analysis is time intensive.


Evaluation Question How are state IHEs impacted by this initiative? (L4Q20)

Measure(s) Interviews

Data Collection Methods Phone interviews with colleges and universities with pre-service programs that prepare students to be work in the field of early intervention Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) I10-I12, L6, L7, L10, L11



There's technology things that you have to get used to that are frustrating but it's a very worthwhile experience and I think in the end I'll be happy that I did it. The technology things that were challenging were like trying to get things uploaded. Making your camera work and how to set-up the camera at the house to make it effective. I still haven't figured that one out quite yet. The kids not wanting to look at it and feeling like it's right in the middle of everything.


56


Evaluation Question What lessons learned can we share with other entities and states wanting to develop a focused PD initiative? (L4Q21)

Measure(s) Surveys, interviews, focus groups previously listed in the plan

Data Collection Methods Case study documentation using results of data collected over complete project period Administered by external evaluator

Outcomes Addressed (from Logic Models) (progress reported in report Section E) L4, L6, L7, L10, L11



I really think that the philosophy and the ideas behind DMM are great. I think that they are what's best for families. It makes a lot of sense. It's just difficult to get from point A to point B. From where we are now to where we want to be. It's a big shift, but I think it will be worth it once we get there.


57

SECTION D: Data Quality Issues

There are no concerns or limitations related to quality or quantity of the data used to report progress or

results. Five levels of professional development evaluation data is collected (Guskey, 2000): (1) participant's

reactions; (2) participants' learning; (3) organization support and change; (4) participants' use of new

knowledge and skills; and (5) family outcomes. In addition, no single piece of data is used to assess progress

of the desired changes that will lead to the SiMR. Multiple measures are used at all levels of evaluation.

Collecting data from families means being aware of and sensitive to all that is going on in their life. Balancing

that with the required paperwork for participating in Early ACCESS and the consents to participate in the IA

DMM trainings, which include agreeing to be video recorded, adding "pre-test" surveys would be too much to

require of families and the service providers. This is taken into consideration when no "baseline" data is

available for a family measure.

When a selected measure does not answer an evaluation question, replacement measures are selected. For

example, using the quizzes in the AI Hub for learning Active Implementation Frameworks is not the best

measure for assessing change in actual understanding and use in the early intervention context. Regional and

State Implementation team reporting processes combined with Early ACCESS Leadership Group and Iowa

Council for Early ACCESS meeting notes provide evidence of contextual learning and use AI Frameworks.

Future plans remain the same, to hold regular evaluation team meetings where the evaluation plan is

reviewed and updated as needed.

I think it's a big learning process. I have an open mind and I think it is going to help families do better things for their kids. I especially like the components that we learned first about sitting back and observing what parents are already doing with their kids instead of coming in with their own agenda and saying "this is what you need to do. We need to fix everything because obviously you are not doing it right or I wouldn't be here”. But really looking and observing what they are already doing. Seeing what great things they are already doing and just trying to support and add more and embed interventions to the things that they do with their kids every day.


58

Section E: Progress Toward Achieving the Intended Improvements

The focus of Section E is on progress towards meeting outcomes which ultimately will lead to Iowa's State-

Identified Measurable Result (SiMR), increase in families who are confident and competent in helping their

child develop and learn, which then will improve outcomes for infants and toddlers served in Early ACCESS.

Tables are used to share progress towards each outcome identified in the logic models unless it was an

intermediate or long-term outcome where progress will be reported in future SSIPs. When that is the case,

text has been greyed out so you can still see the outcome but no progress is entered. Each table includes:

the outcome,

codes to indicate if it is a short-term (S), intermediate (I) or long-term (L) outcome,

evaluation codes which directly link to an evaluation question, measure and method of gathering evidence on

progress, and

accomplishments towards achieving the outcome.

Note: Data will not be duplicated between this section and Section C: Data on Implementation and Outcomes.

This section will explain progress and refer back to data in Section C tables where appropriate.

Progress on Instructional Practices: Shifting to Family-Guided Routines-Based Interventions and Caregiver

Coaching

Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes

Evidence-Based Practices — Short-term (S); Intermediate (I); Long-term (L) Outcomes

from Logic Models Eval Plan Code

Increased participant awareness, knowledge and skills of FGRBI and coaching caregivers.

Data from the Post-then-Pre evaluations shows participants are learning from the face-

to-face trainings. Participants are coming in with strong pre-existing learning and add to

that foundation during the face-to-face training sessions (see Table 15, Figures 11-15).

Service providers are applying what they have learned and improve performance over

the training period. Evidence is found in increased scores in video recorded

observations of home visits (see Table, 15, Figure 15).

Data from interviews (Table 22, Figures 17-18) supports the data from the Post-then-Pre Survey findings. Many participants made comments about the value of learning the SS-OO-PP-RR framework for coaching caregivers during home visits. Survey data shows the most learning occurred in practicing the use of this framework.

S1

L2Q5 (Table 10), L2Q6a-d (Table 15) L2Q8 (Table 22)


59

Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes (cont.)


from Logic Models (continued) Eval Plan Code

Increased participant satisfaction and confidence in service provision.

Data from the second face-to-face training (Table 10, Figures 14-15), held approximately

6 months into the training cycle, indicate that providers are bringing in a large amount of

pre-existing learning, benefiting from the training and still have room for growth

(learning lag).

Data from provider interviews recorded at the second face-to-face training reveal that,

although challenging, using family-guided routines-based interventions and coaching

caregivers is working. The full transcripts are still being analyzed over the next reporting

period for more details. Interviews as well as anecdotal information from face-to-face

trainings point towards the need for continued support beyond the 9 month training

period. Providers want to further develop skills and knowledge with support from

trainers.

Combined with information from all data sources, it is clear that progress is being made

towards meeting this outcome with additional support created to make that happen.

S2 L2Q5.1, (Table 10) L2Q8, (Table 22)

Increased # of providers coaching families.

4 cohorts have been trained as of this reporting period. New cohorts will be added as

scheduled (Figure 16). S3 L2Q6.2 (Table 16)

Increased # of families satisfied with service.

Data collection did not start during this reporting period. Difficulties in getting families

to commit to an interview slowed progress in gather data for this outcome. S4

L1Q2a-b (Table 25)

Increased families' awareness, knowledge and skills of FGRBI and coaching.

Although interviews have not been conducted as of this reporting period, evidence from

the newly developed Caregiver Key Indicator tool shows promising results (Table 26,

Figures 19-20). Through review of recorded home visit, caregiver behavior is observed

and measured. When comparing the results of the Video 1 (baseline) to Video 2, there is

noticeable improvement. This measure was not used beyond the development and

testing during this reporting period. Update will be reported in subsequent SSIP reports.

S5 L1Q3 (Table 25), L1Q4 (Table 26)

Increased understanding of multiple evidence-based approaches for supporting FGRBI and coaching caregivers.

All agencies have brought up the desire to learn more about the Primary Service Provider (PSP) approach to teaming. Efforts to support learning included conversations at the Joint Implementation Team Meeting (regional and state teams) with Rush and Shelden (via Zoom), book studies at some agencies, holding electronic meetings with other states that have or are still using PSP teaming, and many discussions during regularly scheduled stakeholder meetings.

A few agencies are trying PSP teaming and bringing their learnings to the Early ACCESS Leadership Group (EAGL) meetings for discussion.

Florida State University staff join conversations at EAGL meetings.

Agencies who have invested in Rush and Sheldon coaching training bring learnings to the EAGL meetings for discussion.

Webinars facilitated by Florida State University bring in Robin McWilliam, Dathan Rush and M'Lisa Shelden to help Iowa service providers integrate the multiple sources of support.

S6 L2Q6.3 (Table 17)


60

Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes (cont.)



Increased provider proficiency in and use of coaching caregivers in FGRBI with fidelity.

Fidelity checks are done during the next reporting period and will be in next SSIP. I1

L2Q6a-d (Table 15)

Increased # of providers who are internal coaches.

Internal coach training occurs during next reporting period. I2 L2Q6.1 (Table 16)

Increased family participation in helping their child develop and learn.

As reported in S5 above, the newly developed Caregiver Key Indicator tool shows

promising results (Table 26, Figures 19-20). It is premature to say we are making

progress on this intermediate outcome.

I3 L1Q1 (Table 23) L1Q4 (Table 26)

Increased family proficiency in using FGRBI. I4 L1Q1, L1Q4

Increased integration of multiple evidence-based approaches for supporting FGRBI and coaching caregivers.

I5 L2Q6.3

Increased quality of practice by early intervention professionals. L1 L2Q6a-d, L2Q8

Increased families who are confident and competent in helping their child develop and learn (Iowa's Part C SIMR).

L2 L1Q1, L1Q2b, L1Q3, L1Q4, APR C4

Improved outcomes for infants and toddlers with disabilities. L3 APR C3


61

Progress on Implementation Practices: Using Active Implementation Frameworks Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes

Implementation Strategies—Short-term (S); Intermediate (I); Long-term (L) Outcomes

from Logic Models Eval Plan Code

Increased # of strategic partnerships.

The Levels of Collaboration Scale measures different characteristics of collaboration: (1)

networking, (2) cooperation, (3) coordination, (4) coalition, and (5) collaboration and

results are displayed as maps of these relationship characteristics (Figures 21-28) in the

Regional Implementation Teams (RIT). In addition, the maps show the changing numbers

of people in the in the networks. Over time we will be able to see if the members

increase and if the relationship move nearer to collaboration.

Information from the RIT Report updates supports the above data. Teams share who has

attended the meetings. Comparisons between updates will help identify changes in

strategic partnerships. Over time, there has been an increase in mid level management

participation as well as increases in early interventionists other than teachers such as

occupational therapists (OT), physical therapists (PT), speech language pathologists (SLP).

Other evidence of progress towards increasing partnerships includes the numbers of

invitations to present Early ACCESS work to groups and councils (e.g. teachers of the

deaf/hard of hearing and audiologists, Early Head Start/Head Start, Early Hearing

Detection and Intervention Advisory Council).

S7 L3Q9, L3Q10, L3Q11 (Table 27)

I would like more time to talk with my peer about the things that we are doing sometimes we kind of do things on the fly as far as trying to watch our videos. I'm lucky to have a peer that's very tech savvy and can help on that end of things because there's been some learning that's had to happen for technology for me…getting the files sent to Florida State for example. The first one said it was going to take seven hours, so my peer had to help me send those things off and we couldn't figure out why the file was so long and why it was going to take so long. So we had to have our technology consultant come in and help with those things. Learning that the IPad is not the thing to use to video tape because that was partly the size issue.


62

Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)



Increased awareness and buy-in for evidence-based early intervention practices and implementation practices.

Occurs naturally as the number of partnerships increases the awareness of the work

increases.

Buy-in for implementing FGRBI and using Active Implementation Frameworks takes

time and deliberate actions to foster. Written administrative and participant

agreements have been modified as needed to increase buy-in. Creating written and

recorded products helps buy-in. Sharing information with the many groups that

implementation team members and State Work Team have contact with all

contribute to meeting this outcome.

Progress on building understanding, appreciation and buy-in for this work is slow,

steady and never ending. Iowa is in year 4 of training and shifting to statewide

implementation of evidence based practices. The need for improving awareness and

buy-in is still very apparent.

S8

L3Q12 (Table 29), L3Q13-14 (Table 31), L4Q17 (Table 33)

Increased knowledge and use of the 5 Active Implementation (AI) Frameworks.

Steady progress is being made in building knowledge and use of AI Frameworks.

Several approaches have been used including direct teaching through modules on

the AI Hub (http://implementation.fpg.unc.edu/) and presentations by State Work

Team members, Mary Beth Bruder from the Early Childhood Personnel Center and

Juliann Woods from Florida State University.

Using AI Frameworks and especially the continued development of implementation

teams takes a few key people to champion the work. Webinars helps support

growth in understanding and practice (Table 4, Figure 8). Most helpful is doing the

implementation work and then putting it in context of the frameworks. Simply

teaching what the frameworks are just introduces the concepts. It must be practiced

in the context of the system change to become a natural way for doing business.

Iowa is not there yet.

S9

L3Q12-12.1, (Table 29) L4Q17-17.1 (Table 33)

Increased readiness for implementation of EBPs.

At the very beginning of the IA DMM work, all agencies completed both practice and

technology needs assessments in order to develop professional development

components that align with participant and agency needs. From that point forward,

feedback from each cohort is used to continue alignment with needs, supporting

progress on this outcome.

Administrator, provider and trainer agreements evolve each year. Learning from the

previous years, agreements are tailored to address known barriers (e.g. time,

technology, data collection) up front in order to be better prepared for a training

cycle.

Each new cohort has higher levels of baseline skills observed during the first recorded

home visit done prior to any training (Figure 15). This shows that providers have

increased readiness for implementation of evidence-based practices.

S10

L2Q5 (Table 10), L2Q6a-d (Table 15), L2Q7 (Table 18)

http://implementation.fpg.unc.edu/


63

Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)



Increased support for PD at regional & state levels.

Interviews to gather data are not scheduled to occur during this reporting period

(Table 31).

ECTA System Framework self-assessments were done during this reporting period so

baseline data is available for change comparison next year.

Meeting notes will show that support from regional administrators has increased over

time with some negotiating over caseloads and contract hours to support the extra

work required as part of IA DMM.

At the beginning of this school year, Fall 2016, 3 agencies—separately from one

another—informed the state that service providers came to beginning-of-the-year

meetings and said they were all on board with the shift to using family-guided

routines-based intervention and caregiver coaching. Providers understand that this

shift is here to stay and that it is the "right" work to be doing. This is the end of year 4

and beginning of year 5 of implementation. Shift happens…and it takes time and

intense dedication to the process. These comments indicate a tipping point has been

reached. Providers and administrators realize this is not "one more thing" the state is

requiring today only to be gone tomorrow. However, this is not a statewide

sentiment and, although progress is evident, we still have a long way to go.

S11

L3Q13-14, (Table 31) L4Q18a-b

(Table 34)

Increased use of data related to implementation and intervention activities.

Using data is a constant conversation at Early ACCESS Leadership Group (EAGL), State

Work Team (SWT), Iowa Council for Early ACCESS (ICEA) and implementation team

meetings. Each audience has a different level of desire to use data and then

individuals within the groups have differences as well. Striking the right balance with

the right audience on data use has proven to be tricky.

Every June (since 2014) a Joint Implementation Team meeting is held with members

of the State Implementation Team (SIT), Regional Implementation Teams (RIT), State

Work Team (SWT), Florida State University and the external evaluator. Data sharing is

a natural part of these meetings so that the next round of training can be built from

what is learned. At the June 2016 event, a "data placemat" was used to assist in data

conversation and planning (see Figure 31 on following page).

EAGL meetings are used to share, examine, and collect regional and state level data.

In addition, these meetings are used to talk about what data would be useful for RIT

members to have. ICEA meetings do the same. The SWT serves as the group that

data flows through to the rest of the state.

Analyzing and sharing data more timely is a work in progress. Iowa does not have a

full-time data manager dedicated to Part C. Duties are shared with the part-time data

manager and the remaining State Work Team members.

S12

L3Q12.1 (Table 29), L4Q17.1 (Table 33)


64

Figure 31. Data Placemat (2-sided, 8.5 x 14) from June 15, 2016 Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)



Active Implementation Stages, Drivers, and Improvement Cycles are carried out by skilled individuals with the expertise to help individuals, organizations, and systems successfully use evidence-based practices.

I6 L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1

Increased capacity for identifying, implementing, scaling up and sustaining evidence-based practices and programs for continuous improvement of the Early ACCESS system.

I7 L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1

Increase in skilled providers who feel supported at multiple levels to use evidence-based practices.

I8 L2Q7, L3Q13, L3Q14

AI Frameworks are used to make a persistent and consistent effort to bring about change and improve outcomes for children and families.

L4 L3Q12, L3Q12.1, L4Q17, L4Q17.1, L4Q19, L4Q21

Early ACCESS system is aligned to support practitioners’ efforts to use evidence-based practices.

L5 L2Q7, L3Q13, L3Q14, L4Q18a-b

State and regional policies promote implementation science and best practices. L6 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21

Diverse stakeholders' voices and experiences influence and guide implementation efforts. L7 L3Q9, L3Q10, L3Q11, L4Q18a-b, L4Q20, L4Q21

Reliable and valid data is regularly available for decision making at all levels: practitioners, managers of provider organizations, state departments, policy makers and funders.

L8 L3Q12.1, L3Q13, L3Q14, L4Q17.1, L4Q18a-b

A proactive and thoughtful style of solving problems is used routinely. L9 L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q18a-b


65

Progress on High Quality Infrastructure: ECTA System Framework

Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Governance

Governance Infrastructure

Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models Eval Plan Code

Staff and stakeholders increased knowledge of structures and partnerships that are needed to support an effective, efficient statewide service delivery system.

Focus groups and interviews with Regional Implementation Teams are not scheduled

to occur during this reporting period. Data will be shared in next reporting period.

ECTA System Framework Self-Assessment has been scored once (baseline) and an

update will be shared in next reporting period.

Data collection has taken place for state staff interviews; analysis is not scheduled to

occur during this reporting period.

Results from the baseline assessment analysis for the Governance Component

indicated a high priority was Quality Indicator 5: State and regional and/or local system

entities are designed to maximize meaningful family engagement in the development

and implementation of the system. A multi-agency task team of 6 people, half being

parents of children with disabilities, has formed to organize a system wide self-

assessment specific to family engagement. Activities are scheduled to occur during the

next reporting period.

S16

L3Q13-14 (Table 31) L4Q18a-b (Table 34) L4Q19 (Table 35)

Staff and stakeholders use knowledge and skills to implement plans.

The infrastructure improvement plan submitted during Phase II (Memorandum of

Agreement Action Plan) is used to ensure activities happen within the designated

timelines as much as possible.

Balancing infrastructure development and a shift to evidence-based practices which

must happen simultaneously is challenging. Having a written agreement

(Memorandum of Agreement) that binds 4 signatory agencies (Iowa Department of

Education, Department of Public Health, Department of Human Services, Child Health

Specialty Clinic) together as responsible for the Early ACCESS systems allows for

progress to be made no matter how small the steps may be at times.

S17

L3Q12, L3Q12.1 (Table 29) L4Q17, L4Q17.1 (Table 33) L4Q18a-b (Table 34) L4Q19 (Table 35)


66

Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Governance (cont.)

Governance Infrastructure

Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models (continued) Eval Plan Code

Effective and well-coordinated governance infrastructure that provides enforceable decision-making authority to effectively implement the EA system.

I13 L3Q14, L3Q13, L4Q18a-b, L4Q19

Leadership advocates and leverages sufficient fiscal and human resources to support quality services.

I14

L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19

Mechanisms are in place that facilitate clear communication, collaboration and relation-building with stakeholders and partners at all levels.

I15 L3Q12, L3Q12.1, L4Q17, L4Q17.1, L4Q18a-b

Early ACCESS system encourages, supports and requires implementation of effective practices.

L10

L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19, L4Q20, L4Q21

EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system.

L11 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21

Table 41. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Personnel/Workforce

Personnel/Workforce Infrastructure


Staff and stakeholders increased knowledge of a CSPD.

Iowa received intensive technical assistance from the ECTA Center to create an Early

Childhood Comprehensive System of Personnel Development (CSPD) for people serving

children ages birth to five. Early ACCESS reviewed the larger plan and specifically

noted items that directly impact early intervention personnel. The vision for personnel

development was create as a statewide vision for all early childhood entities and the

visioning activity was done as an Early Childhood Iowa subcommittee activity.

Information about what an early childhood CSPD still needs to be shared around the

state even though a broad representation of stakeholders were involved in the

visioning process and development of the improvement plan for Personnel/Workforce

infrastructure improvement.

S13 L4Q18a-b (Table 34)

Staff and stakeholders increased knowledge of personnel available in the state to meet service needs.

Information on staff shortages in different disciplines that service infants and toddlers

with disabilities has been gathered and shared with a few. Further distribution will

occur during the next reporting period (see Table 38 on following page).


Staff and stakeholders use knowledge and skills to implement plan.

A core team of public and private early childhood agencies are responsible for moving

the work forward. Re-assessing will happen in the fall and change will be reported in

the next reporting year.



67

Table 41. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Personnel/Workforce (cont.)

Personnel/Workforce Infrastructure

Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models (continued) Eval Plan Code

Effective and well-coordinated personnel/workforce infrastructure that guides planning, development, implementation and evaluation of a CSPD.

I9 L4Q18a-b

Personnel/workforce infrastructure ensures that infants and toddlers with disabilities and their families are provided services by knowledgeable, skilled, competent, and highly qualified personnel.

I10 L4Q18a-b, L4Q19, L4Q20

EA CSPD promotes both effective practices and the implementation of legal requirements. I11 L4Q18a-b, L4Q19, L4Q20,

EA has increased coordination between preservice programs and inservice training to ensure consistency of practices.

I12 L4Q18a-b, L4Q19, L4Q20


L10




Table 42. Licensed Area Education Agency and District Staff that Could Retire in Next Five Years (Based on Rule of 88 or Retirement at Age 65.

Retiring

FTE 2014 0-2

FTE 2014 3-5

FTE 2014 3-21

TOTAL PEOPLE

Total FTE % leaving

PK teacher 32 99.13 26.56 471.00 6.79%

OT 24 17.54 77.12 94.66 25.35%

PT 18 16.1 51.36 67.46 26.68%

Sch SW 38 18.52 209.83 228.35 16.64%

Sch Psych 62 5.52 286.87 292.39 21.20%

SLP 154 49.78 470.12 519.90 29.62%

Rather than based on numbers of retiring staff, the FTE from each AEA were used to calculate the actual percentages (based on number of staff in DE October Count data from recent 2014-2015 were used). Of the programs with greater than 25% loss, there is only one program in Iowa graduating OT, three programs graduating SLP, and four programs graduating PT. Drake and Allen College have started OT programs but graduating classes are still 4-5 years away.

It's been really good for me to be insightful about my own practice and just being more thoughtful about each visit when I go in, who is this child, who is this family, where were they, where are we going, that kind of thing.


68

Table 43. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Accountability & Quality, Quality Standards, Data System, Finance

Accountability & Quality, Quality Standards, Data System, Finance Infrastructure


Staff and stakeholders increased knowledge of Accountability & Quality, Quality Standards, Data System, and Finance components.

S18 L4Q18a-b

Staff and stakeholders commitments to completing self-assessments by agreed upon deadlines.

Scheduled to occur in Fall 2017. S19

L4Q18a-b (Table 34)

Improvement plans are developed for all components. I16 L4Q18a-b


L10




SECTION F: Plans for Next Year

In 2016-2017, IA DMM will continue as planned:

Cohort 5 training for groups of 3 new providers or trios;

Cohort 5 training for first group of internal coaches;

Webinars for cohorts, implementation teams, Community of Practice;

FaceBook and website maintenance;

Meetings of implementation teams

Engagement of stakeholder groups

Use of task teams made up of diverse stakeholders to address needs as they arise

Evaluation will continue as written in the Systems Level Evaluation Plan for Early ACCESS with one change. We

will no longer hire an external evaluator and the State Work Team staff will take on those responsibilities.

Details and timelines are in the plan.

Systems are in place for addressing barriers to the work. Protocols and communication between trainers,

regions and the state are well established.

There is no additional support required at this time.


69

References

Borgatti, S.P., Everett, M.G., Freeman, L.C. (2002). Ucinet for Windows: Software for Social Network Analysis

[Computer Software]. Harvard, MA: Analytic Technologies. Frey et al. (2006). Measuring collaboration among grant partners. American Journal of Evaluation 27, 383-392.

DOI: 10.1177/1098214006290356 Gowen, J.W., Johnson-Martin, N., Goldman, B. D., & Appelbaum, M. (1989). Feelings of depression and

parenting competence of mothers of handicapped and nonhandicapped infants: A longitudinal study. American Journal on Mental Retardation, 94, 259-271.

Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press. Hastings, R.P., & Brown, T. (2002). Behavior problems of children with autism, parental self-efficacy, and

mental health. American Journal on Mental Retardation, 107, 222-232. Pit-ten Cate, I., Kennedy, C., & Stevenson, J. (2002). Disability and quality of life in spina bifida and

hydrocephalus. Developmental Medicine and Child Neurology, 44(5), 317-322. Preskill, H. & Torres, R.T. (2000). Readiness for Organizational Learning and Evaluation Instrument (ROLE),

based on Preskill, H. & Torres, R. T. (1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage. For more information about the survey, please contact Hallie Preskill at [email protected]. Reliability data for the instrument are can be found in: Preskill, H., Torres, R. T., & Martinez-Papponi, B. (1999). Assessing an organization’s readiness for learning from evaluative inquiry. Paper presented at the American Evaluation Association annual conference. Orlando, FL, November.

Scheel, M.J., & Rieckmann, T. (1998). An empirically derived description of self-efficacy and empowerment for

parents of children identified as psychologically disordered. American Journal of Family Therapy, 26, 15-27.

Woods, J., Kashinath, S. & Goldstein, H. (2004), Effects of embedding caregiver implemented teaching

strategies in daily routines on children’s communication outcomes. Journal of Early Intervention, 26, 175-193

One of the most significant things I have identified is that I would typically go in with a mindset of what I was going to do, which handout I would give the family, what intervention I would model and I was comfortable with that. So now I am having to be flexible and be really open to whatever they want to work on and whatever they want to talk about. That's hard.

mailto:[email protected]

Early ACCESS, IDEA Part C Early Intervention System Iowa Department of Education, March 11, 2107 Page 70

Inputs Outputs Short-term (direct results of activities and outputs)

Intermediate (changes in the actions or behaviors)

Long-term (anticipated results based on goals)

Outcomes

A. Design and implement professional development plan to build and expand capacity of providers in coaching caregivers and family-guided routines-based intervention (FGRBI)

Increased provider proficiency in and use of coaching caregivers in FGRBI with fidelity (I1) Increased # of providers who are internal coaches (I2) Increased family participation in helping their child develop and learn (I3) Increased family proficiency in using FGRBI (I4)

B. Develop and implement plan for tracking other related professional development that occurs within individual Area Education Agencies and signatory agencies that supports coaching caregivers and FGRBI

Comprehensive list of statewide professional development and capacity building opportunities in the Part C system Common data collection tools for FGRBI and coaching caregivers

Increased quality of practice by early intervention professionals (L1) Increased families who are confident and competent in helping their child develop and learn (Iowa's Part C SIMR) (L2)

Improved outcomes for infants and toddlers with disabilities (L3)

Activities

Increased integration of multiple evidence-based approaches for supporting FGRBI and coaching caregivers (I5)

Field tested coaching curriculum, coaching implementation competencies and learning modules Fidelity checklists Timeline for cohort participation Cohort peer mentoring pairs with technology supported action plans Participant videos of coaching and performance feedback Number of individuals accessing webinars, CoP, social media Number & type of print materials, EB practice guides, coaching curriculum, online learning modules

Increased participant awareness, knowledge and skills of FGRBI and coaching caregivers (S1) Increased participant satisfaction and confidence in service provision (S2) Increased # of providers coaching families (S3) Increased # of families satisfied with service (S4) Increased families' awareness, knowledge and skills of FGRBI and coaching (S5)

Increased understanding of multiple evidence-based approaches for supporting FGRBI and coaching caregivers (S6)

DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA Centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds

Early ACCESS Logic Model: New Instructional Practice

Evaluation: Measures/indicators, data collection, analysis and interpretation, feedback for program improvement, assessment & validation

External Factors/Context: staff turnover; other professional development efforts related to improving practice and outcomes; Area Education Agency system; signatory agency shared leadership structure

Appendix A, SSIP Phase III Year 1


Inputs

DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA Centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds

Outputs Short-term (direct results of activities and outputs)



Outcomes

A. Establish collaborative planning & implementation teams & networks

AI Frameworks are used to make a persistent and consistent effort to bring about change and improve outcomes for children and families (L4)

Early ACCESS system is aligned to support practitioners’ efforts to use evidence-based practices (L5)

State and regional policies promote implementation science and best practices (L6) Diverse stakeholders' voices and experiences influence and guide implementation efforts (L7)

Reliable and valid data is regularly available for decision making at all levels: practitioners, managers of provider organizations, state departments, policy makers and funders (L8)

A proactive and thoughtful style of solving problems is used routinely (L9)

Activities

D. Maintain communication with stakeholders, PD participants, EA leadership team, and council

Educational trainings/outreach events with stakeholders and public Documents, meeting notes, communication with stakeholders (e.g. IHEs, EA leadership group, ICEA, signatory agencies)

PD vision and plan developed with stakeholder input Administrator & provider participation agreements Completed AI Hub modules

Active Implementation Stages, Drivers, and Improvement Cycles are carried out by skilled individuals with the expertise to help individuals, organizations, and systems successfully use evidence-based practices (I6)

Increased capacity for identifying, implementing, scaling up and sustaining evidence-based practices and programs for continuous improvement of the Early ACCESS system (I7)

Increase in skilled providers who feel supported at multiple levels to use evidence-based practices (I8)

B. Design and implement plan to build capacity of state and regional teams in Active Implementation Frameworks

C. Assess readiness in the Area Education Agencies

Needs assessment Assess current EI policies, practices, etc. Assess and acquire resources needed to do the work

Regional and state teams with diverse stakeholder membership

PD components that align with the identified needs of participants Acquired or repurposed resources needed to do the work

Increased # of strategic partnerships (S7) Increased awareness and buy-in for evidence-based early intervention practices and implementation practices (S8)


Increased knowledge and use of the 5 Active Implementation Frameworks (S9) Increased readiness for implementation of EBPs (S10) Increased support for PD at regional & state levels (S11) Increased use of data related to implementation and intervention activities (S12)


Early ACCESS Logic Model: New Implementation Strategies


Early ACCESS Logic Model: New High Quality Part C System

Inputs

DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds


Outputs Short-term (direct results of activities and outputs)



Outcomes

A. Complete ECTA self-assessment for Governance (GV) component; engage stakeholders to develop and implement GV improvement plan

Effective and well-coordinated governance infrastructure that provides enforceable decision-making authority to effectively implement the EA system (I9) Leadership advocates and leverages sufficient fiscal and human resources to support quality services (I10) Mechanisms are in place that facilitate clear communication, collaboration and relation-building with stakeholders and partners at all levels (I11) Early ACCESS system

encourages, supports and requires implementation of effective practices (L10)

EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system (L11)

Activities

Improvement plans are developed for AC, QS, DS and FN components (I16)

Effective and well-coordinated personnel/workforce infrastructure that guides planning, development, implementation and evaluation of a comprehensive system of personnel development (CSPD) (I12) Personnel/workforce infrastructure ensures that infants and toddlers with disabilities and their families are provided services by knowledgeable, skilled, competent, and highly qualified personnel (I13) EA CSPD promotes both effective practices and the implementation of legal requirement (I14) EA has increased coordination between preservice programs and inservice training to ensure consistency of practices (I15)

B. Complete ECTA self-assessment for Personnel/Workforce (PN) component; engage stakeholders to develop and implement PN improvement plan

C. Review and prioritize remaining ECTA System Framework component for future completion:

Accountability & Quality (AC) Quality Standards (QS) Data System (DS) Finance (FN)

Completed GV self-assessments with areas identified for improvement Coordinated plan to address needed improvements in GV component

Prioritized list for addressing all remaining system components Timeline for completing all remaining self-assessments

Staff and stakeholders increased knowledge of structures and partnerships that are needed to support effective, efficient statewide service delivery system (S13) Staff and stakeholders use knowledge and skills to implement plans (S14)

Staff and stakeholders increased knowledge of Accountability & Quality, Quality Standards, Data System and Finance components (S18) Staff and stakeholders commitments to completing self-assessments by agreed upon deadlines (S19)

Completed PN self-assessments with areas identified for improvement Coordinated plan to address needed improvements in PN component

Staff and stakeholders increased knowledge of a comprehensive system of personnel development (CSPD) (S15) Staff and stakeholders increased knowledge of personnel available in the state to meet service needs (S16) Staff and stakeholders use knowledge and skills to implement plan (S17)


Iowa Department of Education pg. 73 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17

Systems Level Evaluation Plan for Early ACCESS Plan updated 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17

Project periods are Federal Fiscal Years (FFY) 10/1 through 9/30 of each year with the exception of Project Year 1 which was 4/1/13 – 9/30/13.

System Level Evaluation Questions

Measures & Formative or Summative

Data Collection Methods Cohort Year 1

2012-2013 Year 2

2013-2014 Year 3

2014-2015 Year 4

2015-2016 Year 5

2016-2017 Year 6

2017-2018 Year 7

2018-2019

I. Program

Recipients Family Participants

L1Q1: How confident and competent do IA-DMM family participants feel about helping their child develop and learn? I3, I4, L2

Early Intervention Parenting Self-Efficacy Scale (EIPSES); Guimond, Wilcox & Lamorey (2008) Summative

Survey online or via mail done not sooner than 9 months from initial entry into IA-DMM By external evaluator and DE

1 Fall 2015

2 Fall 2015

3 Fall 2015

4 Summer 2016

5 Summer 2017

6 Summer 2018

7 Summer 2019

L1Q2a: Were family members satisfied with this type of intervention? What worked? What barriers? S4

Pre-interview survey Formative

For a sample of families in DMM, surveys online or via mail done after the EIPSES has been completed. By FSU

1 Spring 2016

2 Spring 2016

3 Spring 2016

4 Fall 2016

5 .Fall 2017

6 Fall 2018

7 Fall 2019

L1Q2b: Were family members satisfied with this type of intervention? What worked? What barriers? S4, L2 L1Q3: How confident do family participants feel about working with their child throughout the day? Has coaching changed how effective families feel about helping their child? S5, L2

Family interviews Formative & Summative

For a sample of families in DMM, after the end of the cohort DMM training cycle. By FSU

1 Spring 2016

2 Spring 2016

3 Spring 2016

4 Fall 2016

5 Fall 2017

6 Fall 2018

7 . Fall 2019

L1Q4: Are families demonstrating increased participation and proficiency in helping their child develop and learn? S5, I3, I4, L2

Caregiver Key Indicators Summative

Video coding for a sample of families in DMM By FSU

1

2

3

4 Summer 2016

5 Summer 2017

6 Summer 2018

7 Summer 2019

Red text = Outcome # from Logic Models Appendix B, SSIP Phase III Year 1





2012-2013 Year 2

2013-2014 Year 3

2014-2015 Year 4

2015-2016 Year 5

2016-2017 Year 6

2017-2018 Year 7

2018-2019

II. Direct Service

Provider Early ACCESS service providers

L2Q5: How do service providers' perceptions of their abilities to implement FGRB Interventions change? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10

Post-then-Pre Evaluation Survey Formative & Summative

Paper survey completed by each participant at the end of each face-to-face PD session By FSU

1 Fall 2013 Winter 2013

2 Spring 2013 Fall 2013

3 Fall 2014 Spring 2015





L2Q5.1 How do service providers' satisfaction and confidence in service provision change? S2

Post-then-Pre Evaluation Survey Summative

Paper survey completed by each participant at the end of 2nd face-to-face PD session By FSU

5 Spring 2017

6 Spring 2018

7 Spring 2019

L2Q6a: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1

Family Routine Categories Formative & Summative

IA-DMM session summary form By FSU

1 Summer 2014

2 Summer 2015

3 Summer 2016

4 Summer 2017

5 Fall 2017

6 Fall 2018

7 Fall 2019

L2Q6b: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1

FGRBI Key Indicators Formative & Summative


1 Summer 2014

2 Summer 2015

3 Summer 2016

4 Summer 2017

5 Fall 2017

6 Fall 2018

7 Fall 2019

Red text = Outcome # from Logic Models





2012-2013 Year 2

2013-2014 Year 3

2014-2015 Year 4

2015-2016 Year 5

2016-2017 Year 6

2017-2018 Year 7

2018-2019

II. Direct Service

Provider (cont.)

Early ACCESS service providers

L2Q6c: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1

SSOOPPRR Framework Formative & Summative


1 Summer 2014

2 Summer 2015

3 Summer 2016

4 Summer 2017

5 Fall 2017

6 Fall 2018

7 Fall 2019

L2Q6d: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1

Coaching Strategies Formative & Summative


1 Summer 2014

2 Summer 2015

3 Summer 2016

4 Summer 2017

5 Fall 2017

6 Fall 2018

7 Fall 2019

L2Q6.1 Are there more service providers participating as peer internal coaches? I2

Training completion count (# and %) Formative

Online survey collected from EAGL at least annually + Participant lists from FSU By DE

Summer 2017 Summer 2018 Summer 2019

L2Q6.2 Are there more service providers participating as coaches to caregivers? S3

Training completion count (# and %) Formative

Online survey collected from EAGL at least annually + Participant lists from FSU By DE

Summer 2016 Summer 2017 Summer 2018 Summer 2019

L2Q6.3 Are service providers able to understand and integrate multiple evidence-based approaches for supporting FGRBI and coaching caregivers? S6, I5

Anecdotes from regional liaisons Formative

Notes from EAGL meetings & Regional Imp Team updates By DE

Summer 2014 Summer 2015 Summer 2016 Summer 2017 Summer 2018 Summer 2019

L2Q7: How do service providers assess the “readiness” of their agency for organizational change? S10, I8, L5

Readiness for Organizational Learning and Evaluation Instrument (ROLE); Preskill & Torre (2000) Formative

Online survey collected from each participant after their first face-to-face training By external evaluator

1 Fall 2013

2 Spring 2014

3 Fall 2014

4 Fall 2015

5 Fall 2016

6 Fall 2017

7 Fall 2018






2012-2013 Year 2

2013-2014 Year 3

2014-2015 Year 4

2015-2016 Year 5

2016-2017 Year 6

2017-2018 Year 7

2018-2019

II. Direct Service

Provider (cont.)

Early ACCESS service providers

L2Q8: What was the most significant change as reported by providers? S1, S2, L1

Reflection interview using the following questions: 1) what is working, 2) what is tough, 3) what I would do

differently next time, 4) what has been the most

significant change so far Summative

Video recording of participant reflection interviews By Larry Edelman completed at the second face-to-face training (about 6 months into the training cycle)

1 Spring 2014

2 Fall 2014

3 Spring 2015

4 Spring 2016

5 Spring 2017

6 Spring 2018

7 Spring 2019





Data Collection Methods

Year 1 2012-2013

Year 2 2013-2014

Year 3 2014-2015

Year 4 2015-2016

Year 5 2016-2017

Year 6 2017-2018

Year 7 2018-2019

III. Implementing Agency Regional Implementation Teams for each AEA and DMPS

L3Q9: How do partnerships on the regional implementation teams change over the project period? S7, L7 L3Q10: What types of network/ collaboration changes occur? S7, L7 L3Q11: How and when do they begin to identify "others" who need to be part of their implementation team? S7, L7

Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006). Summative

Online survey collected from each regional implementation team member annually beginning Summer 2014 Collaboration maps from survey results By external evaluator

Summer 2014 Fall 2015 Fall 2016 Fall 2017 Fall 2018 Fall 2019

L3Q12: How do implementation team members shift in their understanding of Active Implementation Frameworks? S8, S9, S17, I6, I7, I14, I15, L4, L10

Regional Imp Team (RIT) updates SWT agenda/notes EAGL conversations Formative

RIT Update documents SWT meeting notes EAGL meeting notes By DE

Fall 2016 Fall 2017 Fall 2018 Fall 2019

L3Q12.1: How do implementation team members shift in their use of Active Implementation Frameworks? S9, S12, S17, I6, I7, I14, I15, L4, L8, L9, L10

Regional Imp Team (RIT) updates SWT agenda/notes EAGL conversations Formative

RIT Update documents SWT meeting notes EAGL meeting notes By DE

Fall 2017 Fall 2018 Fall 2019

L3Q13: How did systems change to accommodate this initiative? (same as L3Q14) L3Q14: What are the barriers that impact agencies/service providers in supporting this model; what policies might impact usage of this model? S8, S11, S16, I6, I7, I8, I13, I14, L5, L6, L8, L9, L10, L11

Focus group and interviews Formative

Focus group with the regional implementation team with phone interviews with members unable to participate-to be completed in project years 3 and 5 By DE

Fall 2017 Fall 2018 Fall 2019






Year 1 2012-2013

Year 2 2013-2014

Year 3 2014-2015

Year 4 2015-2016

Year 5 2016-2017

Year 6 2017-2018

Year 7 2018-2019

IV. State State Implementation Team

L4Q17: How do state implementation team members shift in their understanding of Active Implementation Frameworks? S8, S9, S17, I6, I7, I14, I15, L4, L9, L10

Regional & State Imp Team reports Implementation Webinars Formative & Summative

RIT & SIT Update documents State Work Team agendas & notes EAGL agendas & notes Webinar participation lists By DE


L4Q17.1: How do implementation team members shift in their use of Active Implementation Frameworks? S9, S12, S17, I6, I7, I14, I15, L4, L8, L10

Regional & State Imp Team updates State Work Team meetings EAGL meetings Formative & Summative

RIT & SIT Update documents State Work Team agendas & notes EAGL agendas & notes By DE


L4Q18a: How did systems change to accommodate this initiative?

Governance (GV) Personnel/workforce (PN) Other components

S11, S13-19, I9-I16, L5-L11

ECTA Self-Assessments Formative & Summative

Excel self-assessment forms collected annually for GV, PN By DE Fall 2015 Fall 2016 Fall 2017 Fall 2018 Fall 2019

L4Q18b: How did systems change to accommodate this initiative?

Governance Personnel/workforce Other components

S11, S13-S19, I9-I16, L5-L11

Imp Team and State Work Team EAGL, ICEA meetings Formative & Summative

Agenda and note documents from each group. By DE Fall 2017 Fall 2018 Fall 2019

L4Q19: What is working? What are the positive impacts? What, if any, barriers? What would you do differently? S16, S17, I10, I11-I14, L4, L6, L10, L11

Interviews Formative

Phone interview with DE Early ACCESS state coordinator and CSPD consultant By external evaluator

Winter 2013 Spring 2014 Winter 2014 Winter 2015 Winter 2016 Winter 2017 Winter 2018

L4Q20: How are state IHEs impacted by this initiative? I10, I11, I12, L6, L7, L10, L11

Interviews Summative

Phone interviews with colleges and universities with pre-service programs that prepare students to be work in the field of early intervention By external evaluator

Spring 2017 Spring 2018 Spring 2019

Red text = Outcome # from Logic Models L4Q15 and L4Q16 deleted. Discontinued using Level of Collaboration Scale for State level evaluation Fall 2015.





Year 1 2012-2013

Year 2 2013-2014

Year 3 2014-2015

Year 4 2015-2016

Year 5 2016-2017

Year 6 2017-2018

Year 7 2018-2019

IV. State State Implementation Team (continued)

L4Q21: What lessons learned can we share with other entities and states wanting to develop a focused PD initiative? L4, L6, L7, L10, L11

Surveys, interviews, focus groups previously listed in the plan Formative & Summative

Case study documentation using results of data collected over complete project period By external evaluator

Fall 2017 Fall 2019

References Frey, B., Lohmeier, J.H., Lee, S.W. & Tollefson, N. (2006). Measuring collaboration among grant partners. American Journal of Evaluation, 27(3). 383-392 Guimond, A.B., Wilcox, M.J. & Lamorey, S.G. (2008) The Early Intervention Parenting Self-Efficacy Scale (EIPSES). Journal of Early Intervention, 30(4), 295-320 Preskill, H. & Torres, R. T. (1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.
