Training is Not Enough:Intervention and Evaluation

Supplements

Christina J. Groark and Robert B.

McCallUniversity of Pittsburgh

Office of Child Development

Overview—Intervention

• Training alone is a minimally effective intervention

• Need:−Hands-on technical assistance in

situ−Supportive work environment−Continuous monitoring, positive

supervision

• Assessing trainee’s satisfaction and

learning is a minimum evaluation

• Need:−Changes in trainee’s on-the-job behavior−Changes in the behavior/outcomes of

clients of the trainees−Analyses to show trainees behavior

change mediated the intervention's effect on clients’ behavior/outcomes

Overview—Evaluation

Intervention

Training alone is minimally

effectiveWhy? Trainees often do not change

their behavior

Need:• On-the-ground technical assistance• Supportive work environment• Monitoring and positive supervision

Intervention

On-the-ground technical

assistance

•Observe (videotape) client on job•Constructive, positive feedback•Modeling with client on job•Daily review

Intervention

Intervention

Work environment

•Supports behaviors taught in training•Hand washing—need a sink, waterless soap handy•Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers

Chaotic Child Environment

Orderly Child Environment

9

Monitoring and positive

supervision•Sets expectations, standards, reminds, rewards•Continuing system, supervisor, staff meetings

•Hand washing•Early care and education—teachable moments, responsiveness to child-directed initiatives

Illustrative Example ILatin American Orphanage Intervention (birth-6 years)•Training—sensitive, responsive interactions

◦ 6 one-day training sessions

•On-the-ground technical

assistance◦ 12 days more than planned

because caregivers were not implementing

BUT –

• Limited environmental change

◦ 4 primary caregivers per ward

◦ 8-10 children per caregiver

• No continuing supervision

1

5

7

Intervention

Pre- Post-

Results—Caregiving behavior/environment (ITERS/ECERS)

2

3

4

Younge

r

Older

6

Typical USA Early Care

60

70

80

Intervention

Pre- Post-

Results—Children’s development (Battelle)

Young

er

Older

90

100Typical Parent-Reared

Conclusion

• Needed hands-on technical assistance to promote implementation

• Limited supportive work environment

• No supervisory system• Some improvement in caregivers,

children, but NOT MUCH

Illustrative Example IISt. Petersburg (Russia) Orphanage Intervention

•More intensive, comprehensive

•All major components

• Minimum warm, sensitive, contingently-responsive caregiver-child interactions

• Perfunctory, business-like; little talking

• Caregiver directed

• Conformity

04/21/23 18

St. Petersburg-USA Orphanage Research Team, 2005, 2008

• 12-14 children/ward

• 9-12 caregivers/week

• Homogeneous age, disability groups

• Periodic graduations

• 60-100 caregivers before age 19

months

04/21/23 20

St. Petersburg-USA Orphanage Research Team, 2005, 2008

Intervention

•Training◦ Train Trainers◦ 12 sessions, 2-3 hrs.◦ Warm, sensitive, responsive◦ “Love these kids…”

•Hands-on technical assistance

Intervention—Supportive Work Environment

• Reduced group size to 6-7• Assigned primary, secondary

caregivers—6 vs 9 cgrs.• Changed work schedules• Integrated groups by age,

disabilities• Eliminated periodic graduations to

new groups• Family hour

Intervention—Monitoringand Positive Supervision

• Training for supervisors• Trainer observed, coached both

caregivers and supervisors• Supervisors met periodically to

problem solve, case reviews, mutual support

Three orphanages (Baby Homes), birth – 4 years

• T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision

• TO—Training Only

• NoI—No Intervention

Evaluation

Evaluation—Trainee Learning

Longitudinal Sample

Evaluation—Improved Caregiving

T+SC

Evaluation – Children’s Development

40

50

60

70

80

90

First 4-9 mos 9+ mos

Time in Intervention

Dev

elo

pm

enta

l Qu

oti

ent T+SC

TO

NoI

100

Typical Parent Reared

Evaluation--Mediation Analysis

• Intervention improved caregiving• Intervention improved children’s

development

• But was children’s

developmental improvement

associated with improved

caregiving?

Evaluation—Mediation Analysis

Evaluation--Mediation Analysis

Independent Variable T+SC vs

NoIIntervention

Outcome Battelle

Total EffectĈ =

14.14(2.07)***

MediatorHOME

Independent Variable T+SC vs

NoIIntervention

OutcomeBattelle

Effect of Mediator on Outcome Controlling for Indep. Variable

Direct effect Controlling for Mediator

Ĉ = 8.00(2.87)***

Mediator accounts for 43% of Total Effect

Effect of Indep. Var. on Mediator= 3.41(.22)*** = 1.80

(.59)**

• Intervention◦Training alone is often minimally

effective◦Has greater effect if accompanied by

On-the-ground technical assistance Supportive work environment Monitoring and supervision

Conclusion

• Evaluation is improved if◦Some training vs. No Training comparison

◦Measures of–Learning–Work-place behavior change–Outcome behavior of clients

◦Mediation analysis

Conclusion

Evaluation Mediation Analysis:MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2), 144-158.