1 what works in the real world? evaluating social-emotional programmes, paths, and incredible years...
TRANSCRIPT
1
What works in the real world? Evaluating social-emotional
programmes, PATHS, and Incredible Years
Louise Morpeth &Tracey BywaterInaugural IEE Conference: 30.09.2011
Social Research Unit at Dartington
What is ‘real world’ evaluation?
Efficacy trials – does it work under optimal conditions?
Effectiveness trials – does it work under real world conditions?
Dissemination trials – what happens when the programme is ‘taken to scale?
What is PATHS?
A school based programme for primary schools age children that facilitates the development of self control, emotional awareness and interpersonal problem solving skills.
It seeks to reduce aggression and behaviour problems, and enhance emotional development and the educational process.
PATHS lessons of 20-30 minutes are taught 3-5 times a week throughout primary school.
Detailed lesson plans with teaching instruction are supplemented with materials (feelings cards, puppets etc)
Training is provided to all teachers and head teachers. Fidelity is maintained by a PATHS coach.
PATHSPromoting Alternative Thinking Strategies
How do Social and Emotional Learning programmes work?
From CASEL (www.casel.og)
Evaluation questions
•Does PATHS meet the needs of and improve outcomes for the children that participate?
•Are programmes implemented efficiently and effectively with fidelity?
•Do programmes offer value for money and net benefit to the local authority, national government, society and participants?
Evaluation design
PATHS (60 schools) • cluster randomised trial• stratified by size & % FSM• 1:1, Reception & Year 1 classes•3 data main collection points (Oct - June 2010, June 2011)A wait-list, intention to treat, randomised controlled trial
PATHS measures SDQ – child behaviourPaths Teacher Rating Scale - child behaviour Teacher background & school climate Attendance records – teacher & pupils Academic gradesObservations in class (Oct & April both academic years)
Preliminary results show…..
A great deal has been written about lookedafter children, or children in state care. Muchrecent legislation A great deal has been written about looked after children, or children in state care. Muchrecent legislation.
•1 million people•30% Black, Minorities, and Ethnic•23% under 16 yrs of age•30% single parents
Birmingham
London
context
•62 children’s centers•60,000 professionals working with children and families
A great deal has been written about lookedafter children, or children in state care. Muchrecent legislation A great deal has been written about looked after children, or children in state care. Muchrecent legislation.
•1 million people•30% Black, Minorities, and Ethnic•23% under 16 yrs of age•30% single parents
Birmingham
London
context
•62 children’s centers•60,000 professionals working with children and families
No longer pushing PATHS, there is a pull coming from schools
Implementation extended from 27 - >60 schools out of 299
Working on a sustainable costing model (£6000 per school), while maintaining fidelity
Next stepsFind image of anxiety – nail biting
Parenting Overview
1. What is parenting and what are the benefits?2. What does a parent programme look like, and what
does it do?3. Do parent programmes work?4. What are the issues around recruitment and
programme implementation?5. What are the associated costs of the programme? 6. Is there a benefit to link parent programmes to
school/early learning environments?
12
1. What is parenting and what are the benefits?
... the process of promoting and supporting the physical, emotional, social, and intellectual development of a child from infancy to adulthood. Parenting refers to the activity of raising a child rather than the biological relationship.
Davies, Martin (2000). The Blackwell Encyclopaedia of Social Work. Wiley-Blackwell. p. 245. ISBN 9780631214519.
13
14
Parenting factors•Harsh and ineffective•Poor monitoring•Low stimulation
Child factors•Poor conflict control•Impulsivity, ADHD•Learning delays•Poor social skills
Contextual factors•Poverty•Parental crime, substanceabuse or mental illness•Life stressors
School factors•Classroom aggression•Deviant peers•Poor teacher response•Peer rejection
ConductProblems
Most influential & easiest to change
30% (£22.5bn)
50% (£37.5bn)
20% £15bn)
Crime (& annual cost) attributable to people who had conduct problems in
childhood and adolescence in UK
Conduct disorderOther conduct prolemsNo conduct problems
15
Sainsbury Centre for Mental Health, 2009The chance of a lifetime: Preventing early conduct problems and reducing crime
2. What does a parent programme look like, and what does it do?
• 1 to 1, practitioner & parent • DIY, parent is given material and learns at own
pace• Group format – typically 4-18 2hr weekly sessions,
12 parents• Targeted or universal, for specific group or open to
everybody• Has structured content, with a manual and
handouts
16
Components of effective programmes
•New parenting skills must be modelled and rehearsed
•Home-based practice or ‘homework’•Parenting programmes should be collaborative
& emphasise principles rather than prescribe techniques
•Non-violent sanctions for negative behaviour•Relationship building, praise, fun & play,
reinforcement
17
Incredible Years suite of programmes
This one has been trialled in Wales, Ireland & England in RCTs
19
3. Do parent programmes work?
1. Wales - Sure Start Trial• Delivered by Sure Start staff• Children 3-4 years at risk of conduct disorder2. Ireland - National Evaluation of Incredible Years• Delivered by Archways staff in disadvantaged areas• Children 5-8 years at risk of conduct disorder3. England – part of Birmingham’s Brighter Futures
Initiative• Delivered across the city by children centre and local service
staff• Children 3-4 years at risk of conduct disorder
20
Measures used in the trials About the family:
1. Demographic questionnaire
2. Service Utilisation Questionnaire (Chisholm et al., 2000)
3. Dyadic Parent-Child Interaction Coding System (DPICS; Eyberg & Robinson, 2000)
About the child:
1. Eyberg Child Behavior Inventory (ECBI; Eyberg & Ross, 1978)
2. Strengths & Difficulties Questionnaire (SDQ; Goodman, 1997)
3. Conners Parent Rating Scale (Conners, 1994)
4. Kendall Self Control Rating Scale (SCRS; Kendall & Wilcox, 1979)
5. Social Competence Scale (Fast Track Project)
About the parent:
1. Beck Depression Inventory (BDI-II; Beck et al., 1961, 1996)
2. Parenting Stress Index (PSI; Abidin, 1990)
3. EQ-5D Health-Related Quality of Life Questionnaire (Kind et al.,1995)
4. The Parenting Scale (Arnold, O’Leary, Wolff & Acker, 1993)
21
Child Behaviour – parent report (ECBI)
22
Programme delivery between baseline & 6 months
4. What are the issues around recruitment and programme
implementation?
“Success is a science: If you have the conditions you get the results” Oscar
Wilde (1854-1900)PROCESS EVALUATION:
• Was engagement, recruitment and retention successful?
• Is the programme implemented with fidelity in all fidelity categories?
23
Parent engagement
24
Referral method Number referrals
Eligible(& could be contacted)
Participated(from contactable, eligible, consented)
Wales (3-4yrs) Health visitors 240 74% 93%
Ireland (5-8yrs)
Health service waiting lists, schools, community-based agencies, self-referrals
233 64% 100%
England (3-4yrs)
Initially children centres, then nursery nurse, family support, schools, social workers
299 (early 2010)
17-40% variable across 6 areas
Approx 80%
Key Themes linked to participant engagement:
25
1. Organisation readiness2. Target group & accuracy of demand
analysis3. Recruitment procedures 4. Publicity & referral materials5. Context of an RCT6. Staff workload7. Cultural context
26
Delivery & Implementation Fidelity
• is the programme delivered as designed?• are all the core components present?• to the right population?• with appropriately trained staff?• using the right protocols, techniques &
materials? • in the right context?
27
28
5. What are the associated costs of the programme?
Or - do the programmes offer value for money?
•What are the delivery costs?•Is the programme cost effective?•What are the longer term costs & benefits?
29
Micro-costing exercises establish delivery costs•Wales & Ireland studies: £12-1300 per child for 12 children, (approx. £14,500 including facilitator training, and staff time)
Cost-effectiveness• Wales & Ireland: £70-80 per point improvement on the ECBI
Cost benefits• Ireland: Cost and benefit data yields an internal rate of return for the programme of 11% (assuming that individuals with conduct problems spend an additional 5 months in unemployment, and that the savings from reductions in imprisonment occur only once and at the age of 30)
30
6. Is there a benefit to link parent programmes to school/early learning environments?
• Early Years Foundation Stage• Nursery staff trained in IY parent programmes can
show less stress, with improved children behaviour• Parent programmes delivered in schools - IY school
readiness• FAST – Families & Schools Together • SPOKES - Supporting Parents on Kids Education in
Schools (Helping Children Achieve) combines a parent & a literacy programme
• School based social/emotional programmes e.g. IY Dina increases parent input to child’s schooling
Lessons Learned to inform policy & practice
1. Know what outcomes/change are required2. Select an evidence-based programme for the
target population3. Develop a strategy for recruiting the target
population4. Address relevant service access issues 5. Staff training and ongoing support6. Reference/stakeholder groups/steering
committee7. Ensure implementation fidelity (process &
delivery)8. Evaluate programme delivery, costs, and
outcomes LT (to be embedded within services)
31
Thank you for your participation
www.dartington.org.uk
www.york.ac.uk/iee
32