Measuring effects in the Nordic context:

Transferability of international programmes and the importance of local evaluation

Knut Sundell

Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU)

Photograph by The Sydney Justice & Police Museum

Implications

• We need to evaluate the effects of interventions

• We can evaluate with rigorous designs

• Culture/context probably moderate effects

• A well designed trial always teach us something

• Effectiveness is necessary – but not enough

Photograph by The Sydney Justice & Police Museum

Safe

Accessible

Effective

Cost

efficient

Acceptable

Equal

Quality

Photograph by The Sydney Justice & Police Museum

Safe ?

18 Motivating interview

18 ART

8 Repulse

3 12-stegsbehandling

3 Family therapy (ospecified)

2 Dialektisk beteendeterapi

2 Kriminalitet som livsval

2 Marte meo

1 TEACCH (Treatment of Education of Autistic and related communiction-handicapped Children)

1 Tejping

<1 Active parenting

Akupunktur

Antroposofisk behandling

Avslappning

Bekymringssamtal

Bildanalys

Bildterapi

CAP (Children Are People Too)

COPE (Community Parent Education program)

Community Reinforcement Training

Depression in swedish adolescents (DISA)

EMDR (Eye Movement Desensitisation and Reprocessing)

Funktionell familjeterapi

Gestaltterapi

Haschavvänjningsprogram

Hästterapi

Vägledande samtal

Komet

Lekterapi

Livsbok

Läkepedagogik

Lösninginriktad pedagogik

Massage

Mindfulness

Multisystemisk Miljöterapi

Parenting Young Children

Problembaserad skolutveckling

PMTO (Parent Management Training)

Psykodrama

Robo-memo

Rollspel

Trappan

Traumabehandling

Transaktionsanalys

Trestegsmodell

Youth at risk program (YAR)

Interventions in 250 institutions for children and youth (%)

Arbetssätt på HVB för barn och unga (Knut Sundell och Malin Hultman, Inspektionen för vård och omsorg, 2013-11-19)

Negative effects might be because:

• Harmful intervention (e.g., Lilienfeld, 2007)

• Ineffectiv intervention (e.g., psychodynamic therapy for treatment of autism)

• Malpractice – the intervention is not used appropriately (e.g., poor fidelity, low competence)

• Contraindikation – the interventions was used for the wrong population (Lowenkamp & Latessa, 2006)

Can talk harm?

Lilienfeld, S. O. (2007). Psychological treatments that cause harm. Perspectives on psychological science, 2, 53-70.

Michael Linden (2012). How to Define, Find and Classify Side Effects in Psychotherapy: From Unwanted Events to Adverse

Treatment Reactions. Clinical Psychology and Psychotherapy. DOI: 10.1002/cpp.1765

Debriefing for Road Traffic Accident Victims

Swedish research

Internet-based CBT for social anxiety (Boettcher et al, 2014)

• 14 % reported of unwanted effects (new symptoms, increased anxiety, et cetera)

• 7 % had enhanced problems between pre and post measurement

”Open survey” (Rozenthal & Carlbring, in prep)

• During one month 464 individuals could reported negative effects of psychoterapy (e.g., ”I became more anxious”; ”I didn’t understand the treatment”; ”My expectations was not met”)

• Longer therapies was associated with more negative effects

-40 -20 0 20 40 60 80

The Effectiveness of Correctional Rehabilitation: A Review of Systematic Reviews (Lipsey & Cullen, 2007)

Behavioral treatment (risk & needs) ……….. Family and parenting ……..……….………… Multidimensional treatment foster care …..

Interpersonal skills ………………………………

Individual counseling .…………………………

Victim-offender mediation .…….……………

Life skills ………...…………………………………

Multisystemic therapy …………………………

Aggression Replacement Training …….……

Victim-offender mediation ……..……………

Challenge programs ………………..…………

Milieu therapy ……………………..……………

Community supervision …….…………………

Group counseling .………………..……………

Psychodynamic therapy …..…………………

Vocational and employment programs …..

Boot camps ……………………….……….

Incarcaration, prison ….……………….

Scared straight …………………

Change in

recidivism %

Decreased risk Increased risk

Photograph by The Sydney Justice & Police Museum

The Swedish

MST-trial

The Swedish Trial

Started in 2004

MST vs Treatment-As-Usual (50/50)

Randomized controlled trial

27 local authorities / 6 MST-teams

Low attrition = 6 % (24 months)

Intention-to-treat (randomization intact)

Explicit inclusion criteria (Conduct Disorder)

Multiagent / multimethod assessment

Study of ”real life” (effectiveness)

No financial or other interests in MST

Participants

Conduct disordered youth

12 – 17 years

Exclusion criteria:

No motivated guardian

Sexual offending

Autism, acute psychosis, risk of suicide

The presence of the youth in the home posed a serious risk to the youth or to the

family

MST

Treatment As Usual

Pre- measure

7 month

follow-up

24 month

follow-up 60 month

follow-up

Rando-

misation Recruit-

ment

168 156

(93%)

149

(96%)

147

(94%)

155

(99%)

Child Behavior Checklist / Youth Self-Report (Achenbach,

1991a; 1991b)

Sense of Coherence (Antonovsky, 1987)

Self-Report Delinquency Scale (Elliott et al., 1983)

Alcohol Use Disorder Identification Test (Babor et al., 1992)

Drug Use Disorder Identification Test (Berman et al., 2005)

Pittsburgh Youth Study (Keenan et al., 995)

Social Competence with Peers Questionnaire (Spence,

1995)

Social Skills Ratings System (Gresham & Elliott, 1990)

Symptoms Checklist-90 (Derogatis & Cleary, 1997)

Parenting skills (Stattin,2004)

School attendance

Social services (case file review)

Police reports

Measures

0

20

40

60

80

0 7 24

MST

Trad

0

20

40

60

80

0 7 24

Self-reported

(YSR)

Parent

(CBCL)

Mental Health Problems

Antisocial behaviors (SRD)

0

0,5

1

1,5

2

0 7 24

MST

TAU

Arrested during the last 6 months (parent reports)

0

20

40

60

80

100

0 7 24

%

MST

TAU

0

2

4

6

8

10

0 7 24

MST

TAU

0

2

4

6

8

10

0 7 24

Drugs

(Dudit score)

Alcohol

(Audit score)

Substance abuse (scores)

0

0,5

1

1,5

2

0 7 24

MST

Trad

Mothers’ mental health (SCL-90)

Days in out-of-home care (Cumulative %)

0

20

40

60

80

100

0 7 24

%

MST

TAU

Days with services (Cumulative)

0

100

200

300

400

500

600

700

0 7 24

days

MST

TAU

Multisystemic Therapy (Sweden, Canada)

PATHS (Switzerland)

Multidimensional treatment Foster Care (UK)

Triple-P (Canada, Switzerland)

Strengthening Families Program 10-14 (Sweden)

Nurse Home Visiting (Germany)

Effekt (Örebro Prevention Program) (the Netherlands)

Multisystemic Therapy (Norway, the Netherlands, UK)

PATHS (UK)

Multidimensional treatment Foster Care (Sweden)

Triple-P (Germany, Hong Kong, Japan, Switzerland, USA)

Functional Family therapy (Sweden)

Incredible Years (UK, Canada, Norway; Sweden)

International RCTs of some family interventions on the Blueprint list

Photograph by The Sydney Justice & Police Museum

Why?

Possible reasons for

contradictory results?

1. Variations in research methodology

2. Poor cultural adaptation

3. Poor treatment adherence

4. Context moderate the effects

5. Conflict of interest bias

Sundell, K., Ferrer-Wreder, L., & Fraser, M. W. (2013). Going Global: A Model for Evaluating Empirically-Supported Family-Based Interventions in New Contexts. Evaluation & the Health Professions. Published online before print January 4, 2013, doi: 10.1177/0163278712469813

1 - Research methodology

Non-randomized trials produce lager d’s (e.g., Kunz et al., 2008)

Efficacy trials produce lager d’s (e.g., Curtis, et al., 2004; Emshoff, 2008;)

Passive control group produce lager d’s (e.g., Magill & Ray, 2009; Baldwin, et al., 2012)

“Treatment of the treated” approach to attrition produce lager d’s (e.g., Kunz, Vist & Oxman, 2008)

Shorter follow-up time, tighter inclusion criteria & measures on delinquency and substance abuse measures produce lager d’s

1 - Research methodology

A review of 13 MST RCTs (Andrée Löfholm et al., 2013)

• No two studies used the same research methodology

• The TAU condition included a wide variety of treatment alternatives

• A significant heterogeneity in effect sizes (83% of the variation was attributable to heterogeneity between studies)

A review of 24 RCTs of MST, FFT, BSFT, MFT (Baldwin et al., 2012)

• A significant heterogeneity in effect sizes

• Significant effects vs. TAU d = .21

vs. alternative treatment d =.26

vs. no active interventions d = .70

Interventions in the control group (n = 77)

17 Family therapy

14 Counseling youth

6 Counseling parent

12 Mentor

4 Institution care

6 Foster care

4 Substance abuse treatment

3 Aggression Replacement Training

2 Unspecified support youth

1 Structured open care treatment

1 Special education

7 Nothing

2 - Poor cultural adaptation

Distinction deep vs. surface structure (Resnicow et al., 2000)

• Deep structure – causal model, “theory of change”, mediators, risk factors

• Surface structure –make an EBI in tune with culturally linked aspects of participants’ life experience (e.g., program materials)

The issue is not weather to adapt a program or not – it is how to control the adaptation taking place (Resnicow et al., 2000)

Few program developers specify the core components of the intervention

3 - Poor treatment adherence

Data from the MST Institute

March 2003 to August 2009

2 948 TAMs / 973 families (87%) from 68 therapists in 10 teams

19 TAM items + post treatment youth outcomes as reported to MST Institute

Multilevel structural equation modeling

Andrée Löfholm, C., Eichas, K., & Sundell, K. (2014). The Swedish implementation of Multisystemic therapy for adolescents: does treatment

experience predict treatment adherence? Journal of Clinical Child & Adolescent Psychology. Published online before print March 20, 2014,

http://dx.doi.org/10.1080/15374416.2014.883926

70

75

80

85

90

95

1 2 3 4 5 6

2003 (7 team)

2005-6 (3 team)

TAM

Years

Treatment fidelity MST

Andrée Löfholm, C., Eichas, K., & Sundell, K. (2014). The Swedish implementation of Multisystemic therapy for adolescents: does treatment experience predict treatment adherence? Journal of Clinical Child & Adolescent Psychology, 43, 643-655.

87%

91%

94% 95%

60%

70%

80%

90%

100%

65 75 85 95

Po

st-T

rea

tme

nt

TAM

Living at Home

71%

76%

80%

84%

60%

70%

80%

90%

100%

65 75 85 95

Po

st-T

rea

tme

nt

TAM

In School or Work

26%

22%

19%

16%

0%

10%

20%

30%

40%

65 75 85 95

Po

st-T

rea

tme

nt

TAM

New Arrests-Male

13%

11% 9%

7%

0%

10%

20%

30%

40%

65 75 85 95

Po

st-T

rea

tme

nt

TAM

New Arrests-Female

Aggression Replacement Training in Sweden (Catrine Kaunitz och Anna Strandberg, 2009)

According to the manual:

– 3 modules: social skills, anger control, moral reasoning

– 3 lessons x 10 weeks (at least 20 lessons)

– Roll play, reinforcement and homework

Prevalence study 2006 + 2012

Repeated study of random sample of local authorities (N=102)

Aggression replacement Training (Kaunitz & Strandberg, 2009; Kaunitz in prep)

65%

88%

58%

25% 22%

50% 89%

39% 13% 9%

0

20

40

60

80

100

ART Role

play

etc

3

moduls

20+

lessons

All 3 ART Role

play

etc

3

moduls

20+

lessons

All 3

2006 2012

Kaunitz, C.& Strandberg, A. (2009). Aggression Replacement Training (ART) i Sverige- evidensbaserad socialtjänst i praktiken. Socionomens forskningssupplement, 26, 36-52. Kaunitz (in prep).

4 - Context moderates the effects

Context can be conceptualized in a variety of ways (e.g., Sussman et al., 2008)

The exact effects of cultural differences is widely acknowledged and poorly understood

Possible examples: • Prevalence of various risk factors (e.g., drug use, powerty)

• National policies on social services

Mental health (CBCL) change T-scores in

evaluations of MST (6-12 month follow-up)

0

4

8

12

Sundell et al

(2008)

Ogden &

Halliday-

Boykins

(2004)

Henggeler et

al (2006)

Henggeler et

al (2006) -

cont man

Rowland et al

(2005) -

external

Rowland et al

(2005) -

internal

MST TAU

Cannabis use (life-time)

males aged 15 (ter Bogt et al, 2006)

0

10

20

30

40

50

60

70

80

%

Cannabis use (life-time)

females aged 15 (ter Bogt et al, 2006)

0

10

20

30

40

50

60

70

80

%

Risk factors for cannabis use (von Sydow et al, 2002)

Availability of drugs

Peers’ drug use

5 - Conflict of Interest Bias

Conflict of interest is a threat to the validity of findings in evaluation research

The problem has increased because of a growth in international prevention firms

The label “evidence-based” is a major sales factor (but not protected)

This undermines the goal of providing policy-makers with the best evidence of what works

Eisner, M. (2009). No effects in independent prevention Trials: Can we reject the cynical view. Journal of Experimental Criminology, 5(2).

Advert or Research?

Advert or Research?

A Flourishing Prevention Industry

Triple P:

– Sold in 24 countries.

– 62,000 providers trained

USD 2000/provider training > 120 Mio (estimate)

– 6 million parent training courses sold

USD 250/parent training seminar > 1 500 Mio (estimate)

http://www.brighton-hove.gov.uk/downloads/bhcc/children/Matt_Sanders_Presentation.pdf

Effect Size by Financial COI Result

The findings suggest a small

nonsignificant effect for studies

without financial COI and a

large significant effect for

studies with financial COI

Eisner, M. P., & Humphreys, D. (2012). Measuring conflict of Interest in prevention and intervention research: A feasibility sudy. In T. Bliesener, A. Beelmann & M. Stemmler (Eds.), Antisocial behavior and crime: Contributions of developmental and evaluation research to prevention and intervention (pp. 165-180). Cambridge: Hogrefe.

Photograph by The Sydney Justice & Police Museum

Are interventions transportable?

• 4 interventions for reducing behavioural problems in children aged 3-10 (Incredible Years; Parent-Child Interaction Therapy; PMTO, Triple P)

• 14 RCTs + 3 non-randomised transported trials

• Tested in 10 countries in 5 regions

• Cohen’s d = .71 (comparable to those in the origin country)

• No association effect size x implementation factors, country-level policy or resource factors

• Stronger effect sizes in countries with more traditional values about family life and childrearing (Hong Kong, Costa Rica and Iran).

Gardner, F., Montgomery. P., & Knerr, E´W. (2015). Transporting Evidence-Based Parenting Programs for Child Problem Behavior (Age 3–10) Between Countries: Systematic Review

and Meta-Analysis. Journal of Clinical Child & Adolescent Psychology. DOI:10.1080/15374416.2015.1015134

• RCT: 4 interventions for reducing behavioural problems in children aged 3-12 (Incredible years, Cope, Connect,

Komet) vs. book on parent training / waiting list)

• 6 regions and 908 children (95% retrieved at 4 months)

• Self-referral

• Self-reporting (Eyeberg, SNAP-IV, PSOC, ARC, CGSQ, CES-D...)

• Acceptable fidelity

• All interventions were effective compared to the waiting list

Stattin, H., Enebrink, P., Özdemir, M., & Giannotta , F. (2015). A National Evaluation of Parenting Programs in Sweden: The Short-Term Effects Using an RCT Effectiveness Design. Journal of Consulting and Clinical Psychology. doi.org/10.1037/a0039328

All interventions were effective compared to the waitlist control

Photograph by The Sydney Justice & Police Museum

Effective compared to what?

116

17 25 0

25

50

75

100

125

150

175

Germany (N = 158)

96

32 11 0

25

50

75

100

125

150

175

Sweden (N = 139)

34% RCT

80% universal prevention

90% passive controls

100% child and youth

interventions

Based on the average

means of all outcomes in

the articles

89% RCT

12% universal prevention

30% passive controls

14% child and youth

interventions

Based on the primary

outcomes

Sundell, K., Beelmann, A., Hassan, H. & von Thiele Schwarz, U. (2015). Novel programs, international adoptions, or contextual adaptations? Meta-analytical results from

German and Swedish intervention research. Journal of Clinical Child & Adolescent Psychology. DOI: 10.1080/15374416.2015.1020540

Coding and analysis

Inter-coder agreement (3 broad categories) = 96.2% (German) and 92.8% (Swedish sample)

Calculation of effect sizes according to Lipsey and Wilson (2000)

Meta-analysis according to Hedges and Olkin (1985)

Because of significant heterogeneity, calculations = Random Effect Model

Controlled for study design (only RCT) and sample size (50+)

,22 ,10 ,16 ,19

0

0.25

0.5

0.75

1

Germany (N = 119)

.45

,27

,44 ,41

0

0.25

0.5

0.75

1

Sweden (N = 94)

Cohen’s d (Controlled for design and sample size)

Note – all groups were significant compared to

control

Tentative conclusion

Transported programs are

effective but should not be

implemented without

considering adaptation.

Photograph by The Sydney Justice & Police Museum

Can we trust controlled

research?

Publiced in a peer-review journal 1990-2014

Randomised or non-randomised with control group

Pre- and post measurement

Psychological or social intervention

Effects measured on client

Trends in methodological quality in Swedish controlled trials of psychological and social

interventions

Sundell, K., & Åhsberg, E. (in press). Trends in methodological quality in Swedish controlled trials of psychological and social interventions. Research on Social Work Practice

0

5

10

15

20

25

30

35

40

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014

Trends in methodological quality in

Swedish trials (N = 302)

Sundell, K., & Åhsberg, E. (in press). Trends in methodological quality in Swedish controlled trials of psychological and social interventions. Research on Social Work Practice

140

90

25 14 11 11

5 6 0

20

40

60

80

100

120

140

160

Target area (N = 302)

0

20

40

60

80

100

90-99 00-04 05-09 10-14

Randomised (%)

0

20

40

60

80

100

90-99 00-04 05-09 10-14

Intent-to-treat (%)

0

20

40

60

80

100

90-99 00-04 05-09 10-14

Dropouts presented (%)

0

20

40

60

80

100

90-99 00-04 05-09 10-14

Significant effect (%)

4.33 6.04

6.98

9.09

0

4

8

12

16

1990-99 2000-04 2005-09 2010-14

Publication year (N=302)

Methodological quality index

Conclusions

Increased methodological quality, but still

not good-enough (single outcome trials

may be biased)

Only 51% of the interventions were effective

(reasonable theory is not enough)

RCTs was the standard – 85% (is doable and

should be used)

Implications

• We need to evaluate the effects of interventions

• We can evaluate with rigorous designs

• Culture/context probably moderate effects

• A well designed trial always teach us something

• Effectiveness is necessary – but not enough

Photograph by The Sydney Justice & Police Museum

Safe

Accessible

Effective

Cost

efficient

Acceptable

Equal

Quality