Daniel J. Bowen

Registered Psychologist

PhD Candidatewww.danielbowen.com.au/

Dr James T. Neill

Associate Professorwww.wilderdom.com/

A Meta-Analysis of Adventure Therapy Outcomes and Moderators

University of Canberra

AustraliaBowen, D. J., & Neill, J. T. (2013). A meta-analysis of

adventure therapy outcomes and moderators.

The Open Psychology Journal, 6, 28-53.

doi: 10.2174/1874350120130802001

July, 2016

Meta-Analysis

A meta-analysis is:

“a set of statistical methods

for combining quantitative results

from multiple studies

to produce an overall summary

of empirical knowledge

on a given topic”

(Littell, Corcoran & Pillai, 2008, p. 1-2)

Effect Sizes (ES)

Effect size (ES) (Ellis, 2010):

= Magnitude of the treatment effect

Calculated for each study

Combined to compute a summary effect

Effect Size Statistic Used: Hedges’ g

= Difference between the means of 2 groups ÷ population standard

deviation: (M1-M2)/Sdpooled

Adjusts for studies with small sample sizes

Interpreting: 0.20 (small), 0.50 (medium), 0.80 (large)

Effect Size Confidence Intervals (CIs)

Confidence Interval (CI) (Ellis, 2010):

A range of values that describes the uncertainty surrounding an estimate

(e.g., of an effect size (ES))

Indicates a range of values that's likely to encompass the true value

Is also itself an estimate

– The two numbers that make up the lower and upper ends of the CI are called the

lower and upper confidence limits. The lower and upper figures provides an

indication of the expected or possible range surrounding the true value.

95% CIs (used in this meta-analysis) contain the true value 95% of the time

and fail to contain the true value the other 5% of the time.

If the CI excludes the null value of zero, then the mean effect size is

considered to be statistically significant

Significant ES Non-Significant ES

ES = 0.40 (CI.025= +0.20; CI.975= +0.80) ES = 0.40 (CI.025= -0.40; CI.975= +0.40)

Previous Related Meta-Analyses

Justification & Aim

In the 21st century, it is necessary for rigorous research to

document the effect of adventure therapy programs, and

analyse the contributing factors.

• For: Boards, treatment facilities, funding bodies, policy makers

and consumers.

There are currently no published meta-analyses directly

concerned with adventure therapy, across the life span.

Broadly, this study aimed to:

Consolidate and provide up-to-date information regarding the

therapeutic uses and treatment effectiveness of adventure therapy

Provide valuable insight regarding the utility and therapeutic

outcomes of AT programs

Research QuestionsMeta-Analysis of Adventure Therapy Program Effects

1. What are the effects of adventure therapy?- What are the overall psychotherapeutic and psychosocial effects

of adventure therapy?

- What are effects on specific types of outcomes?

- How do the effects of adventure therapy compare to adventure

education?

- How do the effects of adventure therapy compare to other forms

of therapy?

2. What are the major correlates of adventure therapy

program outcomes? For example:- Outcomes (e.g., clinical, behavioural, family, social, academic)

- Study (e.g., publication year/type; size, quality of research)

- Participants (e.g., age, race, gender)

- Program (e.g., funding, delivery, group structure, model, length)

Selection Criteria

Included studies must:

Primarily use adventure activities specifically for

psychological and/or behavioural therapeutic purposes

Report at least pre- and post-psychological and/or

behavioural outcomes

Provide sufficient statistical information to allow

calculation of standardised mean effect sizes

– E.g., means, standard deviations, number of participants

Be reported in 1960 or later, and in English

Adapted from: George (2011); Wilson & Lipsey (2000)

Search Strategy & Data Extraction

Completed a comprehensive search:

– Electronic data bases

– Related journals, internet sites, meta-analyses

– Listserv and emails to experts in the field

– Scanned bibliographies and reference lists

Identified and obtained studies were included or

excluded based on the selection criteria.

Data was extracted from included studies via a

Coding Manual (Adapted from George (2011) and Wilson & Lipsey (2000))

Data AnalysisFollowed the Method of Borenstein et al. (2009)

Completed using Comprehensive Meta-Analysis - Version 2

The random-effects model was used

A single mean effect size from each sample was calculated

(where available) for each:

– Treatment group: Adventure; Alternative; No

– Outcome category:

Academic, Behaviour, Clinical, Family Development, Morality/Spirituality,

Physical, Self-Concept, and Social Development

– Time Comparison: Base-Pre; Pre-Post; Post-Follow-Up

– Overall effect

Additional comparisons were conducted to explore the

existence of potential moderators

Results Based On

2,908 effect sizes

From 206 unique samples located within 197 studies

Published between 1967 and 2012

Treatment Group No. & %

Adventure Therapy 2,275; 78.2%

Alternative Therapy 335; 11.5%

No Therapy 298; 10.3%

Time Comparison No. & %

Base-Pre 55; 1.9%

Pre-Post 2,274; 78.2%

Post-Follow-Up 579; 19.9%

Europe NZ Asia Canada Australia USA

2

(1.0%)

3

(1.5%)

7

(3.4%)

7

(3.4%)

26

(12.6%)

161

(78.2%)

Results

Based On:

Participants: ~ 17,728 (M = 86.1; SD = 148.3 per study)

Gender: male (62.5%) and female (37.5%)

Age: between 9 and 64 years (M = 16.9; SD = 7.0)

Program location: USA (78%), Australia (13%), Canada (3%),

Asia (3%), NZ (2%), & Europe (1%)

Program Model: Ropes/Challenge/Adventure-Based (42%),

Expedition (27%), Mixed (21%), Base Camp (5%), Residential

(4%), & Outpatient (1%).

Program length: 1-534 days (M = 63.5 days; SD = 148.0)

Mean time b/w Base-Pre: 20.8 days (SD = 15.3)

Mean time b/w Post-Follow-up: 181.4 days (SD = 276.0)

Overall Effect Sizes for Treatment Group by Time Comparison

Note: Caution should be used when interpreting some of the results due to high heterogeneity.

Interpreting g: 0.20 (small), 0.50 (medium), 0.80 (large)

0.09

0.56 0.59

0.14

-0.03

0.08

-0.08-0.20

0.00

0.20

0.40

0.60

0.80

Base-Pre Base-Post Base-FU

Adventure Tx Alternative Tx No Tx

Pre-Post Adventure Therapy Treatment Effect by Outcome Category

Hedges g ↓ 95% CI ↑ 95% CI

0.53

0.59

0.490.49

0.50 0.51 0.50

0.55

0.35

0.470.50

0.43 0.42 0.41 0.410.36

0.32

0.17

0.410.42

0.36 0.340.33

0.31

0.23

0.10

-0.01-0.05

0.05

0.15

0.25

0.35

0.45

0.55

Overall Clinical Self-Concept

SocialDevelopment

Academic Behaviour FamilyDevelopment

Physical Morality/Spirituality

Post-Follow-up Adventure Therapy Treatment Effect by Outcome Category

Hedges g ↓ 95% CI ↑ 95% CI

0.09

0.70

0.31

1.00

0.15 0.110.06 0.08 0.050.03

0.230.21 0.11

0.05 0.01-0.03 -0.05 -0.06-0.04

-0.24

0.12

-0.77

-0.07 -0.08-0.12 -0.17 -0.17

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

Overall Clinical Self-Concept

SocialDevelopment

Academic Behaviour FamilyDevelopment

Physical Morality/Spirituality

Publication Bias Results

Publication Bias was Assessed Using the Trim & Fill Method

Suggested 18 studies were missing

With:

g = .52

Without:

g = .47

Moderator Results

Conducted for Pre-Post Adventure Therapy Treatment

In most instances, there are no clear or notable trends

In general, higher effects were found for:

– Samples which were published or had a lower quality rating

– Programs whose use of adventure was adjunctive, or had an

open group structure or private placement

– Participants who were older or female

∴ On the whole, these variables do not greatly moderate

the effectiveness of adventure therapy programs

∴ Provide further support for the overall moderate

effectiveness of adventure therapy.

Meta-Regression Results

Meta-Regression:

To investigate whether sample-, program- and

participant-level characteristics explain the variation

(heterogeneity) of results among studies.

Results:

Age was the only significant predictor

Accounted for 6.8% of variance

Stronger outcomes for older participants

Sample: 5.6% Program: 11.5% Participant 27.5%

Pre-Post Adventure Therapy Treatment Effect by Age

Hedges g ↓ 95% CI ↑ 95% CI

0.53

0.44 0.44

0.59

0.921.00

0.47

0.24

0.37

0.50

0.66

0.66

0.41

0.04

0.30

0.41 0.40

0.31

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

All Ages ≤9 Years Old 10-14 YearsOld

15-17 YearsOld

18+ Years Old Mixed(e.g., Families)

Comparison of Different Types of Therapy

Note: 1) As there are several noteworthy differences between adventure therapy and more traditional forms of psychotherapy, comparison of meta-analytic

findings is problematic and should be interpreted with caution, and 2) as confidence intervals overlap, there are no significant differences between treatment types.

Benchmarking(Neill, 2003)

Results support the use of findings from this study for

benchmarking program effectiveness.

Obtained by comparing a particular program’s effect

sizes with appropriate meta-analytic benchmarks.

Provides a systematic means to categorise and evaluate

outcomes.

Provides valuable diagnostic information about a

program’s relative efficacy

Aged-based benchmarks are available from

http://www.danielbowen.com.au/meta-analysis

Example Use of the AT Meta-Analysis for Benchmarking

Limitations

Limitations of this Study & Meta-analytic Analysis in General:

Availability of studies

Heterogeneity

Generalisability

Type of data provided by empirical studies

– Including the provision of sufficient statistical information to

calculate effect sizes

Methodological quality of studies

– Including the use of psychometrically validated assessment tools

Recommendations for Future Research

Report matched samples, and enumerate drop outs.

Provide the necessary statistics to allow calculation of effect sizes.

Provide sample, participant and program demographics.

Capture data about a larger range of outcomes at multiple time

points.

Consult such resources as the Methodological Quality Rating

Scale or Rubric for Evidence-Based Research on Adventure

Programs (Gass et al., 2012) when designing a research project.

Use psychometrically validated assessment tools.

Rigorously develop purpose-built, multidimensional assessment

tools using the best available psychometric techniques.

Conclusion

Most substantial and robust meta-analysis of adventure

therapy program outcomes to date.

Confirms that overall adventure therapy is moderately

effective in facilitating positive short-term change.

On average, effects appear to be maintained in the long-

term.

Heterogeneity of outcomes indicates considerable

variability, although little of variance appears to be

explained by the measured moderators.

Key References

Borenstein, H., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2005). Comprehensive

meta-analysis (Version 2). Englewood, NJ: Biostat.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to

meta-analysis. Chichester, UK: John Wiley & Sons.

Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and

moderators. The Open Psychology Journal, 6, 28-53. doi:

10.2174/1874350120130802001

Gass, M. A., Gillis, H. L., & Russell, K. C. (2012). Adventure therapy: Theory, research,

and practice. New York: Routledge.

Hattie, J. M., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education

and Outward Bound: Out-of-class experiences that make a lasting difference. Review

of Educational Research, 67(1), 43-87. doi: 10.3102/00346543067001043

Littell, J. H., Corcoran, J., & Pillai, V. K. (2008). Systematic reviews and meta-analysis.

New York: Oxford University Press.

Neill, J. T. (2003). Reviewing and benchmarking adventure therapy outcomes:

Applications of meta-analysis. Journal of Experiential Education, 25(3), 316-321.