This article was downloaded by: [University of California, San Francisco]On: 16 December 2014, At: 01:59Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Psychotherapy ResearchPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/tpsr20

Replicating and extending the good-enough levelmodel of change: Considering session frequencyRobert J. Reese a , Michael D. Toland a & Nathaniel B. Hopkins ba University of Kentucky, Educational, School, and Counseling Psychology , Lexington, USAb University of Kentucky, University Counseling Center , Lexington, USAPublished online: 21 Jul 2011.

To cite this article: Robert J. Reese , Michael D. Toland & Nathaniel B. Hopkins (2011) Replicating and extending thegood-enough level model of change: Considering session frequency, Psychotherapy Research, 21:5, 608-619, DOI:10.1080/10503307.2011.598580

To link to this article: http://dx.doi.org/10.1080/10503307.2011.598580

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Replicating and extending the good-enough level model of change:Considering session frequency

ROBERT J. REESE1, MICHAEL D. TOLAND1, & NATHANIEL B. HOPKINS2

1University of Kentucky, Educational, School, and Counseling Psychology, Lexington, USA & 2University of Kentucky,

University Counseling Center, Lexington, USA

(Received 26 November 2010; revised 11 June 2011; accepted 13 June 2010)

AbstractThe good-enough level (GEL) model posits that the rate of change in psychotherapy is related to the total dose of therapy. Thepsychotherapy dose-response literature has typically measured dose as number of sessions attended without considering thenumber of days or weeks it takes to complete the sessions (session frequency). The current study sought to replicate the GELmodel and explore if session frequency moderates the influence that the number of sessions has on the rate of change inpsychotherapy. An archived naturalistic data set with a US university counseling center sample (n�1,207), with treatmentprogress measured using the Outcome Questionnaire-45 (Lambert et al., 1996), was used. Our results are consistent with theGEL model (i.e., clients who attended fewer sessions evidenced faster rates of change), but extended it by showing that therate of change was also influenced by session frequency (i.e., clients who attended more sessions on average per weekdemonstrated more rapid improvement). Evidence suggests that clinicians and researchers should give consideration tosession frequency, both in their work with clients and how ‘‘dose’’ is operationalized in psychotherapy research.

Keywords: psychotherapy; dose response; client progress and outcome; session frequency

The psychotherapy outcome literature (e.g.,

Anderson & Lambert, 2001; Hansen & Lambert,

2003; Lutz, Lowry, Kopta, Einstein, & Howard,

2001; Okiishi et al., 2006) has generally found that

the dose-response relationship is marked by a

negatively accelerated curve. Specifically, clients

generally tend to improve early in treatment and

the benefit of therapy diminishes as clients attend

more sessions. This finding is so prevalent in the

outcome literature that many psychotherapy out-

come researchers assume that this outcome trajec-

tory is a fact of psychotherapy (e.g., Lutz,

Martinovich, & Howard, 1999).

Some researchers have questioned this assump-

tion, however, because the studies that find a

negatively accelerating curve rest on another as-

sumption that all clients attend a similar number of

sessions (e.g., Baldwin, Berkeljon, Atkins, Olsen, &

Nielsen, 2009; Barkham et al., 1996; Barkham et al.,

2008). Barkham et al. (1996) proffered that if the

number of sessions are taken into account, the nega-

tively accelerating curve may disappear because such

consideration may show that clients respond to

treatment at different rates. They referred to this as

the ‘‘good-enough level’’ (GEL) model, and hy-

pothesized that individuals who attend fewer sessions

do so because they respond more quickly to treat-

ment than those who attend more sessions: Thus,

improving to a just good enough level where therapy

is no longer needed. They further suggest that if

separate outcome trajectories were organized and

computed by the number of sessions attended, the

respective outcome trajectories would all be linear in

nature and the slopes would be steeper for those who

attended fewer sessions.

Baldwin et al. (2009) conducted a dose-response

study that compared the traditional aggregate model

(i.e., ignoring number of sessions attended by clients)

and a stratified model (i.e., accounting for the number

of sessions attended by clients as consistent with the

GEL model; we will use ‘‘GEL’’ when referring to

the ‘‘stratified model’’). Using multilevel growth

curve modeling, results indicated that the GEL model

more appropriately described the nonlinear growth in

the sample data, suggesting that the amount of change

on the Outcome Questionnaire-45 (OQ-45; Lambert

et al., 1996) reported by clients is a function of the

number of sessions attended. They found clients who

Correspondence concerning this article should be addressed to Robert J. Reese, University of Kentucky, Educational, School, &

Counseling Psychology, 235 Dickey Hall, Lexington 40506-0017, USA. Email: jeff.reese@uky.edu

Psychotherapy Research, September 2011; 21(5): 608�619

ISSN 1050-3307 print/ISSN 1468-4381 online # 2011 Society for Psychotherapy Research

DOI: 10.1080/10503307.2011.598580

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

attended fewer sessions demonstrated faster rates of

improvement and those who attended more sessions

progressed slower. More specifically, outcome trajec-

tories for clients who attended fewer sessions (i.e.,

four or eight) demonstrated more immediate growth

but began to level out at the end of treatment.

Outcome trajectories for clients who attended several

sessions (i.e., 16 or 20) had slower initial growth,

which leveled out in the middle sessions, and then

became steeper toward the later sessions. Although

this lends support that sessions do not lessen in

potency over time for all clients, it does indicate that

sessions are not equivalent across individuals. More-

over, the Baldwin et al. findings are not entirely

consistent with Barkham et al.’s (1996) predictions

that the respective outcome trajectories would all be

linear in nature, but they did show steeper slopes for

those who attended fewer sessions.

Either way, it appears that both the traditional

aggregate and GEL models directly belie the idea of

sessions as equal in nature with regard to potency of

treatment. The aggregate dose-response model con-

tends that sessions cease to be equivalent given the

effect of diminished returns as treatment progresses,

and the GEL model posits that clients respond to

treatment doses (i.e., sessions) at different rates.

Hansen and Lambert (2003) mentioned this con-

cern when they noted that a limitation of their dose-

response study was the assumption that sessions

were equal in nature. Most, if not all, of the dose-

response literature makes this strict assumption that

one session equals one unit or dose. Psychotherapy’s

use of the word ‘‘dose’’ implies a level of specificity

similar to a dosage of medication given. To continue

this analogy, medication is also typically adminis-

tered at regular intervals, such as 100 mg every

12 hours. There are two potential problems with

applying this analogy to therapy. First, we cannot

assume that all therapy sessions are equal in dosage

given the wide variety of treatments employed by

different clinicians with complex individuals who

present with a myriad of issues that may respond to

any one session differently (Hansen & Lambert,

2003). Second, it is problematic to overlook when

the dosage is delivered. With medication, the interval

between administrations is taken into consideration.

An assumption the psychotherapy outcome literature

often seems to make is that psychotherapy follows

the traditional approach of one 50-minute session

conducted once per week and thus the interval

between the sessions is equal. There is some research

indicating that clients do not cleanly conform to this

attendance pattern (Kraft, Puschner, & Kurdy,

2006), but this variable is rarely mentioned as a

potential confound.

The GEL model has attempted to address the first

concern (i.e., sessions not being equal in potency

across all clients) by attempting to more fully

consider that clients may respond differently to

sessions. The model considers the context of the

client and therapist more fully, positing that clients

improve at different rates and that therapy is

continued until the client is improved enough to no

longer need treatment. However, the second concern

has received less attention. Session limits are becom-

ing the norm rather than the exception. Inspection of

the dose response literature might lead one to

conclude that it would be equally beneficial to space

sessions out, for instance attending therapy once

every 2 weeks in order to extend the time in therapy

or meet the demand by being able to see more

clients. Conversely, one might also conclude that

having more than one session per week may accel-

erate growth with clients. Is either approach plau-

sible? Is the growth rate for those attending eight

sessions in 4 months equivalent to eight sessions in 2

months? The current dose-response literature does

not provide a clear answer.

The reality is that many clients are not seen for

therapy on a strict week-to-week basis for a variety of

reasons: intentionally spacing sessions to meet

agency or insurance policy limits, generally good

functioning, missed and rescheduled appointments,

lack of motivation, poor time management, or even

holidays. As a result, dose-response studies do not

take into account the passage of time between

sessions during which clients may be making sig-

nificant progress, or lack thereof, on their therapeu-

tic concerns. Of course, there may also be differences

between a client-therapist dyad that choose to meet

every other week versus one that is intentionally

more flexible, or a client who periodically misses a

scheduled weekly session. In this study, we investi-

gate how psychotherapy outcome is impacted by the

interaction of the number of sessions attended and

frequency at which sessions are attended, providing a

more detailed picture of how clients respond to

therapy.

Most of the psychotherapy research investigating

the influence of session frequency has been con-

ducted in Europe (Freedman, Hoffenberg, Vores, &

Frosch, 1999; Kordy, Rad, & Senf, 1988; Kraft

et al., 2006; Sandell, Blomberg, & Lazar, 2002) and

is from the psychoanalytic tradition, where it is

common to have more frequent and more numerous

sessions. Freedman et al. (1999) found that clients

attending two or three sessions per week benefitted

more than clients who attended one session per

week. However, Kordy et al. (1988) did not find that

session frequency influenced outcome.

Psychotherapy dose-response 609

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

A more recent study (Kraft et al., 2006) found that

session frequency does matter with psychodynamic

and psychoanalytic therapy but not cognitive-

behavioral therapy. They found that the more weeks

clients went without treatment attenuated the ther-

apeutic alliance for those who received psychody-

namic treatment. Clients in psychoanalytic-based

treatment who missed more sessions early in treat-

ment improved more slowly across their treatment.

Interestingly, they also found that the clients in the

study attended less than one session per week on

average, and that the level of initial impairment did

not impact the total number of sessions attended

regardless of theoretical orientation employed.

We could only identify two studies using a US

sample that focused on session frequency (Reardon,

Cukrowicz, Reeves, & Joiner, 2002; Takuya et al.,

2009). Reardon et al. (2002) found that the longer it

took a client to attend 11 or fewer sessions the

less likely the client was to have a positive outcome.

In contrast, Takuya and colleagues (2009) found a

statistically significant negative relationship

(r � �.04) between session frequency and outcome

scores, but, given the small effect, concluded that the

practical implications were negligible. However,

neither study considered the influence session fre-

quency has on growth rates.

Thus, further delineation of ‘‘dose’’ is needed with

regard to the psychotherapy outcome literature,

which provides an unclear picture on how session

frequency influences the dose-response relationship.

Our exploratory study proposes to extend the GEL

model by including session frequency. We believe

that such inclusion in the GEL model may better

account for patterns of change during treatment than

the traditional aggregate or GEL models. The two

research questions addressed in this study are: (1)

Does the GEL model explain growth trajectories

better than the traditional growth model (i.e.,

aggregate model); and (2) Does the inclusion of

session frequency to the GEL model explain growth

trajectories better than the GEL model?

Method

Participants

The data used in this study were collected from

archived client records from a university counseling

center at a major Southeastern university in the

United States. The data reflect roughly 6 years of

clinical services at the center between 2001 and

2007. The campus enrollment grew from 24,000 to

27,000 undergraduate and graduate students during

the time period of the data sample. The information

gathered was available due to the implementation of

computerized record storage using the Protege

program and the clinical use of the OQ-45 (Lambert

et al., 1996). The data set represents a naturalistic

view of both client outcomes as well as the measure-

ment procedures of those outcomes.

The clinical staff of the counseling center con-

sisted of doctoral-level psychologists and doctoral

practicum students. The center offered a variety of

free services to enrolled students, including under-

graduate, graduate, and professional; however, only

those in individual therapy were administered the

OQ-45. Thus, all of the data in this study are from

clients in individual therapy working on a full range

of mental health concerns (n�3270). The data set

was reduced by excluding those who: attended only

one session (n�837), had a significant gap in treat-

ment (i.e., 90 or more days between two sessions;

n�737), received a second course of treatment

(n�368), did not have a date stamp necessary for

computing length of therapy were removed (in days or

weeks; n�365). Next we removed those who came in

for career or couple counseling (n�62). Finally,

those who had fewer than two administrations of

the OQ-45 (n�59) were removed. The resulting

data set includes 1207 clients who presented with

primary complaints that consisted of depression

(25.3%), anxiety (11.9%), academic issues (14.3%),

interpersonal relationship concerns (19.3%), and other

issues (e.g., stress, obsessive thoughts, alcohol, eating).

The representation of presenting issues appears consis-

tent with other university counseling centers in the

United States (e.g., Minami et al., 2009).

According to available data, 28.3% (n�342) of

clients were male and 69.7% (n�785) female; their

ages at the start of therapy ranged from 17.64 to 63.69

years (Mage�23.72, Mdnage�21.78, SDage�5.78).

The majority of clients were Caucasian (75%;

n�905), followed by African Americans (6.4%,

n�77), international clients (5.5%, n�66), Asian

American (2.2%, n�26), multiracial (1.5%, n�18),

Latino/a American (1.2%, n�15), and Middle

Eastern American (0.2%, n�3). The majority

of clients identified themselves as underg

raduate students (63.7%, n �769) with classifica-

tion as follows: freshmen (15.8%, n�191), sopho-

mores (16.6%, n�200), juniors (13.8%, n�166),

seniors (17.6%, n�212), graduate/professional

(24.3%, n�294), and other (1.1%, n�13). With

regard to enrollment, most clients indicated full time

status (82.1%, n�991), while 3.4% indicated part

time status (n�41). Most clients were single 78.5%

(n�948), followed by married (9.0%, n�5109),

divorced or separated (n�45, 3.7%), and partnered

or other (1.5%, n�18).

The center typically limits sessions to a maximum

of 15 sessions per academic year, but this limit is not

610 R. J. Reese et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

always adhered to based on clinical need. In reality

the number of clients reaching that number of

sessions is a small percentage of the total sample

(i.e., 90.1% of the sample had 15 or fewer sessions).

The median number of sessions attended was

five (M�7.82, Mode�2 [16.9%], SD�7.87,

Skewness�3.77, Kurtosis�21.7), while 43.25% were

seen for more than six sessions. Moreover the median

length in therapy was 8 weeks (M�12.75, Mode�1

[3.6%], SD�16.05, Skewness�3.88, Kurtosis�23.68)

with 49.1% continuing therapy for more than 8 weeks

and 41.5% continuing therapy for more than 10 weeks.

Also, the median number of sessions attended per

week was 0.62 (M�0.67, SD�0.35), which means

each client is attending a session roughly every 10 to

12 days.

The clinical staff (n�49; 65.4% female) were

primarily Caucasian (70.9%), followed by African

American (9.8%), multiracial (18.6%), and Asian

American (0.3%), while 0.3% did not identify their

race/ethnicity. The number of clients per staff ranged

from 1 to 204 with an average of 24.6 (Mdn�8).

Theoretical orientations of staff were diverse, includ-

ing cognitive-behavioral, solution-focused, psycho-

dynamic, interpersonal process, and narrative; most

staff indicated using an integrative/eclectic approach

to therapy drawing upon multiple theoretical orien-

tations.

Measures

All clients receiving individual therapy at the coun-

seling center completed the OQ-45. The OQ-45 is a

45-item self-response inventory that measures sub-

jective distress. Each item is answered on a 5-point

Likert-type response scale ranging from 1 (never) to

5 (almost always) with instructions to respond based

on the client’s experience of the previous week.

Higher scores indicate greater distress. The inven-

tory consists of a total scale score, as well as three

subscales: Symptom Distress, Interpersonal Rela-

tionships and Social Role. The average total scale

score for a clinical population in a counseling center

is 75, while in the normal student population it is 46

(Lambert et al., 1996). The OQ-45 is commonly

used in mental health settings as a brief assessment

tool. It has been shown to generate scores that are

internally consistent (a�.93) and possess short-

term temporal stability (r � .84) with a university

counseling center sample (Lambert et al., 1996).

The measure has demonstrated evidence of concur-

rent validity when compared to other commonly

used brief assessments such as the Symptom Check-

list � 90 � Revised (r �.78 with a counseling center

sample; Umphress, Lambert, Smart, Barlow, &

Clouse, 1997). Internal consistency data could not

be provided for the current sample because client

scores were not recorded and not available at the

individual item level.

Procedure

The OQ-45 was introduced as a standard procedure

of the previously described counseling center several

years before computerized records allowed data

collection. Initially, clients were to be administered

the OQ-45 at intake, again at their first follow-up

session and then at every third session thereafter.

This method of data collection is common to other

counseling centers in the United States and observed

in other research studies (see e.g., Brown, Burlingame,

Lambert, Jones, & Vaccaro, 2001; Spielmans,

Masters, & Lambert, 2006). It was determined by

the center that administering the OQ-45 in the first

session after intake was redundant and that require-

ment was removed part-way through data collection.

Additionally, clinicians could administer the OQ-45

at any session they felt was clinically relevant, though

this was uncommon. As a result, some of the records

in the data set do not follow the proscribed schedule

of OQ-45 administrations. The OQ-45 was adminis-

tered for clients where termination was planned

(n�742). Each clinician was responsible for identi-

fying on the schedule when a particular client was to

be given the OQ-45, with the receptionist staff then

giving the client the measure when they arrived for a

session. Clinicians would then hand-score the mea-

sure and enter the results into a client’s record.

Data Analysis

To evaluate the primary research questions we

applied multilevel growth models (Hox, 2010), also

known as mixed-effects models and hierarchical

linear models (Raudenbush & Bryk, 2002; Singer

& Willett, 2003). Although our data are inherently

nested within therapists at the highest level, initial

analyses indicated that therapist random effects (i.e.,

both intercepts and slopes) were either near zero or

non-estimable for more complex models. In our case

this finding is probably attributed to the fluctuating

number of observations within therapists (i.e., the

number of clients per staff ranged from 1 to 204 with

an average of 24.6 and median of 8). According to

Hox (2010) a solution to this problem is to remove

random effects, which simplifies the model. In

addition, initial analyses comparing the uncondi-

tional (i.e., no predictors or covariates) three-level

growth model (repeated observations within clients

within therapists) to the two-level growth model

(repeated observations within clients) showed no

improvement in fit to the data. As a result we were

Psychotherapy dose-response 611

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

limited to using a two-level growth model to analyze

the nested structure of the data.

The specific models examined in this study con-

sisted of the aggregate model, GEL model, and

modified GEL model using session-by-session as the

metric of time. The aggregate model is the dose-

effect response model or basic two-level uncondi-

tional growth model which estimates the average rate

of change across all clients. The GEL model extends

the aggregate model by specifying that improvement

is a function of the total number of sessions a client

attended. As such, the modified GEL model extends

the GEL model by specifying that improvement is

also conditional on the session frequency (fre-

quency), which can be conceptualized as the average

number of sessions per week:

frequency¼ 1

n� 1

Xn�1

i¼1

Si � Siþ1¼1

n� 1

Xweeks; (1)

where n �total number of sessions attended; Si �Si�1�number of weeks between two consecutive

sessions; weeks � total number of weeks in therapy.

For instance, a client who attended four sessions

(i.e., intake, session 2, session 3, and session 4) in 3

weeks received a session frequency of 1 because the

average amount of time between each session was a

week. To test these models a series of hierarchical

growth models were examined.

The aggregate model is more formally defined as:

Yti ¼ c00þc10ðsessionÞtiþr0iþr1iðsessionÞtiþeti; (2)

where Yti is the OQ-45 score at session t for client i;

g00 is the initial client mean score on the OQ-45

(centered at the start of therapy or the first session);

g10 is the average linear rate of change between

adjacent session points, respectively; r0i, and r1i are

the random client effects or the deviation of client i’s

OQ-45 score from the average initial status and

linear rate of change, respectively; and eti is the

random client effect or client i’s error at time t. Also,

note that the intercept-linear covariance (s01) com-

ponent was estimated.

When we extend the aggregate model to include

the total number of sessions a client attended

(#sessions) we arrive at the GEL model, which is

defined as:

Yti¼ c00þc01ð#sessionsÞiþc10ðsessionÞtiþ c11ðsessionÞtið#sessionsÞiþr0iþr1iðsessionÞtiþeti;

(3)

where the random client (r0i, r1i, and eti) and fixed

effects (g00 and g10) are the same as in the aggregate

model, but because total number of sessions was

centered (see below) these fixed effects and all others

in this model refer to clients with an average of 5.72

attended sessions; g01 is the average difference in

initial client OQ-45 as a function of total number of

sessions attended; g11 is the average difference in

linear growth rates as a function of total number of

sessions a client attended. Note that the natural log

transformation for total number of sessions a client

attended was used because it had a positively skewed

distribution. Then, this variable was grand mean

centered (i.e., ln #sessions �1.7442, which is equal

to 5.72 sessions).

When we extend the GEL model to include the

frequency at which clients attended therapy per week

we arrive at the modified GEL model, which is

defined as:

Yti¼ c00þc01ð#sessionsÞiþc02ðfrequencyÞiþc03ð#sessionsÞiðfrequencyÞiþc10ðsessionÞtiþc11ðsessionÞtið#sessionsÞiþc12ðsessionÞtiðfrequencyÞiþc13ðsessionÞtið#sessionsÞiðfrequencyÞiþ r0i

þr1iðsessionÞtiþeti;(4)

where the random client (r0i, r1i, and eti) and fixed

effects (g00, g10, and g11) of the modified GEL model

are the same as in the GEL model, but because

session frequency was centered (see below) these

fixed effects and all others in this model refer to

clients attending one session per every 11 to 12 days

with an average of 5.72 attended sessions; g02 is the

average difference in initial client OQ-45 as a

function of session frequency; g03 is the average

difference in initial client OQ-45 as a function of

both total number of sessions a client attended and

session frequency; g12 is the average difference in

linear growth rates as a function of session fre-

quency; g13 is the average difference in linear growth

rates as function of both total number of sessions a

client attended and session frequency. Note that the

natural log transformation for session frequency was

used because it had a positively skewed distribution.

Then, this variable was grand mean centered (i.e., ln

frequency�[�.5145]), which is equal to about one

session per every 11 to 12 days).

To compare the improvement in fit between

nested models we used the likelihood ratio test

(LRT) which tests the �2 log likelihood (�2LL)

or deviance statistic between models. Specifically,

the difference in the �2LL values was used to test

the null hypothesis that two nested models do not

have statistically significant different model fits from

each other. The difference in �2LL from two

nested models is Chi-square distributed (Dx2) with

degrees of freedom equal to the difference in

parameters estimated by the two models. We also

(3)

612 R. J. Reese et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

used the Bayesian information criterion (BIC) and

Akaike information criterion (AIC) to examine fit,

where smaller values for both indicate better model

to data fit. All growth models were estimated using

proc mixed in SAS version 9.2 using a full informa-

tion maximum likelihood estimation method, which

utilizes all available data and treats any missing data

as missing at random. Kenward-Roger degrees of

freedom were also estimated. Then, a global

pseudo-R2 effect size statistic for each longitudinal

model was computed by taking the predicted OQ-45

scores and correlating it with observed OQ-45 scores

(see Peugh, 2010). This value was then squared to

get a statistic similar to R2 in traditional multiple

regression, which measures the amount of variation

in OQ-45 scores that can be explained by the model

under consideration. These pseudo-R2 effect size

statistics were estimated in Mplus version 6.1 using

maximum likelihood estimation which also utilizes

all available data and treats any missing data as

missing at random.

Results

Growth Analyses

For our data, a linear, quadratic, and cubic compo-

nents model was fitted first per recommendations by

Nagin (2005) as a sensible starting point. A cubic

polynomial was selected as the starting point for the

aggregate model because the general shape of OQ-45

growth across sessions has been found to be cubic

(Baldwin et al., 2009). A simpler growth model was

interpreted if the predicted trajectories showed little

practical utility. Results for the aggregate model

consisting of linear, quadratic, and cubic components

showed each piece to be statistically significant.

A comparison of this cubic model (BIC �30,816.

6, AIC �30,775.8) to a quadratic model

(BIC �30,840.2, AIC �30,804.6) showed the cubic

model had better fit to the data, Dx2(1) �30,790.6�30,759.8 �30.8, pB.001. Also, a comparison of the

cubic model to a linear model (BIC �30,834.2,

AIC �30,803.6) showed the cubic model had better

fit to the data, Dx2(2) �30,791.6�30,759.8 �31.8,

pB.001. However, a plot of the cubic model and

inspection of the components (intercept �70.6993,

linear �-1.733, quadratic �0.03843, cubic��0.00046) indicated the negatively accelerating

rate of change in predicted OQ-45 scores was mostly

linear with subtle to nearly nonvisible nonlinear

growth rates. Thus, from a practical utility and

parsimonious perspective the data appear to be best

described by a linear pattern. The results for the linear

aggregate model are shown in the first column of

Table I. Fixed effects estimates (top) and variance-covariance estimates (bottom) for models of the prediction of growth in OQ-45 scores

across sessions

Parameter Aggregate model GEL model Modified GEL model

Fixed effects

Mean initial OQ-45 (intercept; g00) 70.34** 71.27** 71.45**

Mean linear growth rate (session; g10) �1.36** �2.40** �2.41**

#Sessionsa (g01) 1.66* 2.10**

#Sessions�session (g11) 1.10** 1.09**

Frequencya (g02) 6.35**

Frequency�#sessions (g03) 3.88*

Frequency�session (g12) �0.80*

Frequency�#sessions�session (g13) 0.30

Random effects

Residual variance (eti) 121.81** 122.31** 122.53**

Intercept variance (r0i) 377.45** 374.61** 368.73**

Linear variance (r1i) 1.40** 0.84** 0.79**

Covariance (intercept, linear; s01) �4.18* �4.18** �3.80*

Pseudo-R2 .005 .035 .0524

Deviance (�2LL) 30,791.6 30,693.4 30,675.3

Dx2(df) 98.2 (2)** 18.1 (4)**

BIC 30,834.2 30,750.1 30,760.5

AIC 30,803.6 30,709.4 30,699.3

Note. #Sessions � total number of sessions a client attended therapy; Frequency � frequency at which clients attended therapy per week;

�2LL � �2*log likelihood.aThe natural log transformation for number of sessions and frequency was used because each variable had a positively skewed distribution.

#Sessions was grand mean centered (i.e., ln #sessions �1.7442, which is equal to 5.72 sessions, while frequency was grand mean centered

(i.e., ln frequency-[�.5145]), which is equal to about one session per every 11 to 12 days). The zero-order correlation between the natural

log of the centered predictors frequency and number of sessions was r��.11, which precludes the results as being an artifact of

multicollinearity between these two variables.

**p B .01; *p B .05.

Psychotherapy dose-response 613

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

Table I and a plot of the aggregate model is provided

in Figure 1.

The results for the GEL model (refer to the second

model in Table I) had a statistically significant better

model to data fit than the traditional aggregate model,

Dx2(2) �30,791.6�30,693.4 �98.2, pB.001. A com-

parison of the BIC and AIC statistics (where smaller is

better; see the bottom two rows of Table I) show the

superiority of the GEL model (BIC �30,750.1,

AIC �30,709.4) to the traditional aggregate model

(BIC �30,834.2, AIC �30,803.6). Also, the pseudo-

R2 improved from .005 (0.5%) to .035 (3.5%). This

means the GEL model explained about 3% more

variability in OQ-45 scores than the traditional

aggregate model. When inspecting the GEL model

results, we find clients who attended a different

number of sessions improved at different linear rates.

Descriptively, those who attended more than 5.72

sessions tended to have less steep negative linear

slopes, while those attending fewer than 5.72 sessions

tended to have steeper negative linear slopes. This

provides evidence that the aggregate model does not

accurately capture many clients’ rate of change during

therapy because it assumes rate of change is the same

regardless of number of sessions attended. However,

the GEL model assumes the frequency at which

clients attend therapy is non-consequential to growth

rates.

As a result, the modified GEL model which

included session frequency was compared to the

GEL model and found to have a statistically signifi-

cant better fit to the data, Dx2(4) �30,693.4�30,675.3 �18.1, pB.001 (refer to third model in

Table I). A comparison of the AIC statistic shows

the superiority of the modified GEL model

(BIC �30,760.5, AIC �30,699.3) to the GEL mod-

el (BIC �30,750.1, AIC �30,709.4), but the BIC

did not. Considering, in combination the AIC, BIC,

and likelihood ratio test (Dx2), the modified GEL

model is the better fitting model. The pseudo-R2

improved from .035 (3.5%) to .0524 (5.24%). This

means the modified GEL model explained 1.74%

more variability in OQ-45 scores than the GEL

model. The modified GEL model shows the linear

rate of change differed for those clients who attended

a different number of sessions and for those differing

in session frequency, but no evidence was found for

linear rates of change varying by the combination of

session frequency and number of sessions attended,

giving evidence that the GEL model is not entirely

complete in reflecting client rate of change in therapy.

A plot of the number of sessions attended by

session and session frequency by session interactions

found in the modified GEL model for select total

number of sessions attended and session frequency is

provided in Figure 1. Figure 1 shows that the rate of

linear change for the average client is a function of

total number of sessions attended or session fre-

quency. Specifically, Figure 1 shows that as clients

attend sessions less frequently (i.e., have larger gaps

in weeks between sessions*one session per two

weeks to one session per week) the amount of

change between adjacent sessions tends to diminish

across sessions, regardless of the number of sessions

attended. Also, the modified GEL model in Figure 1

shows that clients who attended fewer sessions

tended to have larger changes between adjacent

sessions, regardless of session frequency.

It is also important to point out that results from

the modified GEL model show that the relationship

between initial OQ-45 scores and session frequency

is moderated by number of sessions attended. The

positive slope for this interaction (g03�3.88) sug-

gests that as the number of sessions attended

increases, the session frequency by number of

sessions attended slope becomes more positive (or

less negative) in direction. This means that those

clients with higher number of sessions attended did

40

45

50

55

60

65

70

75

80

0 2 4 6 8 10 12 14 16 18 20

Pre

dic

ted

OQ

-45

Sco

re

Sessions

5 Sessions Once Per Wk 10 Sessions Once Per Wk 20 Sessions Once Per Wk5 Sessions Once Per 2 Wks 10 Sessions Once Per 2 Wks 20 Sessions Once Per 2 WksAggregate Model

Figure 1. Modified GEL model trajectories for select number of sessions attended by frequency and trajectory for the aggregate model.

614 R. J. Reese et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

not always start out with higher initial distress.

Specifically, those clients with a higher number of

sessions attended (e.g.,�20) coupled with a lower

session frequency (e.g., one session about every 5

weeks) tended to have lower initial distress levels

than those with fewer attended sessions (e.g.,B3)

and low session frequency. The difference in initial

distress between clients attending many (e.g.,�20)

vs. few (e.g.,B3) sessions was reversed at higher

levels of session frequency. However, the differences

in initial distress levels between high and low number

of sessions attended becomes negligible when session

frequency approached about once every 2�3 weeks.

Interestingly, the correlation between the intercept

and linear variance components in the modified

GEL model was statistically significant, r��.22,

p�.01, indicating that those clients with lower initial

OQ-45 scores tended to have steeper linear slopes,

while clients with higher initial OQ-45 scores tended

not to have as steep linear rates of change across

sessions, controlling for or partialing out all other

predictors in the model. Clients in less distress

tended to improve more rapidly, and clients in

more distress improved more slowly. This finding is

consistent with outcome research that generally finds

individuals in more distress take more time to

improve in treatment, they simply respond more

slowly. A similar correlation was found between

these two components in the GEL and aggregate

models, r��.23, p�.006, and r��.18, p�.01,

respectively.

Given the possibility that missing OQ-45 scores at

the last session could influence estimated parameters

in the three models (aggregate, GEL, and modified

GEL) we statistically addressed this concern by

contrasting completers vs. dropouts (i.e., clients

with or without data at the last session; com-

pleter �1, dropout �0). To examine the longitudi-

nal dropout seen in our study we focused on

Hedeker and Gibbons’ (1997) approach to ran-

dom-effects pattern-mixture modeling by applying

their technique to each of the four longitudinal

models (see also Little, 1995). Specifically, ran-

dom-effects pattern-mixture modeling involved add-

ing the variable dropout to each parameter estimated

in each model which resulted in two, four, and

eight additional parameters being estimated for

the aggregate, GEL, and modified GEL models,

respectively (labeled pattern-mixture).

A comparison of the aggregate model BIC

(30,834.2) and AIC (30,803.6) statistics to the

pattern-mixture aggregate model BIC (30,841.1)

and AIC (30,800.3) statistics showed no improve-

ment in model fit when using the BIC statistic or a

subtle improvement in model fit when using the AIC

statistic with the inclusion of the variable completers

and interaction with completers. Moreover, a like-

lihood ratio test indicated that the pattern-mixture

aggregate model had a statistically significant better fit

to these data than the aggregate model,

Dx2(2) �30,791.6.8�30,784.3 �7.3, p�.03. How-

ever, an inspection of the two additional parameters

included in the pattern-mixture aggregate model

showed both were not statistically significant

(completers � �1.96, p�.11; completers by

session � �0.27, p�.10). Given these findings, we

determined that the aggregate model provided an

accurate representation of the data for completers and

dropouts and that initial status and linear rate of

change did not vary by completer status. A compar-

ison of the pattern-mixture GEL model

(BIC �30,776.7, AIC �30,715.6) to the GEL

model (BIC �30,750.1, AIC �30,709.4) showed

no improvement in fit, Dx2(4) �30,693.

4.3�30,691.6 �1.8, p�.77. Similarly, a comparison

of the pattern-mixture modified GEL model

(BIC �30,811.4, AIC �30,709.5) to the modified

GEL model (BIC �30,760.5, AIC �30,699.3)

showed a reduction in fit, Dx2(8) �30,699.5�30,675.3 �24.2, p�.002. Overall, these results

show that study completion and interactions with

study completion are not statistically significantly

related to OQ-45 scores, over and above the influ-

ences of the predictors included in the original

aggregate, GEL, and modified GEL models.

Discussion

We had two major findings for the current study. The

first major finding is that our study demonstrated

results consistent with the GEL model. The GEL

model better fit the data than the aggregate model,

indicating that the number of sessions attended at

the individual level needs to be considered in out-

come research and that growth in therapy is linear

when this variable is considered and may not con-

form to the negatively accelerating curve as has been

found using an aggregate dose-response model (e.g.,

Howard, Kopta, Krause, & Orlinsky, 1986; Howard,

Moras, Brill, Martinovich, & Lutz, 1996; Kordy

et al., 1988). This appears to be consistent with

Barkham et al.’s (1996) claim that clients who attend

fewer sessions often improve to a ‘‘good enough

level’’ and then discontinue therapy. Clients who

attended fewer sessions responded more quickly and

clients who attended more sessions responded more

slowly. Research with the aggregate model has often

led to the conclusion that treatment sessions yield

diminished returns, that is the ‘‘dose’’ weakens after

a certain number of sessions. Our results suggest that

this was not the case when the data were stratified

by sessions attended or not stratified. Clients in

Psychotherapy dose-response 615

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

our study appeared to have different rates of

improvement based on either number of sessions or

session frequency, but not the interaction of these

two variables. This indicates session frequency is not

the only important component necessary for describ-

ing growth in therapy.

One finding that we did not anticipate was that the

aggregate model best fit a linear rate of change; there

was not a negatively accelerating curve. Rather,

client improvement was found to be similar through-

out the course of therapy when the number of

sessions a client attended was not considered.

Although this result has been found in other studies

(Barkham et al., 1996; Shapiro et al., 2003), most

dose-response studies have reported an effect of

diminished returns as therapy progresses. It is

possible this may have been revealed with more

sessions plotted, but the session limit policy (ap-

proximately 90% attended 15 of fewer sessions)

prevented us from extending the number of sessions.

Although the aggregate model in our study was

linear, it still did not portray the dose-response

relationship as accurately as the GEL model. The

conclusion, however, that the number of sessions a

client attends is the variable that best represents

differences in rate of improvement is premature.

Baldwin et al. (2009) note that it is unknown

whether the number of sessions attended and

resulting rates of improvement are the results of

different client subgroups or populations. If so, there

could be a dose-response but simply specific to

different groups. For example, it is important to

note that in the current study we found a client’s

initial OQ-45 score was related to the interaction

between session frequency and number of sessions

attended. In other words, clients’ initial level of

distress, number of sessions attended, and session

frequency could be one of several variables that

could be related to possible subgroupings if growth

mixture modeling was used.

The second major finding suggests that when we

consider the number of sessions in treatment with

session frequency, the rate of change presents a

different and perhaps more accurate picture. The

modified GEL model provided a better model fit

than the GEL model and showed that having more

frequent sessions (i.e., more sessions in fewer days or

weeks) resulted in steeper growth curves. Specifi-

cally, clients who attended more sessions within a

period of time (e.g., attended eight sessions rather

than four sessions in a span of 8 weeks) had faster

rates of change. Moreover, session frequency re-

sulted in faster rates of improvement independent of

the number of sessions attended (i.e., when the

number of sessions attended was controlled for).

This finding is consistent with studies conducted by

others (Kraft et al., 2006; Reardon et al., 2002) that

were focused on the amount of time (measured in

days or weeks) that it took to complete a certain

number of sessions. Session frequency appears to

matter*all therapy sessions are not equal. The

results suggest clients who had shorter intervals

between therapy sessions improved more quickly.

Although the major findings are tentative

and warrant replication, they do encourage further

attention be given to how we conceptualize and

operationalize the term ‘‘dose.’’ Both the aggregate

dose-response model (e.g., Hansen & Lambert,

2003; Lutz et al., 1999) and the GEL model

(Baldwin et al., 2009; Barkham et al., 1996) strongly

suggest that sessions are not equivalent in nature.

The GEL model, however, illustrates more clearly

that the effectiveness of any given session may vary

greatly across individuals and a host of other con-

textual factors. Our findings indicate that considera-

tion should be given to the of number of days/weeks

in treatment, or session frequency, and the notion

that sessions can be assumed to be a metric

composed of equal units and equal intervals.

The psychotherapy outcome literature, as can be

observed in its nomenclature, has tethered itself to

the medical/pharmacological paradigm. The results

of our study suggest that more thought be given to

the underlying metric(s) by which we measure

treatment progress. We offer two possibilities. First,

as consistent with a pharmacological model, time

between doses (sessions) needs to be considered

more fully as is done when considering dosage and

efficacy of medications. Prescriptions for medication

provide intervals along with dosage information in

order to maximize effectiveness and to minimize

potential adverse effects. Second, the singular term

of ‘‘dose’’ should be reconsidered. Borrowing from

the logic that reporting null hypothesis results as

‘‘significant’’ rather than the more precise ‘‘statisti-

cally significant’’ is semantically sloppy and leads to

faulty inferences (Thompson, 1996), perhaps ex-

tending the term ‘‘dose’’ to ‘‘weekly session’’ or

simply reporting the mean interval across all sessions

would provide a more precise metric for under-

standing the relationship of the number of sessions

attended in relation to an actual time metric, be it

days or weeks. For example, it may be useful to

report the median and average number of sessions

attended per week. The median number of sessions

attended per week in our study was 0.62 (M�0.67,

SD�0.35), indicating that our data set better

reflected a week-and-a-half between adjacent ses-

sions.

It would be nice to be able to compare our results

with other dose-response studies, particularly studies

that use large data sets from university counseling

616 R. J. Reese et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

centers that consider time in treatment from both

session-to-session and session frequency perspec-

tives. Inferences about the relationship between

sessions attended and session frequency could be

made. Considering the time metric more fully would

also allow researchers to be more confident about the

similarity of the data sets. As a result, assumptions

about session intervals may be inaccurate. For

example, Kraft et al. (2006) found that even for

clients who attended psychoanalytic therapy, where

more than one session per week is the norm, they

attended less than one session per week on average,

which is consistent with what we found from our

sample.

Future Research

Research that further considers the GEL model and

session frequency to evaluate trajectories of change

needs to be conducted. Studies that utilize random

assignment to control for the time between sessions,

time in treatment, and the number of sessions

attended would be ideal. For example, clients could

be randomly assigned to attend therapy every week or

every other week for a specified number of sessions.

Naturalistic studies would also be valuable provided

that studies are structured so variables of importance

could be tracked carefully, including the amount of

time between sessions and reasons for interruptions in

treatment. Such studies would better our under-

standing of how session intervals relate to outcome.

Future research should also address how distress

severity moderates the relationship between outcome

and session intervals. Common sense would seem to

follow that more frequent treatment is necessary when

a client is in more distress (Beutler et al., 2004), but

such a stance does not have a lot of empirical support.

More research should also focus on the trajectory

of change that both replicates and extends recent

work that has challenged the assumption that the

negatively accelerating curve is the most accurate

picture to describe how change occurs in therapy.

Baldwin et al. (2009) and Barkham et al. (2006) have

both provided evidence that many clients do not

necessarily respond to treatment in this manner.

Hayes, Laurenceau, Feldman, Strauss, & Cardaciotto

(2007) take this one step further in their summary by

providing evidence that change does not always

occur in a linear fashion when diagnosis and

theoretical approach are considered. Additionally, it

may be important to consider other treatment out-

comes than just client distress given trajectories of

change may differ for alternate outcomes. Collecting

data across treatment, rather than in a pre-post

fashion, coupled with utilizing multilevel growth

modeling techniques will allow for a more sophisti-

cated understanding of how clients respond to

psychotherapy treatment.

Limitations of Research

Our study has several limitations worth noting. First

are the limitations that are inherent with naturalistic

research. We were not able to account or control for

the variety of treatment interventions provided or the

number of sessions clients attended. Second, using

an archived data set precluded us from being able to

identify the reasons why clients discontinued therapy

or why sessions were missed*were session intervals

intentionally planned or unplanned (no-shows and

cancellations)? A third limitation, and our biggest

concern with the data set, was that the OQ-45 was

not given every session and many clients did not have

OQ-45 data at their last session. Data collected in

this manner may have underestimated treatment

effects. Ultimately, we do not believe this is an

egregious limitation of our study for three reasons.

First, we used multilevel modeling which is designed

to address longitudinal, repeated measure designs

that have missing data. Second, the data collection

format used in this study is not uncommon at

university counseling centers and agencies in the

USA that use outcome assessment (e.g., Brown

et al., 2001; Spielmans et al., 2006). This is certainly

a limitation, but it may also be a strength in terms of

reflecting data collection procedures in real-world

treatment settings where completing measures every

session is not feasible. Third, we compared those

who had OQ-45 data at termination to those without

it to evaluate whether there were differences in any of

the models’ parameter estimates. In general, data for

clients with and without OQ-45 data at termination

were statistically equivalent given support for the

data collection process at the last session.

We also attempted to temper the effects of missing

data by eliminating clients who did not have at least

two observed data points (two OQ-45 measures);

however, this led to a fourth limitation*a smaller

sample size which possibly limits generalizability.

The concerns of sample size and missing data are

lessened by the consistency of our results with other

studies that evaluated the GEL model (Baldwin

et al., 2009; Barkham et al., 2006). Specifically, the

GEL model from our study parallel the Baldwin

et al. (2009) study where data were collected every

session and show evidence of the linear trajectories

per number of sessions attended as predicted by

Barkham et al. (1996). Generalizability is also a

concern because the treatment duration for clients

was not lengthy (approximately 40% had between

five and 15 sessions) and the treatment setting was a

US university counseling center and was composed

Psychotherapy dose-response 617

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

of primarily young, White clients. The duration of

treatment, however, is consistent with other univer-

sity counseling centers, other outpatient treatment

agencies in the United States, and other studies

(Baldwin et al., 2009). A fifth limitation is that we

only used one self-report measure to evaluate treat-

ment outcome. There is evidence, however, that the

OQ-45 has found results consistent with other out-

come measures (Baldwin et al., 2009).

Clinical Implications and Conclusions

Session limits are common to practitioners and

agencies due to resource limitations and external

conditions such as managed care. Our study ad-

dresses how session limits are derived and imple-

mented. First, the replication of the GEL model

provides further evidence that reliance on an aggre-

gate model may not be the most appropriate way to

evaluate how individuals respond to treatment. This

is because an aggregate model tends to generally

underestimate trajectories (i.e., the slopes appear less

steep) for clients with fewer sessions and overesti-

mate trajectories (i.e., the slopes appear steeper) for

clients attending more sessions (see Figure 1).

Deriving session limits based on the aggregate model

may not be beneficial given the differential response

to treatment as indicated by the GEL model. One

size does not fit all. Second, the finding that session

frequency is related to the rate of change suggests

that session frequency should be carefully considered

by therapists and clients. The temptation may be to

spread sessions out (e.g., every other week) to

‘‘ration’’ treatment and increase the length of time

in treatment. Future research is certainly needed,

but our modified GEL models indicate that clients

who attended sessions roughly once per week

incurred more change in fewer sessions compared

to those who had less frequent sessions.

Our findings suggest that clients respond and use

therapy differently, reinforcing the importance of

tracking outcome in treatment to understand how

clients are progressing. The benefits of tracking

client outcome (patient-focused research or client

feedback) across treatment are well-documented

(e.g., Anker, Duncan, & Sparks, 2009; Whipple

et al., 2003), having been found to help clinicians

identify clients not progressing as expected and

reduce premature terminations. Using client feedback

has also been found to generate better therapy out-

comes when compared to treatment that does not

utilize client feedback (e.g., Reese, Norsworthy, &

Rowlands, 2009; Shimokawa, Lambet, & Smart,

2010).

In summary, the results of our exploratory study

offer evidence for the GEL model*how long clients

stayed in treatment was dependent on how quickly

they improved. Those who attended more sessions

responded more slowly and those who attended

fewer sessions improved faster. Additionally, we

found that the frequency of session attendance

influenced the rate of change*more sessions in

fewer days/weeks predicted faster improvement.

Inclusion of session frequency improved the predic-

tion of the GEL model. Considering session fre-

quency may provide researchers and clinicians a

more accurate depiction of the rate of change across

treatment. Further research is certainly warranted.

The availability of statistical analyses such as multi-

level modeling and growth mixture models coupled

with the practice of tracking outcomes with clients

across treatment offers the opportunity to further

explore and evaluate the myriad of client, therapist,

and treatment variables that influence psychotherapy

treatment outcome in a more sophisticated manner.

References

Anderson, E.M., & Lambert, M.J. (2001). A survival analysis of

clinically significant change in psychotherapy. Journal of Clinical

Psychology, 57, 875�888. doi:10.1002/jclp.1056.

Anker, M.G., Duncan, B.L., & Sparks, J.A. (2009). Using client

feedback to improve couple therapy outcomes: A randomized

clinical trial in a naturalistic setting. Journal of Consulting and

Clinical Psychology, 77, 693�704. doi:10.1037/a0016062.

Baldwin, S.A., Berkeljon, A., Atkins, D.C., Olsen, J.A., & Nielsen,

S.L. (2009). Rates of change in naturalistic psychotherapy:

Contrasting dose-effect and good-enough level models of

change. Journal of Consulting and Clinical Psychology, 77, 203�211. doi:10.1037/a0015235.

Barkham, M., Connell, J., Stiles, W., Miles, J.N., Margison, F.,

Evans, C., & Mellor-Clark, J. (2006). Dose-effect relations and

responsive regulation of treatment duration: The good enough

level. Journal of Consulting and Clinical Psychology, 74, 160�167.

doi:10.1037/0022.006X.1.160.

Barkham, M., Rees, A., Stiles, W.B., Shapiro, D.A., Hardy, G.E.,

& Reynolds, S. (1996). Dose�effect relations in time-limited

psychotherapy for depression. Journal of Consulting and Clinical

Psychology, 64, 927�935. doi:10.1037/0022-006X.64.5.927.

Beutler, L.E., Malik, M., Alimohamed, S., Harwood, T.M.,

Talebi, H., Noble, S., & Wong, E. (2004). Therapist variables.

In M.J. Lambert (Ed.), Bergin and Garfield’s handbook of

psychotherapy and behavior change (pp. 227�306). NewYork,

NY: Wiley.

Brown, G.S., Burlingame, G.M., Lambert, M.J., Jones, E., &

Vaccaro, J. (2001). Pushing the quality envelope: A new

outcomes management system. Psychiatric Services, 52, 925�934. doi:10.1176/appi.ps.52.7.925.

Freedman, N., Hoffenberg, J.D., Vorus, N., & Frosch, A. (1999).

The effectiveness of psychoanalytic psychotherapy: The role of

treatment duration, frequency of sessions, and the therapeutic

relationship. Journal of the American Psychoanalytic Association,

47, 741�772. doi:10.1177/00030651990470031001.

Hansen, N.B., & Lambert, M.J. (2003). An evaluation of the

dose-response relationship in naturalistic treatment settings

using survival analysis. Mental Health Services Research, 5,

1�12. doi:10.1023/A:1021751307358.

Hayes, A.M., Laurenceau, J., Feldman, G., Strauss, J.L., &

Cardaciotto, L. (2007). Change is not always linear: The study

618 R. J. Reese et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014

of nonlinear and discontinuous patterns of change in psychother-

apy. Clinical Psychology Review, 27, 715�723. doi:10.1016/

j.cpr.2007.01.008.

Hedeker, D., & Gibbons, R.D. (1997). Application of random-

effects pattern-mixture models for missing data in longitudinal

studies. Psychological Methods, 2, 64�78. doi:10.1037/1082-

989X.2.1.64.

Howard, K.I., Kopta, S., Krause, M., & Orlinsky, D. (1986). The

dose-effect relationship in psychotherapy. American Psychologist,

41, 159�164. doi:10.1037/0003-066X.41.2.159.

Howard, K.I., Moras, K., Brill, P.L., Martinovich, Z., & Lutz, W.

(1996). Evaluation of psychotherapy: Efficacy, effectiveness,

and patient progress. American Psychologist, 51, 1059�1064.

doi:10.1037/0003-066X.51.10.1059.

Hox, J. (2010). Multilevel analysis: Techniques and applications (2nd

ed.). Mahwah, NJ: Lawrence Erlbaum.

Kordy, H., Rad, M.V., & Senf, W. (1988). Time and its relevance

for a successful psychotherapy. Psychotherapy and Psychosomatics,

49, 212�222. Retrieved from http://content.karger.com/

ProdukteDB/produkte.asp?Aktion�JournalHome&Produkt

Nr�223864.

Kraft, S., Puschner, B., & Kordy, H. (2006). Treatment intensity

and regularity in early outpatient psychotherapy and its relation

to outcome. Clinical Psychology and Psychotherapy, 13, 397�404.

doi:10.1002/cpp.505.

Lambert, M.J., Hansen, N.B., Umphress, V., Lunnen, K., Okiishi,

J., Burlingame, G., . . . Reisinger, C. (1996). Administration and

scoring manual for the OQ 45.2. Stevenson, MD: American

Professional Credentialing Services.

Little, R.J.A. (1995). Modeling the drop-out mechanism in

repeated-measures studies. Journal of the American Statistical

Association, 90, 1112�1121. doi:10.2307/2291350.

Lutz, W., Lowry, J., Kopta, S.M., Einstein, D.A., & Howard, K.I.

(2001). Prediction of dose-response relations on patient

characteristics. Journal of Clinical Psychology, 57, 889�900.

doi:10.1002/jclp.1057.

Lutz, W., Martinovich, Z., & Howard, K.I. (1999). Patient

profiling: An application of random coefficient regression

models to depicting the response of a patient to outpatient

psychotherapy. Journal of Consulting and Clinical Psychology, 67,

571�577. doi:10.1037/0022-006X.67.4.571.

Minami, T., Davies, D.R., Tierney, S.C., Bettman, J.E., McA-

ward, S.M., Averill, L.A., . . . Wampold, B.E. (2009). Prelimin-

ary evidence on the effectiveness of psychological treatments

delivered at a university counseling center. Journal of Counseling

Psychology, 56, 309�320. doi: 10.1037/a0015398

Nagin, D. (2005). Group-based modeling of development. Cam-

bridge, MA: Harvard University Press.

Okiishi, J.C., Lambert, M.J., Eggert, D., Nielsen, L., Dayton,

D.D., & Vermeersch, D.A. (2006). An analysis of therapist

treatment effects: Toward providing feedback to individual

therapists on their clients’ psychotherapy outcome. Journal of

Clinical Psychology, 62, 1157�1172. doi:10.1002/jclp.20272.

Peugh, J.L. (2010). A practical guide to multilevel modeling.

Journal of School Psychology, 48, 85�112. doi:10.1016/

j.jsp.2009.09.002.

Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear

models: Applications and data analysis methods (6th ed.). New-

bury Park, CA: Sage.

Reardon, M.L., Cukrowicz, K.C., Reeves, M.D., & Joiner, T.E.J.

(2002). Duration and regularity of therapy attendance as

predictors of treatment outcome in an adult outpatient popula-

tion. Psychotherapy Research, 12, 273�285. doi:10.1093/ptr/

12.3.273.

Reese, R.J., Norsworthy, L.A., & Rowlands, S.R. (2009). Does a

continuous feedback system improve psychotherapy outcome?

Psychotherapy: Theory, Research, Practice, Training, 46, 418�431.

doi:10.1037/a0017901.

Sandell, R., Blomberg, J., & Lazar, A. (2002). Time matters: On

the temporal interactions in long-term follow-up of long-term

psychotherapies. Psychotherapy Research, 12, 39�59. doi:10.1093/

ptr/12.1.39.

Singer, J.D., & Willett, J.B. (2003). Applied longitudinal data

analysis: Modeling change and event occurrence. New York, NY:

Oxford University Press.

Shapiro, D.A., Barkham, M., Stiles, W.B., Hardy, G.E., Reynolds,

S., & Startup, M. (2003). Time is of the essence: A selective

review of the fall and rise of brief therapy research. Psychology

and Psychotherapy: Theory, Research and Practice, 76, 211�235.

doi:10.1348/147608303322362460.

Shimokawa, K., Lambert, M.J., & Smart, D.W. (2010). Enhan-

cing treatment outcome of patients at risk of treatment failure:

Meta-analytic and mega-analytic review of a psychotherapy

quality assurance system. Journal of Consulting and Clinical

Psychology, 78, 298�311. doi:10.1037/a0019247.

Spielmans, G.I., Masters, K.S., & Lambert, M.J. (2006).

A comparison of rational versus empirical methods in the

prediction of psychotherapy outcome. Clinical Psychology and

Psychotherapy, 13, 202�214. doi:10.1002/cpp.491.

Takuya, M., Davies, R.D., Tierney, S.C., Bettman, J.E., Mc-

Award, S.M., Averill, L.A., . . . Wampold, B.E. (2009).

Preliminary evidence of the effectiveness of psychological

treatments delivered at a university counseling center. Journal

of Counseling Psychology, 56, 309�320. doi:10.1037/a0015398.

Thompson, B. (1996). AERA editorial policies regarding statis-

tical significance testing: Three suggested reforms. Educational

Researcher, 25, 26�30. doi:10.3102/0013189X025002026.

Umphress, V.J., Lambert, M.J., Smart, D.W., Barlow, S.H., &

Clouse, G. (1997). Concurrent and construct validity of the

outcome questionnaire. Journal of Psychoeducational Assessment,

15, 40�55. doi:10.1177/073428299701500104.

Whipple, J.L., Lambert, M.J., Vermeersch, D.A., Smart, D.W.,

Nielsen, S.L., & Hawkins, E.J. (2003). Improving the effects of

psychotherapy: The use of early identification of treatment

failure and problem-solving strategies in routine practice.

Journal of Counseling Psychology, 50, 59�68. doi:10.1037/

0022-0167.50.1.59.

Psychotherapy dose-response 619

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a, S

an F

ranc

isco

] at

01:

59 1

6 D

ecem

ber

2014