running head: distress tolerance scale tables: 3

Running head: DISTRESS TOLERANCE SCALE

1

Tables: 3

Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis

Travis A. Rogers1, Joseph R. Bardeen1, Thomas A. Fergus2, & Natasha Benfer1

1Department of Psychology, Auburn University, United States

2Department of Psychology and Neuroscience, Baylor University, United States

NOTICE: this is the author’s version of a work that was accepted for publication in Assessment.

Changes resulting from the publishing process, such as peer review, editing, corrections,

structural formatting, and other quality control mechanisms may not be reflected in this

document. Changes may have been made to this work since it was submitted for publication. A

definitive version was subsequently published in Assessment.

Author Note

Correspondence concerning this article should be addressed to Joseph R. Bardeen,

Department of Psychology, Auburn University, Auburn, AL 36832. Voice: 334-844-6647; E-

mail: [email protected].

mailto:[email protected]

DISTRESS TOLERANCE SCALE

2

Abstract

The Distress Tolerance Scale (DTS) is a self-report measure of perceived capacity to withstand

aversive emotions. Initial factor analysis of this measure suggested a structure comprising one

higher-order factor and four lower-order domain-specific factors. However, there is limited

evidence in support of the DTS’s purported multidimensionality, and despite use of the DTS

subscales, research has yet to assess their incremental utility. The current investigation sought to

rectify the paucity of evidence in support of the DTS’s factor structure and independent use of

DTS subscales via bifactor analysis. In the present study (N = 826 community adults), a bifactor

model of the DTS provided the best fit to the data. However, an examination of statistical indices

associated with bifactor modeling, as well as results from an examination of incremental utility,

suggest that the domain-specific factors are largely redundant with the general factor and do not

provide incremental utility in predicting relevant clinical constructs beyond the general factor.

Measurement invariance between sexes was confirmed. Taken together, results support use of a

DTS total score, but not subscale scores.

Keywords: emotional distress intolerance, distress tolerance scale, bifactor analysis,

psychometrics


3

Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis

Distress tolerance is broadly defined as one’s capacity to withstand aversive physical and

psychological states (Leyro, Zvolesnky, & Bernstein, 2010). Although there is not yet consensus

regarding the operational definition of distress tolerance, compelling conceptualizations have

been put forward that propose that distress tolerance is made up of several distinct, but related,

facets (e.g., uncertainty, physical discomfort; Zvolensky, Vujanovic, Bernstein, & Leyro, 2010).

One facet of the broader construct of distress tolerance is emotional distress intolerance (EDI),

which reflects the capacity to withstand the distress associated with aversive emotions (Simons

& Gaher, 2005). Conceptual models of psychopathology position EDI as a central explanatory

factor in the pathogenesis and maintenance of a wide variety of pathological presentations

(Leyro, Zvolensky, & Bernstein, 2010; Zvolensky & Otto, 2010), including substance use

disorders, disordered eating, and posttraumatic stress (Bardeen & Fergus, 2016; Bernstein,

Vujanovic, Leyro, & Zvolensky, 2011; Vujanovc, Bernstein, & Litz, 2011). Moreover, EDI has

been identified as a potentially important mechanism of change in psychological interventions

for a wide variety of symptom presentations (see Zvolensky, Bernstein, & Vujanovic, 2011),

thus resulting in the development of clinical interventions that directly target EDI in the service

of symptom relief (e.g., Dialectal Behavior Therapy [DBT]; Linehan, 1993). Given EDI’s

transdiagnostic status and clinical utility, it is important to develop psychometrically sound

measures of this construct.

The assessment of EDI has garnered substantive interest in the extant literature, including

through the development of self-report and behavioral methods. As reviewed by Leyro et al.

(2010), Simons and Gaher's (2005) self-report measure, known as the Distress Tolerance Scale

(DTS), is the most commonly used index of EDI.1 Since Leyro et al.'s review, McHugh and Otto


4

(2012) developed a self-report measure that includes items from the DTS and other measures in

an attempt to refine the assessment of the construct. A limitation surrounding McHugh and Otto's

(2012) measure is that theoretically (e.g., Leyro et al., 2010) and empirically (e.g., Bardeen,

Fergus, & Orcutt, 2013b) distinct components of the distress tolerance construct are lumped

together to produce a composite score, thereby limiting the use of this measure as an index of

EDI per se. Veilleux, Pollert, Zielinski, Shaver, and Hill (in press) developed a behavioral

measure of EDI. Because behavioral methods often assess more "state-like" tendencies and self-

report measures often assess more "trait-like" tendencies, Veilleux et al.'s (in press) behavioral

measure holds promise for offering a complementary approach to the DTS for assessing EDI.

Overall, additional research is needed to identify optimal ways to assess EDI and, because of its

widespread use, further examination of the DTS is warranted, particularly related to the factor

structure and resulting operationalization of the DTS item scores.

The DTS was originally validated in two large, independent samples of university

students. Specifically, Simons and Gaher (2005) generated a pool of 16 items to be consistent

with their conceptualization of EDI. These items were submitted to an exploratory factor analysis

(EFA). Fifteen items were retained for further testing, and the EFA supported a four-factor

solution. Confirmatory factor analysis (CFA) supported a hierarchical structure, in which four

lower-order factors loaded onto a higher-order factor. The lower-order factors of the DTS

purportedly reflect (1) the perceived capacity to tolerate or endure negative emotions (i.e.,

Tolerance; “Feeling distressed or upset is unbearable to me”), (2) the extent to which attention is

occupied by emotional distress (i.e., Absorption; “When I feel distressed or upset, all I can think

about is how bad I feel”), (3) the subjective appraisal of negative emotions (i.e., Appraisal; “My


5

feelings of distress or being upset are not acceptable”), and (4) efforts aimed at managing

emotional distress (i.e., Regulation; “I’ll do anything to stop feeling distressed or upset”).

Psychometric support for scores on the 15-item measure has been provided, including

evidence of internal consistency for the total score (α range = .82 – .91) and subscale scores (α

range = .72 – .82; Leyro, Bernstein, Vujanovic, McLeish, & Zvolensky, 2011; Simons & Gaher,

2005), as well as retest reliability for the total score over a six-month interval (r = .61; Simons &

Gaher 2005). In addition, the DTS has been used across multiple domains of psychological

research, and evidence has accumulated in favor of (a) its concurrent validity (Bernstein,

Marshall, & Zvolensky, 2011; Simons & Gaher 2005), and (b) incremental utility in accounting

for variance within relevant clinical constructs (e.g., symptoms of bulimia, impulsivity) beyond

that explained by anxiety, depression, and other self-report and behavioral measures related to

EDI (e.g., Anestis et al., 2012). However, some evidence calls the validity of the subscales into

question. Cougle, Bernstein, Zvolensky, Vujanovic, and Mcatee (2013) found that the DTS total

score provided a significant incremental contribution in predicting tolerance and perceived threat

of anger, fear, sadness, and disgust in response to emotionally evocative film clips, even after

accounting for sex, emotional intensity, and anxiety sensitivity. In contrast, DTS subscales

exhibited inconsistent associations with constructs to which they should be theoretically related.

For example, the DTS Tolerance subscale was incrementally associated with perceived threat

and tolerance for the anger film clip, but not for three other film clips used to elicit sadness, fear,

and disgust. Such results run counter to the generally supported validity of these sub-domains of

emotional distress intolerance.

Cougle et al.’s (2013) results notwithstanding, the extant literature has been largely

supportive of the utility of the DTS total and subscale scores. However, the structure of this


6

measure has yet to be adequately examined (see Table 1). Leyro et al. (2011) provided the only

known CFA of the DTS after its initial publication and found several areas of possible model

strain that raise concerns about (a) the conceptual distinctions between the DTS factors (i.e.,

multiple potential cross loadings, strong inter-factor correlations [rs > .80]), and (b) the under-

performance of some DTS items relative to others (i.e., standardized item-factor loading range =

.29 – .86). Furthermore, Leyro et al. used a 14-item version of the DTS that was developed prior

to Simons and Gaher’s (2005) publication. Specifically, item 14 of the Regulation subscale

(“When I feed distressed or upset, I must do something about it immediately”) was omitted.

Leyro et al. also found that correlations between three of the DTS subscales (Tolerance,

Absorption, and Appraisal) were strong enough (rs > .80) to raise concerns about the conceptual

distinctions between them.

Following Leyro et al.’s (2011) CFA, Hsu, Collins, and Marlatt (2013) examined the

factor structure of the DTS via principal component analysis (PCA). Contrary to Simons and

Gaher’s (2005) original publication, Hsu et al.’s results yielded one unitary factor, suggesting

that the DTS might be better modeled as a unidimensional measure of EDI. It should also be

noted that PCA is primarily a data reduction technique (see Abdi & Williams, 2010). Thus, Hsu

et al.’s results are less likely to generalize to other factor analytic work (see Suhr, 2005). Put

succinctly, these limited investigations leave a number of questions about the structure of the

DTS unanswered.

The paucity of empirical evidence in support of the factor structure of the DTS remains a

significant limitation to future work with this measure. Studies using the DTS have tended either

to use the DTS total score as the sole index of EDI (e.g., Anestis, Bagge, Tull, & Joiner, 2011;

Anestis et al., 2012; Bernstein, Marshall, et al., 2011; Vujanovic, Bonn-Miller, Potter, Marshall,


7

& Zvolensky, 2011) or to interpret both the total and subscale scores (e.g., Ameral, Palm Reed,

Cameron, & Armstrong, 2014; Leyro et al., 2011; Simons & Gaher, 2005; Shorey et al., 2017;

Timpano, Buckner, Richey, Murphy, & Schmidt, 2009). These approaches make at least two

assumptions about the nature of the DTS. First, use of a total score assumes that each DTS

domain-specific factor represents an aspect of the same overarching general construct. Second,

use of subscale scores assumes that the lower-order factors capture unique information beyond

the total score. However, the existing literature base for the DTS has yet to adequately address

the tenability of these two approaches. While investigation of the purported hierarchical structure

of the DTS is useful for examining the first of the two assumptions, it cannot directly test the

assumption that items of domain-specific factors provide unique information beyond a higher-

order or general factor (Reise, 2012). Such analyses are necessary because, as noted, a number of

investigations have used DTS subscales and found differential relations with constructs of

interest. Thus, it is important to use an analytic approach that can simultaneously address both of

these assumptions (i.e., bifactor analysis).

Bifactor models permit investigation of the presence of a general factor and the degree to

which each domain-specific factor is meaningfully distinct from the general factor (Reise,

Bonifay, & Haviland, 2013; Reise, Moore, & Haviland, 2010; Rodriguez, Reise, & Haviland,

2016). Moreover, bifactor models have demonstrated good fit to transdiagnostic self-report

measures in prior research (e.g., Ebesutani, McLeish, Luberto, Young, & Maack, 2014;

Ebesutani et al., 2011). To our knowledge, the DTS has yet to be examined via a bifactor

modeling approach. Such an analysis is important and would likely yield new insights into the

performance of the DTS. For example, redundancy between a general factor and domain-specific

factors may suggest that a total score should be used in lieu of subscale scores (Reise, 2012).


8

Therefore, the primary aim of the current investigation was to examine the fit of a bifactor model

of the DTS in a large adult sample. Consistent with standard methodology for comparing model

fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model to competing models

(i.e., a correlated four-factor model, a one-factor model, and a hierarchical model). Some have

suggested that the utility of comparing bifactor models to alternative models is limited because

bifactor models often provide better fit to the data due to their inherent qualities (e.g., high

flexibility; Bonifay, Lane, & Reise, 2017; Reise, Kim, Mansolf, & Widaman, 2016). As such, a

number of additional statistical indices, developed for use with the bifactor approach, were

examined (Rodriguez, Reise, & Haviland, 2016). These statistics provide information central to

the aims of the present study (e.g., determining the degree to which domain-specific factors have

value beyond the general factor, determining the stability and replicability of the factors).

An additional benefit of bifactor modeling is that the relation between domain-specific

factors and criterion variables can be examined while holding the general factor constant

(Brown, 2015). Thus, we also examined the incremental utility of the DTS domain-specific

factors in predicting theoretically relevant constructs (i.e., depression, anxiety, and stress) after

accounting for a general EDI factor. Depression, anxiety, and stress were selected as outcome

variables because research suggests that emotional distress intolerance plays an important role in

the etiology and maintenance of emotional disorders (e.g., Bernstein, Vujanovic, et al., 2011;

McHugh et al., 2014).

Another unaddressed question regarding the structure of the DTS relates to whether it is

similar between males and females. In the original validation study, females reported

significantly poorer tolerance for emotional distress than males (Cohen’s d = 0.32; Simons &

Gaher, 2005). This effect remained significant in a regression analysis partialling out the shared


9

variance between sex and negative affectivity, indicating that mean differences in DTS total

scores could not be attributed to differences in negative affect. This difference may reflect either

a true difference in emotional distress tolerance between males and females or may be a function

of item content that promotes differential responding. Significant mean differences across sexes

have been reported in other investigations (e.g., Cougle et al., 2013; Dennhardt & Murphy, 2011)

and have indicated that females tend to report poorer emotional distress tolerance than males. An

investigation of sex invariance within the DTS can speak to the potential need for differential

scoring and modeling of DTS items among females and males. For example, if the loadings of

DTS items on a general factor and domain-specific factors differ as a function of sex, it may be

possible that certain DTS domain-specific factors are independent and incrementally useful in

one sex but not the other. Therefore, in addition to an examination of (a) the purported

multidimensionality of the DTS and (b) the viability of using the DTS domain-specific factors in

predicting clinically relevant constructs, the present study also sought to investigate (c)

measurement invariance of DTS items across males and females.

Methods

Participants and Procedure

One thousand and nine participants were recruited using Amazon’s Mechanical Turk

(MTurk), an online labor market where individuals can participate in research in exchange for

financial compensation. Previous research has supported the use of MTurk samples, citing their

reliability and diverse demographics relative to other online and undergraduate samples (e.g.,

Behrend, Sharek, Meade, & Wiebe, 2011; Buhrmester, Kwang, & Gosling, 2011; Chandler &

Shapiro, 2016). To minimize the effect of random responding or inattentiveness, three catch

questions (e.g., “Select ‘Much’ if you are paying attention right now”) were imbedded in the


10

online survey (Oppenheimer, Meyvis, & Davidenko, 2009; Paolacci, Chandler & Ipeirotis,

2010). Participants who answered fewer than two of the three catch questions correctly (n = 144;

14.27% of original sample) were excluded from analyses (for precedence, see Bardeen, Fergus,

Hannan, & Orcutt, 2016; Bardeen, Fergus, & Orcutt, 2013a, 2013b; Bardeen & Michel, 2017).

Additionally, 39 participants (3.87% of the original sample) who failed to provide responses for

the majority of DTS items (eight out of fifteen [at least 50%; see Graham & Schafer, 1999]) were

removed from the sample. The proportion of missing cases within all variables of interest was ≤

1%, and covariance coverage was high across all variables, ranging from .985 to 1.00. The final

sample (N = 826) was 60.5% female. Five participants elected not to report their sex. The

average age of the sample was 33.68 years (SD = 12.54, range = 18 to 73). Concerning race,

81.6% of the final sample identified as White, 6.5% as Black, 6.3% as Asian, 1.0% as American

Indian or Alaskan Native, 0.1% as Native Hawaiian or Pacific Islander, 2.8% as “Other,” and

1.7% preferred not to respond. Additionally, 89.1% of the final sample identified as Non-

Hispanic, 6.8% as Hispanic, and 4.1% preferred not to report ethnicity.

All study procedures were approved by the local institutional review board. Informed

consent and self-report measures could be completed from any computer with internet access.

The median time to complete the online battery was 29.19 minutes. Upon completion,

participants were compensated with $0.50, an amount consistent with compensation provided to

MTurk workers who completed studies of similar length (e.g., Buhrmester et al., 2011).

Measures

Distress Tolerance Scale (DTS). As described, the DTS (Simons & Gaher 2005) is a 15-

item measure designed to assess individual differences in EDI. Items are rated on a 5-point scale

from 1 (Strongly agree) to 5 (Strongly disagree). Higher total scores are indicative of greater


11

tolerance for emotional distress, while lower scores reflect a relative intolerance of emotional

distress.

The Depression Anxiety Stress Scales – 21-item version (DASS-21). The DASS-21

(Lovibond & Lovibond, 1995) is a 21-item measure with subscales assessing symptoms of

depression, anxiety, and stress during the past week. Each subscale comprises seven items rated

on a 4-point Likert scale ranging from 0 (Did not apply to me at all) to 3 (Applied to me very

much, or most of the time); thus, higher scores suggest greater symptomatology. The DASS-21

has evidenced good psychometric properties in a number of studies (Bardeen, Fergus, & Orcutt,

2014; Henry & Crawford, 2005; Lovibond & Lovibond, 1995) and has evidenced strong

convergent validity with other measures of depression and anxiety (Antony, Bieling, Cox, Enns,

& Swinson, 1998). The DASS-21 subscales evidenced good internal consistency in the current

sample, with Cronbach’s αs of .93, .90, and .86 for depression, stress, and anxiety, respectively.

Data Analytic Strategy

Confirmatory Factor Analysis. The following four models were examined via CFA.

The first model was a correlated four-factor model proposed by Simons and Gaher (2005), with

three items loading onto the Tolerance factor, three items loading onto the Absorption factor, six

items loading onto the Appraisal factor, and three items loading onto the Regulation factor.

Correlations between all four factors were modeled, and no item had a secondary loading.

Standard procedures for conducting model comparisons, especially for measures that produce a

total score, requires examination of a one-factor model to ensure that a unidimensional model

does not provide better fit to the data (Brown, 2015; Reise et al., 2010). As such, the second

model was a one-factor model, within which all items of the DTS loaded onto one factor. The

third model was a hierarchical model, in which correlations between factors were omitted and


12

direct pathways were modeled from a higher-order factor onto each domain-specific factor (i.e.,

identical to that proposed in the original DTS publication). The fourth model was a bifactor

model, in which all 15 items loaded onto a general factor as well as onto their respective domain-

specific factor, and covariance terms between all factors were held to zero (Brown, 2015).

Model Estimation and Comparison. All models were tested using MPlus 7.4 (Muthén

& Muthén, 2015). Missing data were handled using full information maximum likelihood, which

is the default method for imputing missing data in MPlus. Robust maximum likelihood (MLR)

estimation was used to test all models, as MLR is robust to violations of the assumption of

normality (Brown, 2015; Kaplan, 2009). Therefore, all reported chi-square (χ2) values represent

the Satorra-Bentler scaled χ2 (Satorra & Bentler, 1994). Four commonly recommended fit

statistics were used to evaluate the fit of each model to the data: the comparative fit index (CFI),

the Tucker-Lewis fit index (TLI), standardized root mean square residual (SRMR), and root

mean square error of approximation (RMSEA; Brown, 2015; Kline, 2016). The following

guidelines were used to evaluate model fit: CFI and TLI should be near .95; SRMR should be

less than .08 (Hu & Bentler, 1999); RMSEA should be near .06, and the upper limit of the 90%

RMSEA confidence interval (CI) should not exceed .10 (Kline, 2016). In addition to an

evaluation of model fit, model comparisons were evaluated by way of scaled χ2 difference

testing. However, due to the sensitivity of χ2 distributions to large samples, χ2 difference testing

might suggest significant differences between models when differences in the magnitude of

parameter estimates are trivial (Brown, 2015; Kline, 2016). Thus, alternative tests for comparing

model fit were used, including an examination of differences in sample-size corrected Akaike

Information Criterion (AICc; Akaiki, 1974; Sugiura, 1978), which provides a more quantitative

rather than qualitative (e.g., better/worse) estimate of model comparison. Additionally, RMSEA


13

90% CIs were examined. Relative differences in model fit were considered meaningful if models

differed in AICc by 10 or more (see Burnham, Anderson, & Huyvaert, 2011, for a discussion),

and had non-overlapping RMSEA 90% CIs (e.g., Wang & Russell, 2005).

Bifactor Model Evaluation. In addition to a basic examination of fit statistics between

models, the following statistical indices were examined to further evaluate the bifactor model

(see Rodriquez et al., 2016). All additional bifactor model indices were calculated using the

bifactor indices calculator provided by Dueber (2016). OmegaH (ωH) reflects the proportion of

variance in DTS scores attributable to a general factor, while OmegaHS (ωHS) reflects the

proportion of variance in scores explained by each domain-specific factor after removing

variance explained by the general factor. Both ωH and ωHS are best understood as indices of

factor reliability. Explained common variance (ECV) is calculated by dividing variance

attributable to the general factor by variance attributable to both general and specific factors;

thus, ECV serves as a better index of unidimensionality within a measure than ωH. Percentage of

uncontaminated correlations (PUC) is interpreted along with ECV and serves as an indicator of

the percentage of DTS item correlations contaminated by variance attributed to the general and

domain-specific factors. Item-level ECV (I-ECV) indicates the amount of common variance

from each item that is attributable to the general factor (Stucky, Thissen, & Edelen, 2013), thus

serving as an index of unidimensionality at the item level. Construct replicability (H) reflects the

degree to which a factor is well defined by its indicators. Factor determinacy (FD) is the

correlation between factor scores and the factors and serves as an indication of whether factor

scores are valid for independent use. Finally, average relative parameter bias (ARPB) serves as

an indication of bias across parameters when items are forced into a unidimensional structure.


14

Structural Regression Model. A structural regression model was used to examine

whether the domain-specific factors of the DTS relate to clinically relevant constructs when

simultaneously accounting for a general emotional distress intolerance factor. Specifically, the

general factor and domain-specific factors from the bifactor model were simultaneously

regressed onto the three uncorrelated domains of the DASS-21. Domain-specific factors from the

bifactor model that exhibited clear redundancy with the general factor (e.g., non-significant

residual variance) were removed from the model. Path coefficients from the general factor and

remaining domain-specific factor(s) to each of the DASS scales were freely estimated.

Measurement Invariance. A multiple-group CFA was used to test measurement

invariance, with restrictive models testing for (1) configural invariance (i.e., equal structural

form), (b) metric invariance (i.e., equal factor loadings), and (c) scalar invariance (i.e., equal

indicator intercepts) between males (n = 321) and females (n = 500). To test configural

invariance, the adequacy of the final DTS bifactor structure was examined independently and

simultaneously in both sexes. The test of metric invariance constrained the factor loadings within

the final DTS bifactor model to equality across both sexes. Scalar invariance was examined by

constraining indicator intercepts to equality across sexes. Because χ2 is sensitive to large samples

and interpretation of multiple-group CFA results is complicated by unequal group sizes (Brown,

2015), RMSEA 90% CIs and CFI were also examined between invariance models. Decrements

in model fit were considered meaningful when parameter constraints resulted in either (a) non-

overlapping RMSEA 90% CIs or (b) changes in CFI (∆CFI) greater than .01 (Cheung &

Rensvold, 2002).

Results


15

Model Estimation and Comparison. Goodness-of-fit statistics from the measurement

models are presented in Table 2. The correlated four-factor model evidenced adequate fit to the

data. CFI and TLI approximated the specified criteria of adequate model fit, and RMSEA,

RMSEA 90% CIs, and SRMR of the four-factor model indicated good fit. As the one-factor

model is nested within the correlated four-factor model (Brown, 2015), the one-factor model was

examined next for comparative purposes. Relative to the four-factor model, the one-factor model

provided significantly worse fit to the data, as evidenced by a significant change in χ2, ∆χ2(6) =

296.66, p < .001, differences in AICc of 528.70, and non-overlapping RMSEA 90% CIs. The

hierarchical model was fitted to the data next. CFI and TLI approximated the specified criteria of

adequate model fit, and RMSEA, RMSEA 90% CIs, and SRMR indicated good fit. In

comparison to the correlated four-factor model, there was a significant change in χ2, ∆χ2(2) =

13.15, p < .01, and meaningful change in AICc of 15.23. In contrast, RMSEA 90% CIs of the

correlated four-factor and hierarchical models were overlapping.

The bifactor model provided adequate to good fit to the data; all fit indices met or

approximated specified criteria. Hierarchical models are nested within bifactor models and

therefore can be directly compared (Brown, 2015). χ2, AICc values, and RMSEA 90% CIs were

compared across models. χ2 difference testing indicated a significant difference between the fit

of the (a) correlated four-factor and bifactor models, ∆χ2(9) = 42.78, p < .001, (b) one-factor and

bifactor models, ∆χ2(15) = 347.00, p < .001, and (c) hierarchical and bifactor models, ∆χ2(11) =

55.46, p < .001. Differences in AICc values between the correlated four-factor and bifactor

models, ∆AICc = 53.91, between the one-factor and bifactor models, ∆AICc = 582.61, and

between the hierarchical and bifactor models, ∆AICc = 69.14, also favored the bifactor model.

However, RMSEA 90% CIs were overlapping between all models. As two of three comparative


16

indices suggested better fit in the bifactor model, it was retained for further examination.

The standardized factor loadings from the bifactor model are presented in Table 3. While

nine of 15 items exhibited significant factor loadings (p < .05) onto their domain-specific factors,

factor loadings of all items of the Absorption factor and items 6, 9, and 10 of the Appraisal factor

were rendered non-significant in the bifactor model. In contrast, all items loaded significantly

onto the general factor. Redundancy between the Tolerance and Absorption factors and the

general factor was evidenced by non-significant residual variances within the bifactor model.

The Appraisal and Regulation factors were the only domain-specific factors to exhibit significant

residual variance.

Bifactor Model Evaluation. Additional indices derived from the bifactor model are

presented in Table 3. According to Stucky and Edelen (2015), I-ECV values greater than .80 or

.85 indicate unidimensionality at the item level. The majority of DTS items (nine out of 15)

demonstrated I-ECV values greater than .80. Within the domain-specific factors, two out of three

Tolerance items, two out of three Absorption items, and five out of six Appraisal items,

evidenced content redundancy with the general factor. In contrast, the Regulation factor was the

only domain-specific factor whose items demonstrated domain-specific content, as evidenced by

I-ECV values ≤ .72. As reviewed by Rodriguez et al. (2016), ω and ωS serve as indices of the

reliability of the general and domain-specific factors, respectively. Adequate reliability was

demonstrated by both the general factor (ω = .95) and domain-specific factors (ωS range = .83 –

.88). The general factor explained a substantial amount of variance in DTS scores (ωH = .91).

However, after partitioning out variance explained by the general factor, the domain-specific

factors accounted for a small proportion of variance within their respective items (ωHS range =

.07 – .35).


17

According to Rodriguez et al. (2016), ECV and PUC reflect the degree to which

parameter estimates are biased when multidimensional constructs are forced into a

unidimensional model. When ECV and PUC are greater than .70, common variance within a

model can be regarded as essentially unidimensional. Within the bifactor model, ECV of the

general factor was .80, and PUC was .77, supporting a unidimensional rather than

multidimensional conceptualization of the DTS. When H is high (H > .70; see Hancock &

Mueller, 2001), a latent variable may be considered well defined by its items, and thus, will

demonstrate greater stability across studies. Within the bifactor model, the general factor

evidenced acceptable construct replicability, H = .95. However, no domain-specific factor

demonstrated adequate construct replicability, with H values ranging from .25 to .55. Moreover,

Gorsuch (1983) recommends that factor score estimates should be used only when FD is greater

than .90. Per this criterion, the general factor estimate may be interpreted, with FD = .96.

However, no domain-specific factor met Gorsuch’s guideline, with FD values from domain-

specific factors ranging from .67 to .84. Lastly, ARPB of the bifactor model was .03, suggesting

minimal differences between parameter estimates in a unidimensional and bifactor solution.

ARPB less than 0.10 or 0.15 suggests that multidimensionality within a measure is not

substantial enough to negate the tenability of a unidimensional solution (Muthén, Kaplan, &

Hollis, 1987; Rodriquez et al., 2016).

Modification indices derived from the bifactor model yielded 12 potential item cross-

loadings (i.e., expected change in parameter estimates of > 10; Brown & Moore, 2012; Muthén

& Muthén, 2015). Specifically, allowing all three Tolerance items, all three Absorption items,

two of the Appraisal items (items 7 and 10), and item 13 of Regulation to cross-load onto another

domain-specific factor would substantially improve model fit. Modification indices also


18

suggested that model fit would benefit from allowing three items (2, 7, and 15) to cross-load on

three factors. These results suggest that DTS items may reflect unidimensional rather that

domain-specific content.

Structural Regression Model. Because three of the four DTS domain-specific factors

(Tolerance, Absorption, and Appraisal; see Table 3) exhibited poor performance within the

bifactor model (i.e., non-significant residual variance and/or a preponderance of items that

exhibit content redundancy with the general factor), only the Regulation factor and general factor

were included as predictors in the structural regression.2 The structural regression model

provided adequate fit to the data, with all fit indices approximating specified guidelines: χ2(585)

= 1981.16, p < .001; RMSEA = .05 (90% CIs = .051 – .056); CFI = .91; TLI = .90; SRMR = .05.

The general factor significantly predicted all outcome variables at p < .001, with standardized

coefficient estimates in the expected direction: depression, β = -0.46; anxiety, β = -0.38; stress, β

= -0.41. After accounting for the general factor, the Regulation factor also significantly predicted

all outcome variables at p < .001; however, all standardized coefficient estimates were in a

theoretically inconsistent direction: depression, β = 0.76; anxiety, β = 0.83; stress, β = 0.84.3

In order to test for possible net suppression effects, the structural regression was

modified, such that depression, anxiety, and stress were regressed onto only the Regulation

domain-specific factor (i.e., the general factor was modeled but did not serve as a predictor of

these outcomes). This second structural regression yielded a pattern of results in opposition to

the first. Significant negative associations between the Regulation factor and criterion variables

(depression, β = -0.86; anxiety, β = -0.91; stress, β = -0.94, ps < .001) suggest that the

unexpected positive relationships between the Regulation domain-specific factor and these

clinical outcomes, when controlling for the general factor, most likely represent a suppression


19

effect. Further, the high standardized beta weights between this domain-specific factor and

criterion variables suggests construct overlap.4, 5

Measurement Invariance. The final DTS bifactor model (the general factor and the

Regulation domain-specific factor) was retained for an analysis of measurement invariance. Fit

indices of the configural, metric, and scalar invariance models are displayed in Table 2. The

bifactor model exhibited adequate fit in both males and females when modeled separately and

simultaneously (i.e., evidence of configural invariance). While there was a change in AICc

greater than 10, no significant decrement in model fit was observed between the configural and

metric models as evidenced by Δχ2 (Δχ2[16] = 14.56), overlapping RMSEA 90% CIs, and ∆CFI

of .002. χ2 differences between the metric and scalar models (Δχ2[13] = 41.76) and the scalar and

configural models (Δχ2[29] = 58.70) were significant at p < .001. However, RMSEA CIs across

all invariance models overlapped, and ∆CFI did not exceed Cheung and Rensvold’s (2002)

guideline. Thus, overlapping RMSEA CIs and ∆CFI < .01 were interpreted as evidence of scalar

invariance between males and females. In addition, latent mean scores of the DTS general factor

and Regulation domain-specific factor were compared between sexes. Significant differences

were not observed when comparing the latent general factor (difference = 0.11, p = .14) and

latent Regulation factor means (difference = 0.06, p = .17) between males and females.

Discussion

The primary aim of this investigation was to address the paucity of empirical evidence in

support of the multidimensional structure of the DTS. Consistent with standard methodology for

comparing model fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model of the

DTS to a correlated four-factor model, a one-factor model, and a hierarchical model, and thereby

moved beyond the measurement models examined in previous studies. The adequacy of the


20

hierarchical model proposed in the original publication of the DTS (Simons & Gaher, 2005) was

replicated in this study; however, the hierarchical model and correlated four-factor model did not

differ in their fit to the data. The bifactor model of the DTS, in which all indicators loaded onto a

general factor in addition to their respective orthogonal domain-specific factors, evidenced better

fit relative to all other tested models. In evaluating the bifactor model further, item loadings onto

the domain-specific factors were substantially attenuated, and residual variance of two of the

four domain-specific factors (i.e., Tolerance and Absorption) was rendered non-significant after

accounting for the general factor. Moreover, the majority of DTS items demonstrated content

redundancy at the item level. The general factor accounted for most of the variance in DTS

scores, and no domain-specific factor met specified benchmarks of adequate construct

replicability or factor determinacy. Succinctly, results provide support for a strong general factor

within the DTS but do not support the use of purported domain-specific factors.

This investigation also sought to test the incremental utility of the DTS domain-specific

factors in predicting clinical constructs after accounting for the general factor. To that end, a

structural regression model, in which depression, anxiety, and stress served as outcome variables,

was performed. The Regulation factor was the only domain-specific factor whose residual

variance remained significant in the bifactor model and whose items could be regarded as

providing enough unique variance beyond that of the general factor to be considered a separate

domain. Thus, only the Regulation factor and the general factor served as predictors in the

structural regression. The general factor significantly predicted depression, anxiety, and stress,

indicating that emotional distress intolerance shares a significant relationship with these clinical

outcomes.


21

After accounting for the general factor, the Regulation factor of the DTS also

significantly predicted depression, anxiety, and stress, but in a theoretically inconsistent

direction. These unexpected results necessitated an examination of net suppression. While

domain-specific factors cannot exhibit suppression effects on other domain-specific factors due

to their orthogonal specification in the bifactor model, content redundancy may still emerge from

the simultaneous loadings of all items on both the general factor and respective domain-specific

factors. In other words, the Regulation factor may still have limited practical value on its own.

As reviewed by Paulhus, Robins, Trzesniewski, and Tracy (2004), suppression and/or

redundancy effects can occur in regression models when multiple predictor variables with strong

correlations are entered simultaneously (for an example of the effect of suppression in a bifactor

framework, see Patrick, Hicks, Nichol, & Krueger, 2007). All latent factors of the DTS have

evidenced strong, positive correlations in past studies (e.g., Ameral et al., 2014; Leyro et al.,

2011), and standardized correlations between the domain-specific factors in the current study

were also quite high, falling within a range of .72 to .94. A second structural regression in which

variance from the DTS general factor was not controlled for yielded a pattern of results in

opposition to the first regression, providing strong evidence for net suppression. Further, the

magnitude of the associations between the Regulation domain-specific factor and our outcome

variables suggests content redundancy. That is, the Regulation factor likely has little to no

incremental utility beyond the general factor in predicting depression, anxiety, or stress, because

it may in fact be redundant with measures of these clinically relevant outcomes. Results such as

these are not new to researchers of individual differences (see Wolgast, 2014).

This study provided the first investigation of measurement invariance in the DTS across

males and females. Past research has suggested that females may demonstrate lower scores on


22

this measure (i.e., poorer tolerance of emotional distress) than males (Cougle et al., 2013;

Dennhardt & Murphy, 2011). To our knowledge, however, no other investigation has sought to

explore whether this difference reflects a true deficit in emotional distress tolerance in females or

measure-specific variance of the DTS items across sexes. Our results provide evidence that the

DTS exhibits adequate measurement invariance across males and females and supports the use of

identical scoring and modeling of the DTS in both sexes. Contrary to past findings, no significant

differences were found in the latent means of the DTS’s general factor or Regulation domain-

specific factor between sexes.

Results of the current investigation support a unidimensional conceptualization of the

DTS, contraindicate independent use of the DTS domain-specific factors, and provide evidence

of measurement invariance across males and females. However, it should be noted that fit of the

unidimensional model was relatively poor, suggesting a potential need for improvements to this

commonly used measure of emotional distress intolerance. For instance, given that modification

indices suggested multiple cross loadings, and that two of the four domain-specific factors

exhibited non-significant residual variance after modeling the general factor (i.e., redundant item

content), future investigations might prioritize refining the DTS by removing truly redundant and

underperforming items. Refinement of the DTS in this way would likely result in a shorter, more

parsimonious measure with greater ease of interpretability and potentially better unidimensional

model fit.

Results should be viewed in light of certain limitations. First, although research supports

the use of MTurk for data collection (Chandler & Shapiro, 2016), and a substantial amount of

literature exists to aid researchers in quality-control procedures and other issues related to online

data collection (see Paolacci et al., 2010, for a discussion), MTurk samples should not be


23

considered representative of the general population. For example, evidence suggests that MTurk

samples tend to be more highly educated and younger than the general population (Paolacci &

Chandler, 2014). As such, replication of study results in both general population and clinical

samples, with more individuals scoring at the high end of the continuum of anxiety and

depression, will help to ensure that results generalize. Second, the current sample, though large,

comprised a female majority and was relatively homogeneous in terms of race and ethnicity. The

acceptability of online sampling notwithstanding, future research with the DTS would do well to

examine the psychometric performance of this measure and to cross-validate its structure in

samples with greater diversity. Third, the current investigation was cross-sectional in design.

Future research would benefit from longitudinal data and experimental paradigms that allow

researchers to better characterize the relationship between emotional distress intolerance and

relevant clinical constructs. For example, research designs that recruit participants prior to

developmentally sensitive periods (e.g., late adolescence, early 20s) and follow such participants

across time would be better suited to examine the prospective relationship between emotional

distress intolerance and the psychogenesis of various forms of psychopathology. Further, in line

with a diathesis-stress model, longitudinal designs across multiple time points could help to

expound on past findings related to emotional distress intolerance and the occurrence of stressful

life events (e.g., traumatic exposure and the development of PTSD).

Lastly, an additional limitation of the current study pertains to the growing body of

research on distress tolerance as a multifaceted, transdiagnostic individual difference factor in

the etiology and maintenance of psychopathology. As reviewed by Zvolensky, Vujanovic,

Bernstein, and Leyro (2010), distress tolerance is believed to comprise five facets of experience

that elicit psychological distress: uncertainty, ambiguity, frustration, physical discomfort, and


24

emotional distress (for empirical support for this conceptualization, see Bardeen et al., 2013b).

The DTS was developed to reflect only emotional distress intolerance, and thus, represents only

a part of the expansive work on the broader construct of distress tolerance. The purpose of this

study was to evaluate the psychometric performance of the DTS within different models and

across sexes. In doing so, we have contributed to this expansive literature base, but must

acknowledge the narrowness of our focus relative to the breadth of the distress tolerance

literature.


25

References

Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary

Aeviews: Computational Statistics, 2(4), 433-459.

Akaiki, H. (1974). A new look at the statistical model identification. IEEE Transactions on

Automatic Control, 19, 716-723.

Ameral, V., Palm Reed, K. M., Cameron, A., & Armstrong, J. L. (2014). What are measures of

distress tolerance really capturing? A mixed methods analysis. Psychology of

Consciousness: Theory, Research, and Practice, 1, 357-369.

Anestis, M. D., Bagge, C. L., Tull, M. T., & Joiner, T. E. (2011). Clarifying the role of emotion

dysregulation in the interpersonal-psychological theory of suicidal behavior in an

undergraduate sample. Journal of Psychiatric Research, 45, 603-611.

Anestis, M. D., Lavender, J. M., Marshall-Berenz, E. C., Gratz, K. L., Tull, M. T., & Joiner, T.

E. (2012). Evaluating distress tolerance measures: Interrelations and associations with

impulsive behaviors. Cognitive Therapy and Research, 36, 593-602.

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric

properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in

clinical groups and a community sample. Psychological Assessment, 10, 176-181.

Bardeen, J. R., & Fergus, T. A. (2016). Emotional distress intolerance, experiential avoidance,

and anxiety sensitivity: The buffering effect of attentional control on associations with

posttraumatic stress symptoms. Journal of Psychopathology and Behavioral Assessment,

38, 320-329.


26

Bardeen, J. R., Fergus, T. A., Hannan, S. M., & Orcutt, H. K. (2016). Addressing psychometric

limitations of the Difficulties in Emotion Regulation Scale through item modification.

Journal of Personality Assessment, 98, 298–309.

Bardeen, J. R. Fergus, T. A., & Orcutt, H. K. (2013a). Experiential avoidance as a moderator of

the relationship between anxiety sensitivity and perceived stress. Behavior Therapy 44,

459-469.

Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2013b). Testing a hierarchical model of distress

tolerance. Journal of Psychopathology and Behavioral Assessment, 35, 495-505.

Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2014). The moderating role of experiential

avoidance in the prospective relationship between anxiety sensitivity and

anxiety. Cognitive Therapy and Research, 38, 465-471.

Bardeen, J. R., Kumpula, M. J., & Orcutt, H. K. (2013). Emotion regulation difficulties as a

prospective predictor of posttraumatic stress symptoms following a mass shooting.

Journal of Anxiety Disorders, 27, 188-196.

Bardeen, J. R., & Michel, J. S. (2017). The buffering effect of religiosity on the relationship

between intolerance of uncertainty and depressive symptoms. Psychology of Religion and

Spirituality, 9, 90-95.

Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of

crowdsourcing for survey research. Behavior Research Methods, 43, 800-813.

Bernstein, A., Marshall, E. C., & Zvolensky, M. J. (2011). Multi-method evaluation of distress

tolerance measures and construct(s): Concurrent relations to mood and anxiety

psychopathology and quality of life. Journal of Experimental Psychopathology, 2(3),

386-399.


27

Bernstein, A., Vujanovic, A. A., Leyro, T., & Zvolensky, M. J. (2011). Distress tolerance:

Research synthesis and future directions. In M. J. Zvolensky, A. Bernstein, & A.

Vujanovic (Eds.), Distress tolerance: Theory, research, and clinical applications (pp.

263-280). New York, NY: Guilford Press.

Bonifay, W., Lane, S. P., & Reise, S. P. (2017). Three concerns with applying a bifactor model

as a structure of psychopathology. Clinical Psychological Science, 5, 184-186.

Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York,

NY: Guilford Press.

Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selection and

multimodel inference in behavioral ecology: some background, observations, and

comparisons. Behavioral Ecology and Sociobiology, 65, 23-35.

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source

of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced

convenience samples. Annual Review of Clinical Psychology, 12, 53-81.

Cheung, C. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing

measurement invariance. Structural Equation Modeling, 9, 233-255.

Corstorphine, E., Mountford, V., Tomlinson, S., Waller, G., & Meyer, C. (2007). Distress

tolerance in the eating disorders. Eating Behaviors, 8, 91-97.

Cougle, J. R., Bernstein, A., Zvolensky, M. J., Vujanovic, A. A., & Macatee, R. J. (2013).

Validation of self-report measures of emotional and physical distress tolerance. Journal

of Psychopathology and Behavioral Assessment, 35, 76-84.


28

Dennhardt, A. A., & Murphy, J. G. (2011). Associations between depression, distress tolerance,

delay discounting, and alcohol-related problems in European American and African

American college students. Psychology of Addictive Behaviors, 25, 595-604.

Dueber, D. M. (2016, December). Bifactor Indices Calculator: A Microsoft Excel-based tool to

calculate various indices relevant to bifactor CFA models. Available at

http://sites.education.uky.edu/apslab/resources.

Ebesutani, C., McLeish, A. C., Luberto, C. M., Young, J., & Maack, D. J. (2014). A bifactor

model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of

Psychopathology and Behavioral Assessment, 36, 452-464.

Ebesutani, C., Smith, A., Bernstein, A., Chorpita, B. F., Higa-McMillan, C., & Nakamura, B.

(2011). A bifactor model of negative affectivity: Fear and distress components among

younger and older youth. Psychological Assessment, 23, 679-691.

Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.

Graham, J. W., & Schafer, J. L. (1999). On the performance of multiple imputation for

multivariate data with small sample sizes. In R. Hoyle (Ed.), Statistical strategies for

small sample research (pp. 1-29). London, England: Sage Publications.

Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable

systems. In R. Cudeck, S. Du Toit, & D. Sorbom (Eds.), Structural Equation Modeling:

Present and Future (pp. 195-216). Lincolnwood, IL: Scientific Software International.

Henry, J. D., & Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress

Scales (DASS-21): Construct validity and normative data in a large non-clinical sample.

British Journal of Clinical Psychology, 44, 227–239.


29

Hsu, S. H., Collins, S. E., & Marlatt, G. A. (2013). Examining psychometric properties of

distress tolerance and its moderation of mindfulness-based relapse prevention effects on

alcohol and other drug use outcomes. Addictive Behaviors, 38, 1852-1858.

Hu., L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:

Conventional criteria versus new alternatives. Structural Equation Modeling: A

Multidisciplinary Journal, 6, 1-55.

Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.).

Thousand Oaks, CA: Sage Publications.

Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). New York,

NY: The Guilford Press.

Leyro, T., Bernstein, A., Vujanovic, A. A., McLeish, A., & Zvolensky, M. J. (2011). Distress

Tolerance Scale: A confirmatory factor analysis among daily cigarette smokers. Journal

of Psychopathology and Behavioral Assessment, 33, 47-57.

Leyro, T. M., Zvolensky, M. J., & Bernstein, A. (2010). Distress tolerance and

psychopathological symptoms and disorders: A review of the empirical literature among

adults. Psychological Bulletin, 136, 576-600.

Linehan, M. (1993). Cognitive-behavioral treatment of borderline personality disorder. New

York, NY: The Guilford Press.

Lovibond, S. H., & Lovibond, P. F. (1995). Manuals for the Depression Anxiety Stress Scales

(2nd ed.). Sydney, Australia: Psychology Foundation.

McHugh, R. K., Kertz, S. J., Weiss, R. B., Baskin-Sommers, A. R., Hearon, B. A., &

Björgvinsson, T. (2014). Changes in distress intolerance and treatment outcome in a

partial hospital setting. Behavior Therapy, 45, 232-240.


30

McHugh, R. K., & Otto, M. W. (2012). Refining the measurement of distress tolerance. Behavior

Therapy, 43, 641-651.

Muthén, B., Kaplan, D., & Hollis, M. (1987). On structural equation modeling with data that are

not missing completely at random. Psychometrika, 52, 431-462.

Muthén, L. K., & Muthén, B. O. (2015). MPlus (Version 7.4) [Computer software]. Los Angeles,

CA: Muthén & Muthén.

Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks:

Detecting satisficing to increase statistical power. Journal of Experimental Social

Psychology, 45, 867–872.

Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a

participant pool. Current Directions in Psychological Science, 23, 184-188.

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon

Mechanical Turk. Judgment and Decision Making, 5, 411–419.

Patrick, C. J., Hicks, B. M., Nichol, P. E., & Krueger, R. F. (2007). A bifactor approach to

modeling the structure of the Psychopathy Checklist-Revised. Journal of Personality

Disorders, 21, 118-141.

Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004). Two replicable

suppressor situations in personality research. Multivariate Behavioral Research, 3, 303-

328.

Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral

Research, 47, 667-696.


31

Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological

measures in the presence of multidimensionality. Journal of Personality Assessment, 95,

129-140.

Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring

the extent to which multidimensional data yield univocal scale scores. Journal of

Personality Assessment, 92, 544-559.

Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating

and interpreting statistical indices. Psychological Methods, 21, 137-150.

Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in

covariance structure analysis. In C. C. Clogg, & A. von Eye (Eds.), Latent variables

analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA:

Sage Publications.

Shorey, R. C., Strauss, C., Elmquist, J., Anderson, S., Cornelius, T. L., & Stuart, G. L. (2017).

Distress tolerance and intimate partner violence among men in substance use treatment.

Journal of Family Violence, 32, 317-324.

Simons, J. & Gaher, R. (2005). The Distress Tolerance Scale: Development and validation of a

self-report measure. Motivation and Emotion, 29, 83-102.

Stucky, B. D., & Edelen, M. O. (2015). Using hierarchical IRT to create unidimensional

measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook

of item response theory modeling: Applications to typical performance assessment (pp.

183-206). New York, NY: Routledge.

Stucky, B. D., Thissen, D., & Edelen, M. O. (2013). Using logistic approximations of marginal

trace lines to develop short assessments. Applied Psychological Measurement, 37, 41-57.


32

Sugiura, N. (1978). Further analysts of the data by akaike's information criterion and the finite

corrections: Further analysts of the data by akaike's. Communications in Statistics-Theory

and Methods, 7, 13-26.

Suhr, D. D. (2005). Principal component analysis vs. exploratory factor analysis. SUGI 30

Proceedings. Retrieved from https://pdfs.semanticscholar.org/ 936e/

76d8f2203cb534a3952d3ea7032fc57f8ad0.pdf

Timpano, K. R., Buckner, J. D., Richey, J. A., Murphy, D. L., & Schmidt, N. B. (2009).

Exploration of anxiety sensitivity and distress tolerance as vulnerability factors for

hoarding behaviors. Depression and Anxiety, 26, 343-353.

Veilleux, J. C., Pollert, G. A., Zielinski, M. J., Shaver, J. A., & Hill, M. A. (in press). Behavioral

assessment of the negative emotion aspect of distress tolerance: Tolerance to emotional

images. Assessment.

Vujanovic, A. A., Bernstein, A., & Litz, B. (2011). Traumatic stress. In M. Zvolensky, A.

Bernstein, & A. Vujanovic (Eds.). Distress tolerance: Theory, research, and clinical

applications. New York, NY: The Guilford Press.

Vujanovic, A. A., Bonn-Miller, M. O., Potter, C. M., Marshall, E. C., & Zvolensky, M. J. (2011).

An evaluation of the relation between distress tolerance and posttraumatic stress within a

trauma-exposed sample. Journal of Psychopathology and Behavioral Assessment, 33,

129-135.

Wang, M., & Russell, S. S. (2005). Measurement equivalence of the Job Descriptive Index

across Chinese and American workers: Results from confirmatory factor analysis and

item response theory. Educational and Psychological Measurement, 65, 709-732.


33

Wolgast, M. (2014). What does the Acceptance and Action Questionnaire (AAQ) really

measure? Behavior Therapy, 45, 831-839.

Zvolensky, M. J., Bernstein, A., & Vujanovic, A. A. (Eds.). (2011). Distress tolerance: Theory,

research, and clinical applications. New York, NY: The Guilford Press.

Zvolensky, M. J., & Otto, M. W. (2010). Affective intolerance, sensitivity, and processing:

Advances in clinical science introduction [Special series]. Behavior Therapy, 38, 228-

233.

Zvolensky, M. J., Vujanovic, A. A., Bernstein, A., & Leyro, T. (2010). Distress tolerance:

Theory, measurement, and relations to psychopathology. Current Directions in

Psychological Science, 19, 406-410.


34

Footnotes

1A measure by the same name (Corstorphine, Mountford, Tomlinson, Waller, & Meyer,

2007) has been developed as another measure of distress tolerance. However, this measure was

developed and validated in a sample of treatment-seeking females with disordered eating, and

there are notable conceptual differences between the purported constructs of this measure and the

DTS of interest in the present study (e.g., an emphasis on emotion regulation strategies rather

than tolerance of emotional distress per se). Therefore, Corstorphine et al.’s measure and

confirmatory factor analytic work were not considered in our literature review.

2Confirmatory factor analysis was used to test the post-hoc hypothesis that a modified

bifactor model of the DTS (i.e., with the general factor and the Regulation domain-specific

factor) would provide adequate fit to the data. Model fit for the modified bifactor model was as

follows: χ2(87) = 556.97 (p < .001), RMSEA = .081 (90% CI = .075 - .087), CFI = .918, TLI =

.901, SRMR = .042. Significant Δχ2[12] of 140.13 (p < .001) suggested that the modified

bifactor model provided significantly worse fit to the data than the full bifactor model. However,

RMSEA 90% CIs between the two models were overlapping.

3Readers are directed to supplemental Figure 1 for a depiction of the structural regression

model and parameter estimates.

4Depression, anxiety, and stress shared positive correlations that were large in magnitude,

rdep-anx = .81, rdep-stress = .83, ranx-stress = .86, ps < .001. Thus, consistent with previous research

(Bardeen, Kumpula, & Orcutt, 2013), the DASS-21 was also modeled as a general distress

construct in a second structural regression model. Specifically, general distress was modeled as a

second-order factor; correlations between lower-order factors (i.e., anxiety, depression, stress)

were omitted and direct pathways were modeled from the second-order factor (i.e., general


35

distress) onto each lower-order factor. The DTS general factor and domain-specific factors, from

the bifactor model, were simultaneously regressed onto the general distress latent factor, as

measured via the DASS-21. Consistent with the first structural regression model, only the

Regulation factor and general factor were included as predictors in the model. The results of this

regression were as follows. Model fit was adequate, χ2(586) = 1734.62 (p < .001), RMSEA =

.049 (90% CI = .046-.051), CFI = .926, TLI = .921, SRMR = .052. The general factor of the

DTS significantly predicted general distress in a theoretically consistent direction, β = 0.45, p <

.001, while the Regulation domain-specific factor significantly predicted general distress in a

theoretically inconsistent direction, β = -0.13, p < .01.

5The full item-level correlation matrix is presented in supplemental Table 1.


36

Table 1

Review of Psychometric Literature

Simons & Gaher, 2005 Leyro et al., 2011 Hsu et al., 2013

Sample size

and

composition

Sample 1: 642 students

(70% female);

Sample 2: 823 students

(67% female)

173 adult cigarette

smokers from

community (45%

female)

168 adults seeking

treatment for substance

use (36% female)

Statistical

approach

EFA and CFA

CFA PCA

Items used 1 - 15 1 - 13, 15

(item 14 omitted)

1 - 5, 7 - 15

(item 6 omitted)

Structure

derived

1 general factor, 4

specific factors

(Tolerance,

Absorption, Appraisal,

and Regulation)

1 general factor, 4

specific factors

(Tolerance, Absorption,

Appraisal, and

Regulation)

1 general factor

Areas of

concern

Mean differences in

general distress

tolerance between

males and females

Omission of item 14;

standardized residuals

≥ 1.96; potential item-

cross loadings; strong

inter-factor correlations

(rs > .80)

Omission of item 6;

generalizability of PCA

results with past CFA

work

Note. EFA – exploratory factor analysis; CFA – confirmatory factor analysis; PCA – principle

component analysis.


37

Table 2

Model Comparisons

Model AICc χ2 (df) Scaling

Factor

RMSEA [90%

CI] CFI TLI SRMR

Correlated Four-

Factor 34895.60 432.12* (84) 1.308 .071 [.064-.078] .939 .924 .039

One-Factor 35424.30 824.78* (90) 1.343 .099 [.093-.106] .871 .850 .054

Hierarchical 34910.83 445.61* (86) 1.313 .071 [.065-.078] .937 .923 .041

Bifactor 34841.69 390.70* (75) 1.256 .071 [.064-.078] .945 .923 .036

Configural

invariance 34874.90

663.56*

(174) 1.304 .083 [.076-.090] 0.915 .898 .045

Metric

invariance 34847.24

692.92*

(190) 1.267 .080 [.074-.087] 0.913 .904 .049

Scalar

invariance 34857.43

735.82*

(203) 1.249 .080 [.074-.086] 0.908 .908 .052

Configural-

Metric Δ 27.66 14.56 (16) .003 .002 -.006 -.004

Configural-

Scalar Δ 17.47 58.70* (13) .003 .007 -.010 -.007

Metric-Scalar Δ -10.19 41.76* (29) .000 .005 -.004 -.003

Note. AICc = sample-size corrected Akaike information criterion; * indicates p < .001; df =

degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence

interval; CFI = comparative fit index; TLI = Tucker-Lewis fit index; SRMR = standardized root

mean square.


38

Table 3

Bifactor Evaluation Indices and Standardized Factor Loadings

General Factor Tolerance Absorption Appraisal Regulation

ω/ωS .95 .83 .88 .87 .84

ωH/ ωHS .91 .11 .07 .10 .35

ECV .80 .16 .12 .16 .42

H .95 .27 .25 .42 .55

FD .96 .67 .69 .75 .84

Residual variance .89 .37^ .44^ .06 .26

Item Number I-ECV Factor Loadings

1 .71 .72 .46

3 .90 .80 .27

5 .97 .66 .13

2 .70 .75 .49^

4 1.00 .83 .06^

15 .97 .81 .13^

6† 1.00 .42 .02^

7 .93 .66 .19

9 .97 .76 .13^

10 .97 .82 .14^

11 .53 .63 .59

12 .83 .73 .34

8 .72 .65 .40

13 .50 .64 .64

14 .54 .53 .49 Note. ω = omega; ωS = omega subscale; ωH = omega hierarchical; ωHS = omega hierarchical subscale; ECV =

explained common variance; H = Hancock and Mueller (2001) replicability indicator; FD = factor

determinacy; all factor loadings and residual variances significant at p < .05, except for those identified by ^;

^ indicates p > .05; † indicates reverse-scored item.

Table S1 Full Inter-Item Correlation Matrix

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 DASS1 1.00 2 DASS2 0.42 1.00 3 DASS3 0.54 0.47 1.00 4 DASS4 0.43 0.45 0.53 1.00 5 DASS5 0.47 0.36 0.56 0.43 1.00 6 DASS6 0.50 0.35 0.54 0.43 0.49 1.00 7 DASS7 0.40 0.37 0.43 0.48 0.32 0.43 1.00 8 DASS8 0.54 0.38 0.50 0.48 0.46 0.51 0.54 1.00 9 DASS9 0.44 0.37 0.52 0.47 0.49 0.55 0.51 0.59 1.00 10 DASS10 0.45 0.36 0.70 0.39 0.57 0.48 0.39 0.47 0.51 1.00 11 DASS11 0.54 0.38 0.58 0.42 0.55 0.62 0.42 0.58 0.55 0.58 1.00 12 DASS12 0.65 0.36 0.57 0.40 0.53 0.55 0.41 0.61 0.52 0.53 0.73 1.00 13 DASS13 0.52 0.38 0.70 0.44 0.61 0.53 0.38 0.48 0.49 0.71 0.62 0.61 1.00 14 DASS14 0.52 0.36 0.53 0.40 0.41 0.48 0.39 0.50 0.43 0.46 0.57 0.55 0.50 1.00 15 DASS15 0.52 0.40 0.57 0.51 0.43 0.52 0.48 0.60 0.62 0.52 0.57 0.60 0.55 0.53 1.00 16 DASS16 0.51 0.39 0.71 0.46 0.62 0.51 0.39 0.50 0.51 0.74 0.61 0.62 0.72 0.53 0.62 1.00 17 DASS17 0.47 0.39 0.67 0.41 0.53 0.50 0.40 0.48 0.56 0.73 0.55 0.53 0.70 0.48 0.58 0.70 1.00 18 DASS18 0.54 0.35 0.55 0.37 0.49 0.60 0.41 0.52 0.48 0.52 0.66 0.62 0.59 0.55 0.56 0.58 0.57 1.00 19 DASS19 0.38 0.42 0.45 0.57 0.37 0.40 0.47 0.44 0.43 0.39 0.41 0.40 0.40 0.38 0.50 0.43 0.40 0.38 1.00 20 DASS20 0.47 0.39 0.56 0.56 0.46 0.52 0.54 0.56 0.60 0.51 0.50 0.50 0.50 0.45 0.63 0.55 0.57 0.49 0.54 21 DASS21 0.45 0.38 0.68 0.41 0.52 0.51 0.45 0.47 0.53 0.74 0.53 0.51 0.68 0.44 0.53 0.68 0.78 0.53 0.41 22 DTS1 0.13 0.07 0.23 0.12 0.18 0.19 0.08 0.19 0.22 0.23 0.23 0.21 0.25 0.16 0.19 0.21 0.25 0.17 0.09 23 DTS2 0.17 0.09 0.20 0.13 0.24 0.20 0.07 0.18 0.19 0.24 0.23 0.25 0.25 0.14 0.18 0.21 0.24 0.17 0.10 24 DTS3 0.19 0.13 0.28 0.17 0.23 0.25 0.15 0.22 0.29 0.29 0.29 0.29 0.29 0.19 0.27 0.29 0.30 0.23 0.13 25 DTS4 0.27 0.17 0.33 0.22 0.27 0.32 0.18 0.27 0.33 0.33 0.31 0.30 0.38 0.25 0.33 0.33 0.37 0.26 0.17 26 DTS5 0.09 0.10 0.13 0.06 0.11 0.09 0.03 0.11 0.12 0.17 0.15 0.14 0.18 0.06 0.13 0.14 0.18 0.09 0.04 27 DTS6 0.23 0.18 0.32 0.21 0.21 0.31 0.17 0.21 0.30 0.29 0.25 0.24 0.31 0.22 0.28 0.29 0.30 0.20 0.21 28 DTS7 0.13 0.11 0.22 0.07 0.18 0.15 0.10 0.20 0.20 0.19 0.17 0.17 0.20 0.14 0.17 0.20 0.23 0.15 0.05 29 DTS8 0.10 0.07 0.15 0.05 0.11 0.11 0.08 0.13 0.17 0.17 0.14 0.12 0.17 0.10 0.14 0.15 0.16 0.09 0.04 30 DTS9 0.23 0.16 0.30 0.13 0.27 0.27 0.10 0.23 0.29 0.31 0.28 0.26 0.34 0.18 0.25 0.31 0.32 0.23 0.14 31 DTS10 0.23 0.12 0.30 0.16 0.27 0.31 0.16 0.23 0.31 0.33 0.33 0.31 0.34 0.20 0.27 0.30 0.32 0.26 0.13 32 DTS11 0.23 0.17 0.26 0.15 0.24 0.23 0.18 0.25 0.32 0.29 0.29 0.27 0.30 0.20 0.28 0.29 0.35 0.23 0.14 33 DTS12 0.25 0.17 0.31 0.22 0.26 0.27 0.24 0.33 0.37 0.33 0.32 0.32 0.34 0.25 0.36 0.33 0.34 0.24 0.20 34 DTS13 0.15 0.08 0.21 0.11 0.15 0.16 0.11 0.20 0.22 0.20 0.18 0.17 0.21 0.15 0.22 0.23 0.20 0.12 0.10 35 DTS14 0.06 0.03 0.10 0.04 0.10 0.08 0.00 0.10 0.13 0.11 0.12 0.11 0.12 0.09 0.10 0.14 0.10 0.06 0.00 36 DTS15 0.24 0.16 0.31 0.19 0.29 0.28 0.15 0.28 0.34 0.31 0.32 0.31 0.34 0.22 0.30 0.34 0.32 0.24 0.17 37 SEX 0.00 -0.04 -0.07 -0.06 0.03 0.02 -0.08 -0.05 -0.04 -0.01 -0.04 -0.01 0.04 -0.03 -0.01 0.00 -0.03 0.02 -0.07

Mean 1.04 0.92 0.74 0.50 1.15 0.92 0.41 0.76 0.68 0.86 1.08 1.14 1.01 0.80 0.59 0.82 0.77 0.94 0.71 SD 0.99 1.04 0.94 0.84 1.03 0.98 0.76 0.94 0.95 1.05 0.98 1.04 1.03 0.94 0.90 1.02 1.05 0.97 0.93 n 820 818 821 821 821 821 819 818 821 821 821 820 819 819 821 820 822 820 820

20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 1 DASS1 2 DASS2 3 DASS3 4 DASS4 5 DASS5 6 DASS6 7 DASS7 8 DASS8 9 DASS9

10 DASS10 11 DASS11 12 DASS12 13 DASS13 14 DASS14 15 DASS15 16 DASS16 17 DASS17 18 DASS18 19 DASS19 20 DASS20 1.00 21 DASS21 0.60 1.00 22 DTS1 0.19 0.25 1.00 23 DTS2 0.14 0.25 0.63 1.00 24 DTS3 0.25 0.30 0.70 0.68 1.00 25 DTS4 0.30 0.36 0.58 0.65 0.70 1.00 26 DTS5 0.07 0.15 0.54 0.54 0.56 0.58 1.00 27 DTS6 0.24 0.26 0.29 0.28 0.31 0.39 0.18 1.00 28 DTS7 0.19 0.22 0.52 0.43 0.55 0.53 0.47 0.27 1.00 29 DTS8 0.10 0.20 0.51 0.53 0.50 0.46 0.53 0.19 0.56 1.00 30 DTS9 0.24 0.34 0.51 0.55 0.60 0.63 0.45 0.39 0.55 0.47 1.00 31 DTS10 0.23 0.34 0.56 0.57 0.63 0.70 0.51 0.39 0.56 0.55 0.68 1.00 32 DTS11 0.27 0.30 0.46 0.43 0.52 0.52 0.37 0.27 0.54 0.40 0.55 0.59 1.00 33 DTS12 0.33 0.34 0.55 0.49 0.58 0.62 0.45 0.30 0.53 0.45 0.58 0.67 0.66 1.00 34 DTS13 0.19 0.23 0.48 0.43 0.49 0.47 0.48 0.23 0.49 0.67 0.47 0.53 0.41 0.53 1.00 35 DTS14 0.06 0.11 0.42 0.39 0.38 0.38 0.44 0.13 0.40 0.54 0.36 0.46 0.35 0.40 0.65 1.00 36 DTS15 0.26 0.31 0.58 0.67 0.61 0.69 0.52 0.34 0.50 0.51 0.64 0.68 0.55 0.64 0.55 0.50 1.00 37 SEX -0.11 -0.06 0.03 0.08 0.03 0.03 0.06 -0.03 -0.07 0.06 0.04 0.07 0.03 0.01 0.06 0.11 0.10 1.00

Mean 0.55 0.66 2.82 2.87 2.59 2.48 2.87 2.58 2.58 2.85 2.69 2.56 2.51 2.33 2.83 3.00 2.81 1.61 SD 0.88 1.01 1.31 1.36 1.32 1.42 1.36 1.27 1.28 1.27 1.37 1.37 1.40 1.35 1.29 1.28 1.37 0.49 n 822 819 824 824 825 822 823 823 826 822 825 825 822 822 824 823 823 821

Note: DASS – Depression Anxiety Stress Scales; DTS – Distress Tolerance Scale; SD – standard deviation; n – number of cases with available data per variable

Figure 1. Structural regression with the DTS modified bifactor model.

Note: dts# = DTS items; gendt = general factor of the DTS bifactor model; reg = Regulation domain-specific factor of

the DTS bifactor model; dep = Depression subscale of the DASS-21; anx = Anxiety subscale of the DASS-21; str =

Stress subscale of the DASS-21; dass# = DASS-21 items; all parameter estimates fully standardized (i.e., STDXY

standardization); all estimates significant at p < .01.

running head: distress tolerance scale tables: 3

Documents