Finding Your Voice:The Linguistic Development of Mental Health Counselors

Justine ZhangCornell Universityjz727@cornell.edu

Robert FilbinCrisis Text Line

bob@crisistextline.org

Christine MorrisonCrisis Text Line

christine@crisistextline.org

Jaclyn WeiserCrisis Text Line

jweiser@crisistextline.org

Cristian Danescu-Niculescu-Mizil ∗Cornell University

cristian@cs.cornell.edu

Abstract

Mental health counseling is an enterprise withprofound societal importance where conver-sations play a primary role. In order to ac-quire the conversational skills needed to face achallenging range of situations, mental healthcounselors must rely on training and on con-tinued experience with actual clients. How-ever, in the absence of large scale longitudi-nal studies, the nature and significance of thisdevelopmental process remain unclear. Forexample, prior literature suggests that expe-rience might not translate into consequentialchanges in counselor behavior. This has ledsome to even argue that counseling is a profes-sion without expertise.

In this work, we develop a computationalframework to quantify the extent to which in-dividuals change their linguistic behavior withexperience and to study the nature of this evo-lution. We use our framework to conducta large longitudinal study of mental healthcounseling conversations, tracking over 3,400counselors across their tenure. We revealthat overall, counselors do indeed change theirconversational behavior to become more di-verse across interactions, developing an in-dividual voice that distinguishes them fromother counselors. Furthermore, a finer-grainedinvestigation shows that the rate and natureof this diversification vary across functionallydifferent conversational components.

1 Introduction

Conversations are central to many human en-deavors and professions, from academic advis-ing, to business negotiations, to customer ser-vice, to mental health counseling. The choiceof conversational language has been shown tocorrelate with a broad spectrum of consequential

∗Corresponding senior author.

outcomes such as success in persuasion (Haber-nal and Gurevych, 2016; Tan et al., 2016; Zhanget al., 2016; Packard et al., 2018), in teamperformance (Yang et al., 2015; Niculae andDanescu-Niculescu-Mizil, 2016), or in social sup-port (Atkins et al., 2014; Althoff et al., 2016;Choudhury and Kıcıman, 2017). However, sur-prisingly little is known about how adults de-velop conversational skills. Understanding thischallenging process has the potential to help withtracking and fostering behavioral development inconversation-focused endeavors.

In this work, we address this knowledge gap byexamining how individuals change their conversa-tional language in a domain with profound soci-etal importance, where conversations play a pri-mary role: mental health counseling. Counselorsin this setting face a daunting task: through con-versations, they need to empathetically respond toand support clients undergoing psychological dis-tress (Althoff et al., 2016).

Initial training can orient counselors towards theprinciples of counseling conversations, but cannotcover the broad range of situations they will in-evitably encounter. Practical experience, on theother hand, comes with scarce direct feedback: itis hard for counselors to gauge whether a clientwas positively affected by a conversation. Per-haps as a result, prior studies find no consequentialchange in counselor behavior or effectiveness be-yond their initial training (Dawes, 1996; Hill et al.,2015; Goldberg et al., 2016). This lack of evi-dence has led some to even argue that psychologi-cal counseling is a “profession without any exper-tise” (Tracey et al., 2014).

However, even in the absence of feedback,multiple forces could lead counselors to changetheir behavior with experience. First, practiceexposes—and perhaps familiarizes—the coun-selor to a wide range of client situations, far be-

yond what training can encompass. This could en-able them to increasingly adapt to the situationsthey encounter, differentiating their behavior inone conversation from their behavior in another(within-counselor diversification).

Second, counselors come into the domain withindividual personalities and conversational styles.A strict adherence to training materials can lead tounnatural and robotic conversations, which can betaxing on the counselors (Orlinsky and Rønnestad,2005) and detrimental to the client (Henry et al.,1993; Castonguay et al., 1996). A counselormight therefore seek to adapt training principlesto their own personality, finding a voice that dis-tinguishes them from other counselors (between-counselor diversification).

However, change is far from guaranteed, andseveral forces potentially counteract the two vec-tors of change outlined above. The challenging na-ture and high (sometimes literally life-and-death)stakes of the conversations might lead to an overlycautious approach, deterring counselors from de-parting from language exemplified in the trainingmaterial and instead leading them to seek “securityin a restrictive practice routine” (Skovholt et al.,1997). The lack of feedback might exacerbate thisstagnation and inflexibility, since counselors haveno direct way to assess the potential effects of be-havioral changes (Shanteau, 1992; Kahneman andKlein, 2009).

In this work, we design a computationalmethodology to quantitatively track systematicchanges along the two dimensions of within- andbetween-counselor linguistic diversification. Ourframework additionally exposes the rate at whichthese changes happen across individuals and high-lights particular functional components of theirlanguage where this change is especially salient.

We apply this framework to conduct a large-scale longitudinal study of mental health coun-seling conversations on Crisis Text Line, a text-based crisis counseling platform,1 tracking theconversational behavior of over 3,400 counselorsacross their tenure. Our study reveals that over-all, counselors do indeed systematically changetheir conversational behavior, becoming more lin-guistically diverse across their own conversationsas well as among each other. Furthermore, thisdiversification process advances at different rates

1This study was done in collaboration with Crisis TextLine, using anonymized data and following IRB approval.Details of the data access are included in Section 3.

in different functional components of the con-versation; in particular, within-counselor diver-sification is amplified in components concerningthe client’s specific problems. This suggests thatcounselors accumulate domain fluency, perhapsenabling them to better address the particular situ-ations they encounter.

We complement this intuition by examininghow counselor vocabulary shifts with experience:which words increase in usage or fall out of fa-vor as counselors gain experience? We find thatas they advance in tenure, counselors adopt morespecialized as well as more colloquial language,while slowly abandoning language from the train-ing material.

Overall, these results provide the first evidenceof a systematic development of mental healthcounselors with experience, thus contributing toopen questions about expertise from the counsel-ing and psychotherapy literature. From a practi-cal standpoint, our framework could aid counsel-ing platforms to automatically identify counselorsthat stagnate. Additionally, by highlighting con-versational practices that require more experienceto grasp, this type of understanding can informsustained training programs across a counselor’scareer (Tracey et al., 2014).

More broadly, our work takes initial steps in un-derstanding how individuals develop their conver-sational behavior. Beyond mental health counsel-ing, this methodology can potentially provide in-sights into other domains where having conversa-tions is crucial but potentially hard to teach, suchas academic advising and customer service.

2 Further Related Work

Our work draws on prior literature concerning lan-guage use in conversational settings like counsel-ing, as well as on studies of linguistic change.Language in the mental health domain. Priorliterature has underlined the importance of linguis-tic choices and conversational behavior in mentalhealth-related contexts. Qualitative (Labov andFanshel, 1977; Miller and Mount, 2001; Catleyet al., 2006; Gaume et al., 2010) and computa-tional (Atkins et al., 2014; Tanana et al., 2015,2016; Althoff et al., 2016; Perez-Rosas et al.,2018) studies of dialogic interactions in counsel-ing and psychotherapy have highlighted the po-tential benefits of conversational skills—such asa counselor’s ability to reflect on a client’s par-

ticular concerns. Other work has considered abroader range of settings such as online forums,exploring how psychological support is sought andprovided (Choudhury and Kıcıman, 2017; Ernalaet al., 2017; Yates et al., 2017; Yang et al., 2017,2019; Pruksachatkun et al., 2019). While thesestudies have focused on linguistic behaviors in thescope of single interactions, we seek to understandhow these behaviors develop over time.Linguistic behavior in conversations. Our workrelates more broadly to other computational stud-ies of conversations. As in the mental health set-ting, these studies have largely considered the be-havior of interlocutors in a single conversation, ty-ing it to outcomes such as persuasion, problem-solving and incivility (Curhan and Pentland, 2007;Rosenthal and McKeown, 2015; Wang et al., 2017;Zhang et al., 2018), or to social correlates such aspower, gender and age (Gonzales et al., 2010; Ire-land et al., 2011; Danescu-Niculescu-Mizil et al.,2012; Nguyen et al., 2013; Prabhakaran et al.,2014). In contrast, our work seeks to chart the de-velopment of linguistic tendencies of individualsas they engage in many conversations.Linguistic change. Prior studies have examinedlanguage change in offline (Labov, 1966, 1972,2011; Eckert, 2005; Tagliamonte, 2011) and on-line (Cassell et al., 2006; Lam, 2008; Nguyenand Rose, 2011; Garley and Hockenmaier, 2012;Danescu-Niculescu-Mizil et al., 2013; Bammanet al., 2014; Eisenstein et al., 2014; Nguyen et al.,2015; Kulkarni et al., 2015, 2016; Goel et al.,2016) settings. A core component of such stud-ies of linguistic socialization has been the pres-ence of external social influences that may imposelinguistic norms—for instance, individuals are of-ten able to observe the language used in posts byother members of an online community. In con-trast, such social pressures are minimized in thecounseling domain we consider. Our study hencereflects an alternate set of possible mechanisms forlinguistic change, disentangling individual experi-ence from normative influence.

3 Domain: Mental Health Counseling

In this work we study the linguistic developmentof counselors in Crisis Text Line, a large text-based mental health counseling platform. Theplatform offers a free 24/7 service for individu-als undergoing mental health crises—henceforthtexters—to have one-on-one conversations with

counselors via text messages. In collaborationwith the platform, and in accordance with anIRB, we accessed the complete collection ofanonymized conversations up to September 2018.2

Counselors. The service is driven by a dedicatedroster of mental health counselors, comprised ofvolunteers who join the platform via an applica-tion process.3 Each counselor starts by complet-ing a standardized training curriculum; once theygraduate they can then sign up for shifts duringwhich they take conversations with texters. Togain a better understanding of the domain, the firstauthor completed the training curriculum and par-ticipated in a shift.

Many counselors are quite committed to the ser-vice: the median counselor takes 43 conversationswhile a quarter take at least 120. In taking a lon-gitudinal view, we focus on this latter subpopu-lation of counselors who have taken at least 120conversations, and who started their tenure afterJuly 2015.4 The 3,475 counselors in this subsettook their first 120 conversations over an averageof 5 months; in total they have taken 1,055,924conversations, accounting for 73% of all the con-versations on the platform to date.

We analyze a counselor’s behavior at differentexperience levels by dividing their tenure into aseries of consecutive life-stages S0,S1, . . . of sconversations each; unless otherwise indicated wetake s = 20. Our particular focus is on examiningcounselor behavior over their first 120 conversa-tions. We refer to a counselor as tenured once theyhave taken at least 100 conversations, and denotethe life-stage consisting of a counselor’s 100th to120th conversations as S. Our subsequent analysescompare behavior in earlier life-stages to S, usingthis eventual tenured state as a reference point.Counseling conversations. In each conversation,a counselor is faced with a challenging task: guid-

2This study was done as part of a research fellowship pro-gram organized by Crisis Text Line, which granted the au-thors access to an anonymized version of the dataset, sub-ject to an IRB. The program is open to other researchers byapplication: https://www.crisistextline.org/open-data-collaborations. Pisani et al. (2019) de-scribes the extensive privacy and ethical considerations, andthe policies implemented by the platform to address them.

3Just over half of counselors have no prior experience ina psychology-related domain. Our results are qualitativelysimilar over the subset of counselors who do not have apsychology-related background.

4We enforce the start time to account for variations in thetraining curriculum; our results are qualitatively similar underslight modifications to the start month, and over counselorswith shorter tenures.

ing a texter through a moment of crisis towardsa better mental state. This process requires thecounselor to empathetically engage with the tex-ter, often in difficult, high-stakes situations. Thecomplexity of this inherently interactional taskgives rise to substantive conversations—averaging14 counselor messages per conversation and 28words per counselor message—that yield a rich ar-ray of linguistic behaviors (Atkins et al., 2014; Al-thoff et al., 2016). We focus on tracking counselorbehavior over conversations with at least 10 coun-selor messages (comprising 64% of our dataset).

The challenge of taking conversations is com-pounded by the diversity of texters who contact theservice. Texters can come with a wide range of is-sues, from anxiety to family problems to suicidalideation. The platform assigns each texter to anavailable counselor; under the assignment mech-anism, counselors cannot select which conversa-tions they take, such that longitudinal changesin behavior cannot be accounted for by shifts incounselors’ preferences. In addition, many textersonly contact the service once and the mechanismdoes not specifically assign returning texters to thesame counselor, distinguishing this setting frompsychotherapy contexts with sustained counselor-client relationships considered in other work.

One salient guiding force for new counselors inthis challenging domain is the training curriculum,through which counselors develop skills such asactive listening and collaborative problem-solvingin a 35-hour-long course. Beyond this initial train-ing, other forces may also act to shape a coun-selor’s behavior through the remainder of theirtenure. For instance, as counselors take more con-versations, they are exposed to a broader range oftexter situations beyond what the curriculum canaccount for; these experiences may further rein-force or alter their linguistic practices and fill in-evitable gaps in the training.5

Feasibility check: Is there linguistic change?We first provide a coarse demonstration that thelinguistic behavior of counselors does indeedchange with experience in a systematic way. Weconsider a toy classification task: determiningwhether a conversation is taken by a new coun-

5Each shift is overseen by supervisors who can offer occa-sional feedback to counselors during their shifts; counselorscan also interact with one another and solicit high-level ad-vice. While in this context the level of social feedback isminimal, an interesting line of future work could more explic-itly consider the interplay between an individual’s behavioraldevelopment and such interactions.

selor who has experienced less than 20 conver-sations (S0), versus a tenured counselor who hastaken between 100 and 120 (S), on the basis ofthe language in the counselor’s messages. In par-ticular, we formulate a paired prediction prob-lem of distinguishing between two conversationstaken by the same counselor at S0 and S. Overa random subset of 10% of counselors in ourdata—comprising 3,075 conversation pairs—wetrain a logistic regression model using bigrams incounselor messages as features; we perform 10-fold cross-validation and ensure that no counselorspans multiple folds.6

The model gets a cross-validation accuracy of86%, underlining that counselors undergo dra-matic linguistic changes with experience; the highaccuracy also highlights that such changes exhibitsystematic consistencies across different coun-selors. Moreover, an analogous model that consid-ers texters’ messages achieves only 57% accuracy,suggesting that changes in counselor behavior donot merely reflect changes in the texters they in-teract with.7 This linguistic development is alsonot solely encompassed by population-wide shiftsin trivial surface-level attributes—there is no sys-tematic change in the word-counts of counselors’messages or the lengths of their conversations.

In the remainder of this work, we will develop amethodology to characterize the systematic natureof this linguistic change.

4 Measures of Linguistic Diversity

As discussed in the introduction, certain circum-stantial forces can drive counselors to diversifytheir conversational language with experience,while others can lead to linguistic stagnation. Inorder to operationalize these intuitions, we firstdesign a general framework aimed at capturingthe degree of linguistic diversity across conversa-tions and individuals.8 We then instantiate thisframework in the counseling domain and esti-

6We use logistic regression models with `1 loss, tf-idftransforming features, and grid-searching over C, the num-ber and the maximum document frequency of bigrams.

7That this low accuracy nonetheless exceeds the 50% ran-dom baseline suggests that changes in counselor behavior arereflected in aspects of a texter’s behavior as well. Indeed,prior work (Althoff et al., 2016) has suggested that texterscan be influenced within a conversation; future work couldfurther model whether the nature of this influence on the tex-ter also varies with counselor experience.

8To encourage its application to other conversational do-mains, we release an implementation of this framework aspart of ConvoKit: https://convokit.cornell.edu

mate the relation between the linguistic diversityof counselors and their effectiveness in engender-ing positive outcomes. Finally, we use the result-ing high-level linguistic characterization to ana-lyze the evolution of a counselor’s behavior overtheir tenure—the main focus of this work.Types of diversity. We can measure linguisticdiversity in conversations relative to two refer-ence points. First, an individual can linguisticallyvary across the different conversations they have;second, their language can deviate from that ofother individuals. We develop two complementarylanguage-model-based measures corresponding tothese reference points.Within-individual diversity. Formally, for aparticular individual I and life-stage Si, we di-vide the set of conversations in that stage intotwo temporally-interleaved subsets, henceforth thetraining and test sets. We train a unigram languagemodel Li

I over all of the individual’s messages inconversations in the training set. For each con-versation c in the test set, we can then computeits within-cross-entropy H(c,Li

I) to I’s languagemodel in Si. We average within-cross-entropiesacross all test set conversations to quantify I’swithin-individual diversity during Si.9Between-individual diversity. We can analo-gously define I’s between-individual diversity byreplacing I’s language model at life-stage Si withthe language model of another individual J atthe same life-stage, where J is randomly-selectedamong I’s peers. In this way, we calculate thebetween-cross-entropy H(c,Li

J ) for each conver-sation c in I’s test set of conversations, which weaverage to quantify I’s between-individual diver-sity during Si.

Cross-entropy is known to be sensitive to theamount of text used to train Li

I , as well as to thelength of the document c (Genzel and Charniak,2002). To mitigate trivial length-based effects andensure that our measures of diversity afford mean-ingful comparisons across life-stages and acrossindividuals, we randomly sample W words fromeach training set to construct the language mod-els, and w words from each conversation to com-pute the cross-entropies. For robustness, we takeaverages over several samples for each measure,and over randomly selected peers J for between-cross-entropy.

9For computational efficiency, we account for unseen vo-cabulary with an approximate form of smoothing by treatingunseen words as tokens with frequency 1 in the training set.

Relative diversity. We note that within- andbetween-individual diversity are intrinsically re-lated. If an individual’s own conversations arecloser to each other than they are from conversa-tions of other individuals we can say that the coun-selor has a distinctive voice. This would not be thecase if the between-individual diversity would beentirely explained by their within-individual diver-sity. To capture this intuitive notion we can mea-sure I’s linguistic distinctiveness during each Sias the average difference between the within- andbetween-cross-entropies of each of their conver-sations in the corresponding test set. We interprethigh relative diversities as indicative of individualswho have a consistent voice that stands out whencompared to others.Application to the counseling domain. To ap-ply these methods to analyze mental health coun-selors, we construct language models from W =2, 000 words of each counselor and life-stage,compute cross-entropies with w = 200 wordsper conversation and average over i = 50 randomsamples.10 To account for potential differences inpeer groups that may have been trained with dif-ferent curricula, for between-counselor diversitywe consider as peers counselors who started tak-ing conversations in the same month.Relation to counselor effectiveness. Before weanalyze the dynamics of linguistic diversity, webriefly consider its downstream implications forconversation outcome—to what extent is linguis-tic diversity positively reflective of a counselor’sskill? Higher diversity can signal the ability tohold more natural interactions and to better ad-dress specific texter situations, which according toliterature might translate into positive counselingoutcomes (Gaume et al., 2010; Atkins et al., 2014;Perez-Rosas et al., 2018).

Determining the quality of a counseling conver-sation is a fundamental difficulty in this domain(Tracey et al., 2014). As one imperfect indicatorof quality, we make use of texter responses to asurvey given after each conversation, asking thetexter whether or not they felt helped by the inter-action. In our data, 26% of conversations receiveda rating, out of which 87% were rated as helpful.We consider the proportion of positive ratings acounselor received in a particular life-stage as anindication of their effectiveness.11

10These parameters are chosen to ensure sufficient datawas used to characterize a counselor’s language.

11Here, we ignore conversations where no response was

To gauge the implications of a counselor’s lin-guistic diversity, we compare the effectiveness ofthe most and least diverse counselors, in terms ofboth diversity measures. In the following discus-sion, we focus on experienced counselors, in thelife-stage comprising their 80th to 120th conver-sations,12 and compare the top and bottom third ofcounselors for each measure.

We find that both within- and between-individual diversities are positive signals ofeffectiveness—for instance, the average effective-ness of the most within-individual diverse coun-selors is 89% while the effectiveness of theleast within-individual diverse counselors is 86%(Mann Whitney U test p < 0.001).

We note that the observed correspondences be-tween diversity and effectiveness have a poten-tially trivial explanation: a counselor may have aneasier time varying their language across a seriesof texters with less complicated issues who maymore readily give good ratings, independent of thecounselor’s actual skill. To discount this some-what circular explanation, we can estimate thecounselor’s diversity in an earlier life-stage (40thto 80th conversations), and relate it with their ef-fectiveness in the subsequent life-stage (80th to120th conversations). The above differences stillhold with statistical significance in this new set-ting, showing that diversity is non-trivially infor-mative of a counselor’s future effectiveness.13

5 Analyzing Counselor Diversification

We are now equipped to analyze how linguistic di-versity develops as counselors gain experience.Do counselors diversify? We first quantify theextent to which any development occurs. To thisends, we compute the percentage of counselorsthat increased their diversity from their initial con-versations during S0 to their tenured life-stage S.We find that, overall, counselors tend to diversifyas they gain experience: 58% increase in within-individual diversity—indicating that they grow tovary their language to a greater extent between

received and consider only counselors in life-stages wherethey received at least four ratings.

12Our results are similar, with some loss of significance,for alternate experience levels.

13Developing a model that can estimate the future effec-tiveness of a counselor constitutes an interesting avenue forfuture work. We note that this task would be distinct fromthat of predicting the effectiveness of a presently-observedconversation, as was addressed in prior work (Atkins et al.,2014; Althoff et al., 2016; Perez-Rosas et al., 2018).

different conversations they take—, and 73% in-crease in between-individual diversity—indicatingthat they become more linguistically distant fromtheir similarly-experienced peers. Taking thesemeasures together, we also find that most coun-selors increase in relative diversity (72%), becom-ing linguistically further away from their peersthan from themselves. Overall, these trends sug-gest an increasing inclination for counselors tomove away from an initial linguistic rigidity, even-tually finding their own distinctive voice.14

When do counselors diversify? Is this ob-served diversification sustained over the coun-selors’ tenures, or concentrated in an initialchange? Answering this question is important forunderstanding, and potentially assisting, the de-velopmental trajectory of mental health counselors(Tracey et al., 2014). As before, we consider as areference the language of tenured counselors dur-ing S, and show in Figure 1 (top row) the per-centage of counselors that increase in diversity be-tween each of their earlier life-stages and S. Weobserve that both types of diversity increase overan extended span of counselors’ tenures, but grad-ually level off to match the diversity of S (i.e., per-centages approach 50%), suggesting that linguisticdevelopment plateaus with experience.What diversifies? Conversations are notuniform—different aspects of the interaction mayengender differing evolution dynamics. In thisparticular domain, conversations follow a well-defined structure that is taught as a centralelement of the training curriculum, proceed-ing through functionally distinctive segments—henceforth components—where counselors (1)build initial rapport with the texter (hello),(2) explore the texter’s problems (problemexploration), (3) identify the texter’s goals andpast attempts to cope (goal identification),(4) collaboratively explore steps towards achiev-ing those goals (problem solving), and finally(5) close the conversation (goodbye). As a roughproxy to capture these functional components, wedivide a conversation into fifths, as in prior work(Althoff et al., 2016).15 Characteristic words for

14We note that the time it takes for the counselors to reachtheir tenured life-stage is only mildly correlated to each di-versity measure (ρ < 0.2 for each).

15For this analysis, we only consider conversations with atleast 20 counselor messages to ensure meaningful conversa-tion segments. While we found that this coarse segmentationwas satisfactory for our analyses, future work could considermore involved approaches to segmenting conversations.

Figure 1: Temporal diversification trends across different counselor life-stages (left to right). Each cell showsthe percentage of counselors that increase in diversity from that life-stage to their tenured life-stage S; *’s indicatestatistical significance (binomial test p < 0.05, comparing to 50% by chance). The topmost row shows the temporaltrend across the entire conversation, and the subsequent rows separate this trend by conversational component.

each component are shown in Figure 2. To chartlinguistic development with respect to our mea-sures across functionally-different aspects of theconversation, we compute each measure sepa-rately over the messages in each fifth of the con-versation, and repeat our temporal analysis percomponent (Figure 1, bottom rows).

We observe that development in within-individual diversity is largely concentrated insituation-specific segments of the conversation(problem exploration, goal identification,problem solving); in these segments the devel-opment is sustained throughout counselor tenures.This could suggest that as counselors encountermore diverse situations, they become linguisticallybetter equipped to approach them in a specializedfashion. In contrast, in conversational componentswhich have functions that generalize across tex-ter problems there is no systematic increase indiversity (hello), and there is even a slight de-crease (goodbye)—perhaps indicating that coun-selors develop routines to close the conversation.

On the other hand, between-individual diversityexhibits a comparatively more uniform increaseacross each component. This could suggest thatcounselors adopt individually distinguishing lin-guistic styles throughout the conversation.

In the next section we provide further supportfor these intuitions through a finer-grained analy-sis of vocabulary changes.

6 Development of Counselor Vocabulary

Thus far, we have tracked the evolution of coun-selors according to high-level characterizationsof their linguistic diversification. We now ana-lyze this evolution at a complementary granular-ity: tracking changes in counselors’ use of differ-ent words over the course of their tenure, as tangi-ble and interpretable indicators of linguistic devel-opment. This analysis offers concrete examples oflexical changes that reflect the intuitions gleanedfrom our preceding examination.

A counselor may grow to use some words moreoften, perhaps reflecting continued exposure toparticular texter situations or the development ofa personal style. Such adoptions could enrichand diversify a counselor’s vocabulary, and per-haps distinguish them from others; they could alsomark aspects that converge to a common coun-selor language, counterbalancing the diversifica-tion process. Alternatively, some words may fallout of favor, getting used less frequently over time.This possibility is especially pertinent given theinitial training process, since the curriculum illus-trates many of the counseling practices by way ofpresenting examples of messages for counselorsto build on. These serve as a linguistic referencepoint for starting counselors; linguistic divergencecould then arise as more experienced counselorsstart to move beyond this reference point.

To examine such changes in counselor vocabu-lary, we quantify the population-wide usage shiftof a word as the log-ratio between the proportion

Figure 2: Top: the overall distribution of usage shifts of the words at the core of the counselor vocabulary (i.e.,those used by at least 20% of counselors). Bottom: usage shift distributions per subset of core words characteristicto each conversational component, along with examples. Words are ordered by, and colored according to theirusage shift, such that blue words tend to be used more by counselors in their earlier conversations while red wordstend to be used more by counselors with more experience. Dashed lines indicate the respective medians.

of conversations in S in which counselors used thatword, and the proportion of conversations in S0 inwhich counselors used that word.16 Thus, a posi-tive shift indicates that the word’s usage increaseswith tenure, while a negative shift means that itsusage declines after the counselor’s first conversa-tions. Figure 2 (top) shows a histogram of usageshifts for the core of the counselors’ vocabulary,i.e., the 3,461 words used by at least 20% of thelong-serving counselors considered in the preced-ing analyses.The interplay of lexical shifts and function. Ourprior analyses revealed that diversification patternsvary by functionally-different conversational com-ponents. To tie usage shifts to these observed con-trasts, we examine the usage shifts of words char-acteristic to each component. Concretely, we con-sider a word to be characteristic of a componentif the proportion of messages containing it in thecorresponding fifth of the conversation is greaterthan the proportion of messages with the wordoverall (in particular, the log-ratio of the within-component frequency of the word to its overall fre-quency is at least 0.2). Figure 2 (bottom) showsusage shifts over words in each component, to-gether with example words sorted by their shift.

16To avoid spurious effects akin to Simpson’s paradox,we only consider conversations taken by counselors with atleast 120 conversations, such that each counselor contributesequally to both the S and S0 components of the measure.

Across each component, we see that counselorsshift away from using words which are relativelygeneral and more formal in tone (negative shift,in blue). Many of these words reflect coun-seling practices as presented in the training ma-terial, echoing the language used in training toexemplify these practices. For instance, duringthe first component (hello), counselors are toldto build rapport and bootstrap the exploration ofthe texter’s concerns by having texters elaborateon what prompted them to contact the platform;as counselors proceed in this exploration process(problem exploration) they are taught to reflecton and validate the texter’s concerns as under-standable and normal.

That counselors eventually use such words lessoften does not necessarily mean that the prac-tices underlying them are abandoned with expe-rience; rather, counselors may gravitate towardsnew words that accomplish the same purposes(valid, unfair). Words used in the first two compo-nents are especially prone to decreasing in usagewith experience. This suggests that while incom-ing counselors use the training material as a lin-guistic guide for initiating conversations, they tendto eventually shift towards more informal and lessstandardized language, moving away from usingwords in the core of the counselors’ vocabulary.17

17We note that our usage shift methodology only captures

Conversely, many words which counselorsadopt later in their tenure (positive shift, inred) are more specialized, especially in the goal

identification and problem solving compo-nents. Intuitively, counselors accumulate domainknowledge with experience, allowing them to bet-ter differentiate between cases, especially in termsof potential coping mechanisms (e.g., therapist,meditation, apps). This type of specializationsuggests that much of the broadening in within-individual diversity we observed in these conver-sational segments (Figure 1, left) may be func-tional in nature.

Other words which increase in usage with expe-rience are characterized by a relatively colloquialtone (hey, :), gotcha), pointing to a diversificationprocess that is social rather than functional. Thislinguistic relaxation may trace the observed trendtowards higher between-individual diversity (Fig-ure 1, right) as counselors become more at easein their task. Shifts towards such colloquial lan-guage are especially prominent in the last compo-nent (goodbye), potentially as counselors developindividual routines for closing conversations, andechoing the decrease in within-individual diversityobserved in that component (Figure 1, left).

Overall, these observations complement ourprior discussion about the rate at which counselorsdiversify their language, suggesting functional andsocial mechanics that may be in play.

7 Discussion and Future Work

Having good conversations is challenging andhard to teach, especially in domains where thestakes are high and where feedback is scarce.Computational tools could help us understandhow people acquire conversational skills, and mayeventually assist in their development. In thiswork we provide an initial and limited case studyin a highly consequential domain, showing that wecan track diversification in the linguistic practicesof mental health counselors.

Our approach is necessarily limited in scope.Future work could adapt our framework to exam-ine more complex forms of conversational devel-opment. In particular, we observed that counselorsdiversify linguistically, perhaps signaling a benefi-

population-wide changes in how counselors use a subset ofcore words, so it cannot characterize changes outside thisgeneral vocabulary. Future work could examine individual-level lexical changes (such as the development of uniquelypersonal catchphrases).

cial increase in flexibility that enables them to bet-ter address the specific concerns of a texter. Fu-ture work could more directly model how coun-selors respond to texter behaviors, hence gaugingthe extent to which counselors evolve in their in-teractional practices. Furthermore, while our ap-proach captures linguistic changes in aggregate, acomplementary line of work could explore the tra-jectories of individuals, and probe the factors de-termining whether particular individuals diversifyor stagnate.

The framework we have started to design couldeventually help platforms provide counselors withpersonalized feedback on their development. Itcould assist in identifying counselors who accli-matize quickly and those that require more guid-ance. Understanding how different components ina conversation change with experience could alsoinform the particular aspects to focus their trainingon. In order to derive such prescriptive recommen-dations, further work is needed to causally connectour signals of linguistic experience with actual ex-pertise and skill, as reflected in conversational out-comes. For instance, increased diversification maypoint to flexibility and specialization that are ben-eficial, but might also signal detrimental devia-tions from good counseling practices. Other ap-proaches, such as qualitative labeling by domainexperts, could examine whether such changes inlanguage use also result in better conversations.

Our methodology could also be extended to ex-amine other conversational contexts such as aca-demic advising or business interactions, where in-dividuals are expected to learn from experience.Such domains may contain crucial differences thatmotivate extensions to our framework; for exam-ple, feedback might be more readily available andconversational partners may recur, both of whichcan interact with experience to further shape lin-guistic development.

Acknowledgments. Thanks to Crisis Text Line for making

this work possible. We are grateful to the anonymous re-

viewers for their thoughtful comments and suggestions, and

to Jonathan P. Chang, Liye Fu, Sendhil Mullainathan and

Andrew Wang for helpful conversations. This research has

been supported in part by NSF CAREER Award IIS1750615,

NSF Grant SES-1741441, and a Microsoft Research PhD Fel-

lowship. The collaboration with Crisis Text Line was sup-

ported by the Robert Wood Johnson Foundation; the views

expressed here do not necessarily reflect the views of the

foundation.

ReferencesTim Althoff, Kevin Clark, and Jure Leskovec. 2016.

Large-scale Analysis of Counseling Conversations:An Application of Natural Language Processing toMental Health. Transactions of the Association forComputational Linguistics, 4.

David C. Atkins, Mark Steyvers, Zac E. Imel, andPadhraic Smyth. 2014. Scaling up the evaluation ofpsychotherapy: Evaluating motivational interview-ing fidelity via statistical text classification. Imple-mentation Science, 9(1).

David Bamman, Jacob Eisenstein, and Tyler Schnoe-belen. 2014. Gender identity and lexical variation insocial media. Journal of Sociolinguistics, 18.

Justine Cassell, David Huffaker, Dona Tversky, andKim Ferriman. 2006. The language of online leader-ship: Gender and youth engagement on the Internet.Developmental Psychology, 42.

Louis Georges Castonguay, Marvin R. Goldfried, Su-san Wiser, Patrick J. Raue, and Adele M. Hayes.1996. Predicting the effect of cognitive therapy fordepression: A study of unique and common fac-tors. Journal of Consulting and Clinical Psychol-ogy, 64(3).

Delwyn Catley, Kari Jo Harris, Matthew S. Mayo, San-dra Hall, Kolawole S. Okuyemi, Thuy Boardman,and Jasjit S. Ahluwalia. 2006. Adherence to Princi-ples of Motivational Interviewing and Client Within-Session Behavior. Behavioural and Cognitive Psy-chotherapy, 34(1).

Munmun De Choudhury and Emre Kıcıman. 2017.The Language of Social Support in Social Media andits Effect on Suicidal Ideation Risk. In Proceedingsof ICWSM.

Jared R. Curhan and Alex Pentland. 2007. Thin slicesof negotiation: Predicting outcomes from conversa-tional dynamics within the first 5 minutes. Journalof Applied Psychology, 92.

Cristian Danescu-Niculescu-Mizil, Lillian Lee,Bo Pang, and Jon Kleinberg. 2012. Echoes ofpower: Language effects and power differences insocial interaction. In Proceedings of WWW.

Cristian Danescu-Niculescu-Mizil, Robert West, DanJurafsky, Jure Leskovec, and Christopher Potts.2013. No Country for Old Members: User Lifecy-cle and Linguistic Change in Online Communities.In Proceedings of WWW.

Robyn Dawes. 1996. House of Cards. Free Press, NewYork.

Penelope Eckert. 2005. Variation, convention, and so-cial meaning. In Proceedings of the Annual Meetingof the Linguistic Society of America.

Jacob Eisenstein, Brendan O’Connor, Noah A. Smith,and Eric P. Xing. 2014. Diffusion of lexical changein social media. PLoS One, 9.

Sindhu Kiranmai Ernala, Asra F. Rizvi, Michael L.Birnbaum, John M. Kane, and Munmun De Choud-hury. 2017. Linguistic Markers Indicating Ther-apeutic Outcomes of Social Media Disclosures ofSchizophrenia. In Proceeding of CSCW.

Matt Garley and Julia Hockenmaier. 2012. Beefmoves:Dissemination, Diversity, and Dynamics of EnglishBorrowings in a German Hip Hop Forum. In Pro-ceedings of ACL.

Jacques Gaume, Nicolas Bertholet, Mohamed Faouzi,Gerhard Gmel, and Jean-Bernard Daeppen. 2010.Counselor motivational interviewing skills andyoung adult change talk articulation during briefmotivational interventions. Journal of SubstanceAbuse Treatment, 39(3).

Dmitriy Genzel and Eugene Charniak. 2002. EntropyRate Constancy in Text. In Proceedings of ACL.

Rahul Goel, Sandeep Soni, Naman Goyal, John Pa-parrizos, Hanna Wallach, Fernando Diaz, and Ja-cob Eisenstein. 2016. The Social Dynamics of Lan-guage Change in Online Networks. In Proceedingsof SocInfo.

Simon B. Goldberg, Tony Rousmaniere, Scott D.Miller, Jason Whipple, Stevan Lars Nielsen,William T. Hoyt, and Bruce E. Wampold. 2016. Dopsychotherapists improve with time and experience?A longitudinal analysis of outcomes in a clinical set-ting. Journal of Counseling Psychology, 63(1).

Amy L. Gonzales, Jeffrey T. Hancock, and James W.Pennebaker. 2010. Language style matching as apredictor of social dynamics in small groups. Com-munication Research, 37.

Ivan Habernal and Iryna Gurevych. 2016. What makesa convincing argument? Empirical analysis and de-tecting attributes of convincingness in Web argu-mentation. In Proceedings of EMNLP.

William P. Henry, Hans H. Strupp, Stephen F. Butler,Thomas E. Schacht, and Jeffrey L. Binder. 1993.Effects of training in time-limited dynamic psy-chotherapy: Changes in therapist behavior. Journalof Consulting and Clinical Psychology, 61(3).

Clara E. Hill, Ellen Baumann, Naama Shafran, Shu-darshana Gupta, Ashley Morrison, Andres E. PerezRojas, Patricia T. Spangler, Shauna Griffin, LauraPappa, and Charles J. Gelso. 2015. Is training ef-fective? A study of counseling psychology doctoraltrainees in a psychodynamic/interpersonal trainingclinic. Journal of Counseling Psychology, 62(2).

Molly E. Ireland, Richard B. Slatcher, Paul W. East-wick, Lauren E. Scissors, Eli J. Finkel, and James W.Pennebaker. 2011. Language style matching pre-dicts relationship initiation and stability. Psycholog-ical Science, 22.

Daniel Kahneman and Gary Klein. 2009. Conditionsfor intuitive expertise: A failure to disagree. Ameri-can Psychologist, 64(6).

Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, andSteven Skiena. 2015. Statistically significant detec-tion of linguistic change. In Proceedings of WWW.

Vivek Kulkarni, Bryan Perozzi, and Steven Skiena.2016. Freshman or Fresher? Quantifying the Geo-graphic Variation of Language in Online Social Me-dia. In Proceedings of ICWSM.

William Labov. 1966. The Social Stratification of En-glish in New York City. The Social Stratification ofEnglish in New York City. Washington, D.C.: Cen-ter for Applied Linguistics.

William Labov. 1972. Sociolinguistic Patterns. Uni-versity of Pennsylvania Press.

William Labov. 2011. Principles of Linguistic Change,Cognitive and Cultural Factors. Wiley-Blackwell.

William Labov and David Fanshel. 1977. TherapeuticDiscourse: Psychotherapy as Conversation. Aca-demic Press.

Wan Shun Eva Lam. 2008. Language Socializationin Online Communities. In Nancy H. Hornberger,editor, Encyclopedia of Language and Education.Springer US.

William R. Miller and Kathy A. Mount. 2001. A smallstudy of training in motivational interviewing: Doesone workshop change clinician and client behavior?Behavioural and Cognitive Psychotherapy, 29(4).

Dong Nguyen, A. Seza Dogruoz, Carolyn P. Rose, andFranciska de Jong. 2015. Computational Sociolin-guistics: A Survey. Computational Linguistics.

Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, andTheo Meder. 2013. How Old Do You Think I Am?:A Study of Language and Age in Twitter. In Pro-ceedings of ICWSM.

Dong Nguyen and Carolyn P. Rose. 2011. Languageuse as a reflection of socialization in online commu-nities. In Proceedings of the Workshop on Languagein Social Media.

Vlad Niculae and Cristian Danescu-Niculescu-Mizil.2016. Conversational Markers of Constructive Dis-cussions. In Proceedings of NAACL.

David E. Orlinsky and Michael Helge Rønnestad.2005. How Psychotherapists Develop: A Study ofTherapeutic Work and Professional Growth. HowPsychotherapists Develop: A Study of TherapeuticWork and Professional Growth. American Psycho-logical Association.

Grant Packard, Sarah G. Moore, and Brent McFerran.2018. (I’m) Happy to Help (You): The Impact ofPersonal Pronoun Use in Customer–Firm Interac-tions. Journal of Marketing Research, 55(4).

Veronica Perez-Rosas, Xuetong Sun, Christy Li,Yuchen Wang, Kenneth Resnicow, and Rada Mi-halcea. 2018. Analyzing the Quality of CounselingConversations: The Tell-Tale Signs of High-qualityCounseling. In Proceedings of LREC.

Anthony R. Pisani, Nitya Kanuri, Bob Filbin, CarlosGallo, Madelyn Gould, Lisa S. Lehmann, RobertLevine, John E. Marcotte, Brian Pascal, DavidRousseau, Shairi Turner, Shirley Yen, and Megan L.Ranney. 2019. Protecting User Privacy and Rightsin Academic Data-Sharing Partnerships: PrinciplesFrom a Pilot Program at Crisis Text Line. Journalof Medical Internet Research, 21(1).

Vinodkumar Prabhakaran, Emily E. Reid, and OwenRambow. 2014. Gender and Power: How Genderand Gender Environment Affect Manifestations ofPower. In Proceedings of EMNLP.

Yada Pruksachatkun, Sachin R. Pendse, and AmitSharma. 2019. Moments of Change: AnalyzingPeer-Based Cognitive Support in Online MentalHealth Forums. In Proceedings of CHI.

Sara Rosenthal and Kathy McKeown. 2015. I Couldn’tAgree More: The Role of Conversational Structurein Agreement and Disagreement Detection in OnlineDiscussions. In Proceedings of SIGDIAL.

James Shanteau. 1992. Running Head: Competence inExperts.

Thomas M. Skovholt, Michael Helge Rønnestad, andLen Jennings. 1997. Searching for Expertise inCounseling, Psychotherapy, and Professional Psy-chology. Educational Psychology Review, 9(4).

Sali Tagliamonte. 2011. Variationist Sociolinguis-tics: Change, Observation, Interpretation. Wiley-Blackwell.

Chenhao Tan, Vlad Niculae, Christian Danescu-Niculescu, and Lillian Lee. 2016. Winning Argu-ments: Interaction Dynamics and Persuasion Strate-gies in Good-faith Online Discussions. In Proceed-ings of WWW.

Michael Tanana, Kevin A. Hallgren, Zac E. Imel,David C. Atkins, Padhraic Smyth, and Vivek Sriku-mar. 2015. Recursive Neural Networks for Cod-ing Therapist and Patient Behavior in MotivationalInterviewing. In Proceedings of the Workshop onComputational Linguistics and Clinical Psychology.

Michael Tanana, Kevin A. Hallgren, Zac E. Imel,David C. Atkins, and Vivek Srikumar. 2016. AComparison of Natural Language Processing Meth-ods for Automated Coding of Motivational Inter-viewing. Journal of Substance Abuse Treatment, 65.

Terence J. G. Tracey, Bruce E. Wampold, James W.Lichtenberg, and Rodney K. Goodyear. 2014. Ex-pertise in psychotherapy: An elusive goal? TheAmerican Psychologist, 69(3).

Lu Wang, Nick Beauchamp, Sarah Shugars, andKechen Qin. 2017. Winning on the Merits: TheJoint Effects of Content and Style on Debate Out-comes. Transactions of the Association for Compu-tational Linguistics, 5.

Diyi Yang, Robert Kraut, and John M. Levine. 2017.Commitment of Newcomers and Old-timers to On-line Health Support Communities. In Proceedingsof CHI.

Diyi Yang, Robert Kraut, Tenbroeck Smith, Eli-jah Mayfield, and Dan Jurafsky. 2019. Seekers,Providers, Welcomers, and Storytellers: ModelingSocial Roles in Online Health Communities. In Pro-ceedings of CHI.

Diyi Yang, Miaomiao Wen, and Carolyn PensteinRose. 2015. Weakly Supervised Role Identificationin Teamwork Interactions. In Proceedings of ACL.

Andrew Yates, Arman Cohan, and Nazli Goharian.2017. Depression and Self-Harm Risk Assessmentin Online Forums. In Proceedings of EMNLP.

Justine Zhang, Jonathan P. Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Nithum Thain,Yiqing Hua, and Dario Taraborelli. 2018. Conver-sations Gone Awry: Detecting Early Signs of Con-versational Failure. In Proceedings of ACL.

Justine Zhang, Ravi Kumar, Sujith Ravi, and Cris-tian Danescu-Niculescu-Mizil. 2016. Conversa-tional flow in Oxford-style debates. In Proceedingsof NAACL.