assessment of social engagement and cognitive function for...

Social Engagement, Cognitive Decline and Measurements Assessing Social Engagement Assessing Cognitive Function Conclusion

Assessment of Social Engagement andCognitive Function for Studying Aging

Izhak Shafran

Center for Spoken Language Understanding (CSLU)Oregon Health & Science University

Portland, OR

CMU LTI Colloquium


Number of Social Ties Vs. Cognitive Decline1

• 2812 adults, 65 yrs or older, 1982-94

• 0 vs. 5-6 ties: Twice more likely to decline!!

1S. S. Bassuk et al. “Social disengagement and incident cognitive declinein community-dwelling elderly persons.” In: Ann Intern Med 131.3 (1999).


Social Engagement and Health

Deleterios Affects of Social Disengagement• Cognitive decline2

• Higher depression3

• Slower recovery from health incidents

Understanding Social Engagement• What aspects of social engagement matter?• Can we detect unhealthy levels of disengagement?• Can we intervene and promote engagement? How?

2S. S. Bassuk et al. “Social disengagement and incident cognitive declinein community-dwelling elderly persons.” In: Ann Intern Med 131.3 (1999).

3T. A. Glass et al. “Social engagement and depressive symptoms in latelife: longitudinal findings”. In: J Aging Health 18.4 (2006), pp. 604–628.


Measuring Social Function: Subject’s Perspective

• Questionnaires• E.g. “How many friends do you have?”• Relies on memory, hence confounding

• Experience sampling• E.g. Beep: “Were you alone or with someone?”• No easy trade-off: frequent sampling vs perturbing behavior

In Summary,• Easy to administer• Subject’s perspective, has inherent value, but need more• Need fine-grained information


Measure’s from Observer’s Perspective

Follow a subject and record their everyday life• One Boy’s Day4

• The lived day of an individual5

• Intrusive, measurement perturbs behavior• Labor-intensive

4R. G. Barker et al. “One boy’s day”. In: (1951).5K. H. Craik. “The lived day of an individual”. In: (2000).


An Acoustic Window into Social Behavior

Electronically Activated Recorder (EAR)6

• Record ambient conversations throughout the day• Annotators listen to recording and annotate• Annotations include transcripts, social context, affect• For privacy-protection, recording not continuous

6M. R Mehl and J. W. Pennebaker. “The sounds of social life: apsychometric analysis of students’ daily social environments and naturalconversations.” In: J Pers Soc Psychol 84.4 (2003), pp. 857–870.



• Unobtrusive, doesn’t perturb behavior• Samples subjects’ naturalistic conversations• Layers of information

• Interaction: in-person, on the phone, alone• Talking to: male(s), female(s), mixed group• Location: at home, in transit, dining/bar, recreation• Activity: radio/tv, work, chores, sports, entertainment• Mood: laugh, sing, cry, mad, sigh• Health: cough/sneeze

Many successful social psychology studies7

7M. R Mehl. “The lay assessment of subclinical depression in daily life”. In:Psychol Assess 18.3 (2006), pp. 340–5.



In Summary• No effort by the subject, doesn’t peturb behavior• Observer’s perspective, consistency can be controlled• Easy to record observations• But, need to listen and annotate, labor intensive!• And too noisy for automation with current technology


Social Engagement, Cognitive Decline and Measurements

Assessing Social Engagement

Assessing Cognitive Function

Conclusion


Social Engagement via Telephony

Premise• Older adults are less mobile, rely on telephones heavily8

• Entire interaction occurs through voice– no gestures, facial expressions, . . .

• Many forms of dementia directly effect language• We can recognize the content automatically, can scale !!

8P. Taylor et al. Growing old in America: Expectation vs. Reality. Tech. rep.Pew Research Center, 2009.


From Call Logs: Social Networks


From Call Content: Social Relationships


Social Engagement via Telephony

• Unobtrusive, doesn’t perturb behavior• Samples subjects’ naturalistic conversations• Layers of information

• Talking to: male(s), female(s), mixed group• Affect: happy, sad, angry, . . .• Health: cough/sneeze


Subject Pool: Fairly Active!

Activity Daily Weekly Monthly Yearly RarelyRead a newspaperListen to radio/TV newsUse a computerListen to musicWatch TVWatch moviesFollow finances/investmentsHave visitorsVisit others at their homesEat outTake a classRead a bookAttend a club meetingTravel out of townCare for pet


Subjects and Corpus

• 10 subjects, 79 years or older• Social questionnaire• Unique Corpus

• Call logs, includings numbers called to/from, time, duration• ALL incoming/outgoing telephone conversations recorded• Enrollment and exit interviews, picture description task

• Ongoing collection: 45 residences more, 2500 hours so far

Valuable Orthogonal Data• Cognitive (neuropsychological) tests, MRI, activity reports• Sensor data, including doors, motion, medicine, . . .• Longitudinal analysis: backtrack future health outcomes


ORCATECH’s Living Lab

Secure

Internet

!"#$%&#'(#)**+,-./01-23(4)*3056732(677377*3,.8(

(9%43(:-;-,<(:65=((


Security and Privacy of the Data

Encrypted transcriptAutomatic

RecognitionSpeech

Encryption

Encryption

using

Standard

Advanced

EncryptedLexicon

Speech &Speaker

Detection

Data Storage

OHSU

SpeechEncrypted

the MarkersComputation of

speaker IDsAnnonymized

Subject’s Residence

talking to you!It was good

w23 w56 w24w46 w59 w45!


Our Tasks

Task 1: residential vs. businessTask 2: family vs. non-familyTask 3: familiar vs. unfamiliarTask 4: family vs. other residential

• Subset of data was labeled for training and testing• For example, business vs. residential

• ≈ 8.3k conversations, after trimming short ones• labels for ≈ 4.3k (2.7k residential, 1k business)• no labels for ≈ 4k• balanced training (1.8k) and test (328k) sets


Can Duration Distinguish Calls? No!

1.5 2 2.5 3 3.5 40

0.2

0.4

0.6

0.8

LOG10 [word count]

Estim

ate

d p

rob

ab

ility

Global duration

Res. call duration

Biz. call duration


Can Days of the Week Distinguish Calls? No!

Mon Tue Wed Thu Fri Sat Sun0

0.05

0.1

0.15

0.2

0.25

0.3

Day

Pro

babili

ty o

f call

Biz.

Res.


Can Hours of the Day Distinguish Calls? No!

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230

0.05

0.1

0.15

0.2

0.25

0.3

Hour

Pro

babili

ty o

f call

Biz.

Res.

In Summary• Simple features are not sufficient!• Need to examine the content of the conversations


Content via State-of-the-Art Speech Recognizer

• Acoustic Models• Trained on 2000 hour of speech• 8000 pentaphone clustered states• 150K Gaussians, w/ semi-tied covariance

• Language Models• 47k vocabulary, 10M parameters• 10M n-grams, trigrams

• Three Stage Decoding• Speaker-independent models• Vocal-tract length normalized models• Speaker-adaptation• Speaker-adapted models• Maximum likelihood linear regression models

• 24% word error rate on 2004 NIST RT Task


Simple Supervised Classification

• Audio =⇒ transcript =⇒ features =⇒ classifier =⇒ labels• Transcripts: errorful

E.g., hello, this is mark is• Features: simple word counts or lexical unigrams

E.g., c[hello] = 1, c[this] = 1, c[is] = 2, c[mark] = 1• Classifier: support vector machines, linear, radial basis

functions


Simple Supervised Classification: Results

Task 1: residential vs business, Task 2: family vs non-family, Task 3:familiar vs non-familiar, and Task 4: family vs other residential.

Features Task 1 Task 2 Task 3 Task 4

Unigram 87.2 76.6 72.9 78.0Bigram 85.1 77.8 73.5 77.2Trigram 83.2 74.0 71.4 76.3

Surface 69.6 72.0 62.1 75.7Unigram + Surface 86.9 81.2 74.4 77.2

• High accuracies, 74-87%, in spite of ASR errors• Fully automated classification of social relationships!


What Features Matter?

• Linguistic Inquiry and Word Count9

• 32 psychological constructs (affect, cognition, biological)• 22 linguistic dimensions (POS)• 7 personal categories (work, home, leisure activities)• 3 paralinguistic dimensions (assents, fillers, nonfluencies)

9J. W. Pennebaker. “Linguistic inquiry and word count (LIWC)”. In: (2001).


What Features Matter?

Task 1: residential vs business, Task 2: family vs non-family, Task 3:familiar vs non-familiar, and Task 4: family vs other residential.

Features Task 1 Task 2 Task 3 Task 4

Unigram 87.2 76.6 72.9 78.0

Unigram-stem 87.8 76.0 74.3 76.0LIWC 77.1 74.6 64.8 69.1

POS-unigram 78.4 66.8 59.8 67.1POS-bigram 77.7 70.8 63.9 70.5Unigram × POS 84.2 76.3 72.5 79.8Unigram + POS 86.9 76.0 72.6 77.5

Stemming and Unigram × POS helps, LIWC not so much


Feature Selection via Mutual Information

102 103 10483

84

85

86

87

88

CV

accu

racy

102 103 10480

82

84

86

88

Verif

icat

ion

accu

racy

Dictionary size

Frequency truncationMI truncation

• More effective than POS• Optimal performance with 1000+ words


Mutual Information: Top 30 Words

Business oriented Social orientedPress, thank, calling, infor-mation, service, customer,number, quality, please,pressed, representative,account, zero, seven, moni-tored, transferred, nine, six,transfer, services.

Hi, dinner, she’s, high, home,dad, night, everybody, do-ing, tonight, later, hello, mom,anyway, bad, nice, sleep, to-morrow, house.


Alternative: Classifying using Conversation Topics

• Fortunately, we can learn topics automatically, usingLatent Diriclet Allocations

• Utilize 4k unlabeled conversations to learn topics• Each conversation may contain multiple topics• Estimate the proportion of each topic in a conversation• Then, use that to classify conversations

spoken words =⇒ posterior over topics (θ) =⇒ classifier =⇒ labels


Topic Features from Latent Dirichlet Allocation

2 5 10 20 30 50 75 10076

78

80

82

84

86

88

Number of topics

Accu

racy

Cross validationVerification

• No loss in performance, all the way down to 30 topics• With 2-topics, naturally clusters into biz vs. social calls


LDA Two-Topic Word Distribution

Topic 1 Topic 2Invalid, helpline, eligibility,transactions, promotional,representative, mastercard,touchtone, activation, nom-inating, receiver, voicemail,digit, representatives, ballots,refills, classics, metro, ad-minister, transfers, reselling,exclusive, submit.

Adorable, aeroplanes, Ar-lene, Astoria, baked, bis-cuits, bitches, blisters, blue-grass, bracelet, brains, Char-lene, cheeses, chit, Chris,clam, clientele, cock, crab,Davenport, debating, demen-tia, dime, Disneyland, Eileen,follies, gained

• For biz, probability mass is concentrated on few words• For social, probability mass is more widely distributed


Error Distribution: Across Homes

Home Records Accuracy1 8 87.52 103 84.53 42 81.04 6 100.05 27 77.06 74 94.67 25 88.08 43 90.7

• Accuracy uniformly better than 77%


Error Distribution: Conversation Length

Word Counts Chance AccuracyPercentile Range0-20 30-87 62.12 75.7620-40 88-167 51.52 83.3340-60 168-295 60.61 90.9160-80 296-740 59.09 93.9480-100 741+ 59.38 93.75

• Accuracy degrades for shorter conversations• Accuracy is stable > 300 words (2-3 minutes)


Calls from Certain Number Always Correctly Classified?

• Upto 300 conversations from some numbers

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

Num

ber o

f tel

epho

ne c

onta

cts

Classification accuracy (%)

• 50 / 125 correct all the time• 5 consistently wrong (e.g., 65 calls to a lighting store)


Nature of Everyday Telephone Conversations


Openings & Closings

• Do different parts of the conversations contribute equally?• Schegloff & Sacks: Openings and closings are distinct

30 50 100 250 500 10000

5

10

15

20

25

30

Number of words sampled

Res

/biz

cla

ssifi

catio

n er

rror (

%)

Word sample from startWord sample from endWord sample randomly taken

• Openings are good, but closings are not


Nature of Short vs. Long Calls

• Just saw, first 30 words are sufficient to classify• But, accuracy degrades for short conversations• Sparsity or intrinsic nature of short conversations?


Sparsity or Intrinsic Nature: Why Are Short Calls Difficult?

Truncate all calls to 30-words, then comparing accuracy

Original Length (# Words) Split AccuracyPercentile Range Res / Biz

0-20 30-87 62.1 / 37.9 78.620-40 88-167 48.5 / 51.5 82.840-60 168-295 39.4 / 60.6 91.460-80 296-740 40.9 / 59.1 87.880-100 741+ 59.4 / 40.6 93.4

• Original longer calls are still easier to classify• Degradation is not due to sparsity, but inherent ambiguity


What Length of Observation is Sufficient?

Jensen-Shannon divergence• 12-month estimate vs shorter windows• Averaged over all windows and residences


What Length of Observation is Sufficient?

0 2 4 6 8 10 120

0.02

0.04

0.06

0.08

JS

−D

iv

Duration (months)

(a) Business vs. residential

0 2 4 6 8 10 120

0.005

0.01

0.015

0.02

0.025

JS

−D

iv

Duration (months)

(b) Family vs. non-family

0 2 4 6 8 10 120

0.002

0.004

0.006

0.008

0.01

0.012

JS

−D

iv

Duration (months)

(c) Familiar vs. non-familiar

0 2 4 6 8 10 120

0.01

0.02

0.03

0.04

JS

−D

iv

Duration (months)

(d) Family vs. res. non-family

• Reference labels, discard more calls, need longer obs.• Automatic labels, use all calls, stable w/ shorter obs.


Summary and Implications

• Framework for measuring social engagement• Infer types of social interaction automatically• Accuracies of 74-88%, with 30 topics or first 30 words• Can be improved by collating information across calls• Content more useful than the medium specific features;

applicable to emails, chats, . . . ; cover other demographies• More importantly, our framework allows deeper analysis• Now, expanding to 50 subjects, cross-sectional analysis• Additionally, include affect, health topics, who spoke what


Social Engagement, Cognitive Decline and Measurements

Assessing Social Engagement

Assessing Cognitive Function

Conclusion


Cognitive Function

• Digit Span (forward, reverse)• Stroop test

• . . .• Narrative retelling task


Weschler Logical Memory Test

Reference text

Anna Thompson of South Boston employed as a cookin a school cafeteria reported at the police station thatshe had been held up on State Street the night beforeand robbed . . . police touched by the woman’s storytook up a collection for her.

An example retelling

Ann Taylor worked in Boston as a cook. And she wasrobbed of sixty-seven dollars. Is that right? And shehad four children and reported at the some kind ofstation. The fellow was sympathetic and made acollection for her so that she can feed the children.


Clinical Scoring of WLM

Reference text, chunked into story elements.

Anna / Thompson / of South / Boston / employed / asa cook / in a school / cafeteria / reported / at the police/ station / that she had been held up / on State Street /the night before / and robbed / . . . / police / touched bythe woman’s story / took up a collection / for her.

An example retelling with 12 recalled story elements.

Ann Taylor worked in Boston as a cook. And she wasrobbed of sixty-seven dollars. Is that right? And shehad four children and reported at the some kind ofstation. The fellow sympathetic and made a collectionfor her so that she can feed the children.


Our Task: Emulate Clinical Scoring

Challenges• Diverse lexical variants• Paraphrasings• Disfluencies• ASR errors


One Approach: ASR + MT

• Compute best hypothesis from the ASR• Align the hypothesis with reference text• Use MT word-alignment model for aligning


Detecting Story Elements


Alternate Approach: Tagging Problem

!"#####$%%$##&'!()*!%###&'$&+*####',-#####%$(,#(./0.1#23./453672#

&.84942:3;#<1<=</24># ?###########@################A##############################?#######

$*-#@BC<42#

$*-#DE%#

',F####$%%$#&'!()*!%###&'$&+*#####$%#####$%GH,-#

',F#

'$F#

$%%$#

$%I#

&'!()*!%# &'$&+*#!J-#

',-#

$%#%$(,#-KL&#

$%GH,-#

L<.203<#M<52:34#9NBO6=>#

P:3#2Q<#E-L#

$R?#

$%R?#

S#

'$FR@#

',FR@#

S#

$R?#

$%R?#

$%IR@#

$%%$R@#

S#

$R?#

$%R?#

S#

&'!()*!%R@#

S#

$R?#

$%R?#

S#

&'$&+*R@#

S#

S#

$R?#

$%R@#

S#$%GH,-R@#

S#

-KG'&R@#

$R?#

$%R?#S#

S#

S#-KG'&R@#

%$(,R@#

P(y |x) =exp(Σd

i=1λiφi(x , y))

Z (x)


Experiments

• Training: retellings from 144 subjects• Testing: retellings from 70 subjects


ASR System

Baseline: Broadcast News ASR System• 4000 clustered pentaphones, 150K Gaussians• 84K vocab, 3M language model ngrams• Multistage discriminative decoding• Performance: 13.1% on RT04

System WER (%)Baseline 47.2AM adaptation 38.0LM adaptation 28.3AM+LM adaptation 25.6


Experiments: Configurations

• Two tagging schemes

Tagging anna rent was dueUO-tags U1 U19 U19 U19BIO-tags B1 B19 I19 I19

• Two types of ASR systems: baseline, adapted• Two types of ASR outputs: 1-best, confusion nets (WCN)


Experiments: Results

1-best WCN ManualBL AM+LM BL AM+LM N/A

Context Independent FeaturesUO 79.3 89.3 80.8 88.1 91.0BIO 78.9 89.0 79.3 87.7 91.1

Context Dependent FeaturesUO 78.4 90.0 79.7 87.7 91.6BIO 78.2 89.3 80.5 88.3 91.9

• WCN > 1-best, when ASR errors are high• F-score from ASR close to manual


MCI Experiments: Results

1-best WCN ManualBL AM+LM BL AM+LM N/A

Context Independent FeaturesUO + SVM 0.65 0.72 0.67 0.75 0.78BIO + SVM 0.65 0.72 0.70 0.76 0.77

Context Dependent FeaturesUO + SVM 0.66 0.73 0.67 0.73 0.79BIO + SVM 0.67 0.73 0.69 0.73 0.79

• Surprisingly high AUC, considering this is only one test!• Best results with WCN, again close to that from manual


Summary

• Fully automate scoring of a cognitive task• Easy to include reverse digit recall, animal recall, etc• Applicable for evaluating fidelity of any narrative retellings


Assessing Health & Well-Being: Wish List

In-Clinic −→ Real-WorldEpisodic −→ ContinualSubjective −→ ObjectiveIntrusive −→ Non-intrusiveLabor-Intensive −→ Automated

• Technology is begining to transform assessments• Physcial Domain: AGPS, accelerometer, in-home sensors• Social and Cognitive Domain: Speech & language

technology!


Acknowledgements

• Post-doctoral researchers: Anthony Stark• Doctoral students: Alireza Bayesteh, Meysam Asgari,

Maider Lehr, Emily Prud’hommeaux• Collaborators: Jeffrey Kaye, Kathy Wild, Brian Roark


Questions? Comments? Suggestions?

For publications, see http://www.csee.ogi.edu/~zak

http://www.csee.ogi.edu/~zak

assessment of social engagement and cognitive function for...

Documents