creating salient summaries for people with episodic memory...
TRANSCRIPT
![Page 1: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/1.jpg)
Creating Salient Summaries for People with Episodic Memory Impairment
Matthew L. Lee
Anind K. Dey
![Page 2: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/2.jpg)
episodic memories
![Page 3: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/3.jpg)
caregiver burden
![Page 4: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/4.jpg)
MemExerciser
![Page 5: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/5.jpg)
too much data!
Only the “good” cues should be reviewed!
![Page 6: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/6.jpg)
FIELD STUDYIN SEARCH OF GOOD MEMORY CUES
(Lee & Dey, ASSETS 2007)
![Page 7: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/7.jpg)
field study: identifying good memory cues
Research Question
What are the characteristics of a good memory cue?
Organization of Autobiographical Memory
Activity information dominant in narratives (Reiser 1985; Taylor 1997; Dijkstra 2006)
Who, What, Where can tell you When (Wagenaar 1986)
![Page 8: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/8.jpg)
method
1. WEAR the Microsoft SenseCam during a personally-significant experience.
(Hodges et al., 2006)
![Page 9: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/9.jpg)
method
2. SORT the photos using a card-sorting approach.
…
![Page 10: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/10.jpg)
types of cues
People: daughter, waitress
Places: facade of a store, the dining room
Actions: playing piano, driving home
Objects: birthday cake, stained-glass window
![Page 11: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/11.jpg)
dominant cue type
Every experience had a majority of one type of cue and thus can be characterized by its dominant cue type.
Experience Photo #1 Photo #2 Photo #3 Photo #4 Photo #5 Photo #6
Family Reunion
Person (widow & daughter)
Object (birthday cake)
Person (an old friend)
Person(nephew)
Person(old friends)
Person(relative)
Church performance
Action (hands clapping)
Action(audience getting into the spirit)
Person(pastor)
Action/Person(unexpected speech)
Action (presentation of family)
Object(stained glass windows)
Trip to Philadelphia
Place (tall ceilings in house)
Place/Action(walking through town)
Place/Action (BBQ in backyard)
Place (walking into town)
Place / Object(backyard; treehouse)
Action (ice cream on drive back)
Visit to History Museum
Object(furniture exhibit)
Object(photography exhibit)
Object(kitchenexhibit)
Object(photograph of slave trade)
Object(artifact exhibit)
Object(letters exhibit)
![Page 12: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/12.jpg)
good cues are memorable
Cues are anchors into the original experience
Participants often discarded photos that they did not remember or recognize from their experience
“I don’t know who this is.”
![Page 13: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/13.jpg)
good cues are distinctive
Different forms
– unusual details of an experience (e.g., spilled water glass)
– prototypical details of an unusual experience (e.g., snow at ski lodge)
prototypical
unusual
spilled water glass too much food unusual windows
snow at ski lodge wife at luncheon widow at memorial service
![Page 14: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/14.jpg)
good cues are personally significant
Personally significant details hold more meaning
Kept photos of people they know, not strangers
Greater personal significance results in a more deeply encoded memory trace
1950’s kitchen exhibit Dad’s old pal friend at garden center
![Page 15: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/15.jpg)
design recommendations
What are the characteristics of good memory cues?
Good cues match the dominant cue type of the experience.
=> automate it!
Good cues are memorable, distinctive, and personally significant.
=> difficult, so rely on human expert.
![Page 16: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/16.jpg)
MemExerciser(Lee & Dey, Ubicomp 2008)
![Page 17: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/17.jpg)
system overview
Sensors Wearable cameraAudio RecorderGPS Logger
LifelogData
Selected Memory Cues
Review
CueChooser CueViewer
Caregiver Input
Automatic
CAPTUREHybrid
SELECTIONActive
REVIEW
![Page 18: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/18.jpg)
automatic capture
“set it and forget it”
Content: Visual (photos) + Audio (voices)
Context: location, movement, light
Wearable digital camera, automatically snaps
photos
GPS location loggerDigital voice recorder
CAPTURE SELECTION REVIEW
SenseCam Voice recorder GPS Logger
![Page 19: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/19.jpg)
hybrid cue selection
Automated Content/Context Analysis
+
Human Caregiver Expertise
CAPTURE SELECTION REVIEW
![Page 20: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/20.jpg)
good memory cues (1)
Good cues match the dominant cue type of the experience. (Lee & Dey, ASSETS 2007)
Experience Type Automated Analysis Technique
People-based Face and voice detection
Location-based GPS location stay points
Object-based Accelerometer movements
Action-based Scene Segmentation
CAPTURE SELECTION REVIEW
![Page 21: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/21.jpg)
good memory cues (2)
Good cues are memorable, distinctive, andpersonally significant (Lee & Dey, ASSETS 2007)
Leverage caregiver’s expertise to:- select meaningful, personally-significant cues- optionally annotate the pictures with voice and drawings to create slideshow
CAPTURE SELECTION REVIEW
![Page 22: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/22.jpg)
active cue review
Self-guided interaction
Tablet-based viewer appliance
Reveals cues progressively to encourage user to think deeply about each cue and encounter “Aha!” moments
CAPTURE SELECTION REVIEW
![Page 23: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/23.jpg)
an afternoon at the zoo
Bill, Alice, and Carol at the Zoo4/11/2008
It was a gorgeous Spring day, perfect for walking through the zoo. We saw lions, tigers, bears, and enjoyed watching them in their natural habitats.
Tap the screen with the pen to begin!
DEMO
![Page 24: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/24.jpg)
an afternoon at the zoo DEMO
![Page 25: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/25.jpg)
an afternoon at the zoo DEMO
![Page 26: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/26.jpg)
an afternoon at the zoo DEMO
![Page 27: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/27.jpg)
an afternoon at the zoo DEMO
![Page 28: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/28.jpg)
an afternoon at the zoo DEMO
![Page 29: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/29.jpg)
EVALUATION
![Page 30: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/30.jpg)
participants
3 couples: husband w/ episodic memory impairment, wife as caregiver (CG)
Age Condition
P1/CG1 81/72 Mild Alzheimer’s disease
P2/CG2 79/76 Mild Cognitive Impairment
P3/CG3 82/75 Mild Cognitive Impairment
![Page 31: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/31.jpg)
study design
3 experimental conditions:
Control (no intervention)
Self-Guided (MemExerciser)
Caregiver-Guided (Traditional SenseCam)
Within-subjects design
![Page 32: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/32.jpg)
study design control condition
Week 1
Week 2
Week 3
Week 4
Experience
Recall test
Recall test
Recall test
Sun Mon Tues Wed Thurs Fri Sat
![Page 33: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/33.jpg)
Sun Mon Tues Wed Thurs Fri Sat
study design self-guided condition
Week 1
Week 2
Week 3
Week 4
Experience
Recall test Person with EMI reviews slideshow on their own
Recall test
Recall test
Caregiver authors slideshow
![Page 34: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/34.jpg)
study design caregiver-guided condition
Sun Mon Tues Wed Thurs Fri Sat
Week 1
Week 2
Week 3
Week 4
Experience
Recall test
Recall test
Caregiver guides person with EMI through the pictures
Caregiver selects photos
![Page 35: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/35.jpg)
measures
Hypotheses
1. Improved recall ability
# of details recalled after 4 weeks
Vividness of recalled details
Memory Confidence Ratings
2. Less additional caregiver burden
Amount of time spent
Interviews with caregivers
Warren G. Harding Former U.S. President (1921-1923)
“A Return to Normalcy”
![Page 36: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/36.jpg)
memory recall
Percentage of details recalled four weeks after the original experience.
0%
20%
40%
60%
80%
100%
Caregiver-Guided (SenseCam Viewer) Self-Guided (MemExerciser)
F[1,5]=6.16, p=0.04
![Page 37: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/37.jpg)
maintaining vividness
Percentage change in number of details judged as vividly remembered after four weeks. (Remember-Know-Guess scale†)
-60%
-40%
-20%
0%
20%
40%
Control Caregiver-Guided (SenseCam
Viewer)
Self-Guided (MemExerciser)
F[2,4]=8.44, p=0.04
†(Gardiner et al, 1993)
![Page 38: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/38.jpg)
supporting memory recall
Deeper processing of memories make them easier to remember.†
CG2 said: "It helps [P2] focus his thoughts, plus [the ambient] voices for the slides makes it easier for him to recall each picture.“
Person with EMI controls the pace of review.
P2 said: “[I] would be more inclined to take more time and look for more details…and get more out of it…because you’re looking for things you don’t normally see.“
†(Craik & Tulving, 1975)
![Page 39: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/39.jpg)
caregiver burden
0
50
100
150
200
250
1 2 3 4 5 6 7 8
Min
ute
s ca
regi
ver
spe
nt
usi
ng
syst
em
Weeks
Caregiver-Guided / SenseCam Viewer
Self-Guided / MemExerciser
Self-GuidedCaregiver-Guided
filter + review = total
CG1 67 mins n/a
CG2 64 mins 19 + 52 = 71 mins
CG3 55 mins 12 + 54 = 66 mins
<
<<
![Page 40: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/40.jpg)
normalcy for caregivers
All caregivers said the main bulk of the “work” was going through the cues together with the person with EMI.
CG2 said “I had to go through it with [P2] every time instead of him
just doing it himself.”
CG1, caregiver-guided review method gives an opportunity to engage her husband in a conversation about a shared experience
![Page 41: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/41.jpg)
summary
Evaluation: looking for normalcySelf-Guided
Better recollection: Ambient Audio + Progressive Revealing of Cues
Lower Caregiver Burden: Computer-assisted filtering + one-time authoring
Caregiver-GuidedShared review => opportunities for conversations
=> active role in preventing further decline
Designed lifelogging system
Automated Capture, Hybrid Cue Selection, Active Cue Review
![Page 42: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/42.jpg)
future work
Clinical evaluation with psychometric testing
Fun interactions: scrapbooking, storytelling, photosharing
Salient summaries with machine learning
![Page 43: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/43.jpg)
embedded assessment
![Page 44: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/44.jpg)
acknowledgements
Special thanks to:
Funding:
This material is based upon work supported in part by the National Science Foundation under grant DGE-0333420.
Matthew L. Lee
Anind K. Dey
Contact:
Website: http://www.cs.cmu.edu/~mllee
![Page 45: Creating Salient Summaries for People with Episodic Memory Impairmentmllee/docs/Lee_CreatingSalientSu… · · 2009-10-18Creating Salient Summaries for People with Episodic Memory](https://reader034.vdocuments.us/reader034/viewer/2022051723/5abb496f7f8b9ab1118cd537/html5/thumbnails/45.jpg)
The End
Automatic
CAPTUREHybrid
SELECTIONActive
REVIEW
Matthew L. Lee
Anind K. Dey
http://www.cs.cmu.edu/~mllee