overview of social listening methods · doublevision couldn’t see visión doble blind googley...
TRANSCRIPT
![Page 1: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/1.jpg)
June 3, 2016
Carrie Pierce, Director, Client Delivery
Epidemico, Inc.
Cardiac Safety Research Consortium • FDA White Oak Campus
Overview of Social Listening Methods
![Page 2: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/2.jpg)
![Page 3: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/3.jpg)
Machine Learning
vs.
Natural Language Processing
![Page 4: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/4.jpg)
Machine LearningFind reports of importance.
Natural Language ProcessingExtract the most relevant elements.
![Page 5: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/5.jpg)
Data Processing
![Page 6: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/6.jpg)
![Page 7: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/7.jpg)
Collect unstructured data from
social media APIs, third-party
authorized resellers, and
automated scraping.
Acquire
![Page 8: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/8.jpg)
detectsentiment
languagetranslation
statistics consolidatemultiples
detectadverseevents
detectbenefits
de-identifygeo-tag
Data are passed through a series
of apps, emerging as meaningful
bits of information.
Process
![Page 9: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/9.jpg)
detectsentiment
languagetranslation
statistics consolidatemultiples
detectadverseevents
detectbenefits
de-identifygeo-tag API mobile RSS tables reportsCSV
,visuals
Relevant data are passed to another
series of apps in preparation for
human interpretation and analysis.
Export
![Page 10: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/10.jpg)
API mobile RSS tables reportsCSV
,visuals
Relevant data are passed to another
series of apps in preparation for
human interpretation and analysis.
Export
![Page 11: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/11.jpg)
AstheniaMedDRA: 10003549
DizzinessMedDRA: 10013573
“Proto-AE”
![Page 12: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/12.jpg)
lost their eyesight
seeing weird color
seeing weird colour
doublevision
couldn’t see
visión doble
blind
googley eyed
blurry vision
changes in visioncross eyed
seeing weird
vision change
blindness
cross visionvisual snow
googly eyed
seeing double
making me eat like a mouse
lost appetite
#notevenhungry
appetite is nonexistent
apetite surpressed
didn’t get hungry
dont want to eat
killed my apetite
miss feeling hungry
killed my appetite
can’t eat
sin hambre
lost my appetite
no appetitey
lack of apetite
stomach small
lost teh appetite
never hungrynever want to eat
cant eat
couldntcrosseyed
blurry
anorexic
apetite surpressed
lack of apetite
killed my apetite lost teh appetite
making me eat like a mouse
never want to eat
Implied
#notevenhungry
no appetitey
Invented words and hashtags
googley eyed
googly eyed
seeing weird colour
seeing weird color
Varied Spelling
sin hambre
visión doble
Other Languages
blurry vision
Emoticons
Typos
![Page 13: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/13.jpg)
making me eat like a mouse
anorexic
lost appetite
#notevenhungry
appetite is nonexistent
apetite surpressed
didn’t get hungry
dont want to eat
miss feeling hungry
killed my appetite
can’t eat
sin hambre
lost my appetite
no appetitey
lack of apetite
stomach small
lost teh appetite
never hungrynever want to eat
cant eat
lost their eyesight
seeing weird color
seeing weird colour
doublevision
couldn’t see
visión doble
blind
googley eyed
changes in visioncross eyed
seeing weird
vision change
blindness
cross visionvisual snow
googly eyed
seeing double
couldntcrosseyed
blurry
killed my apetite
blurry vision
Decreased appetite
MedDRA 10061428
Loss of appetite
SNOMED 79890006
Visual impairment
MedDRA 10047571
Visual impairment
SNOMED 397540003
![Page 14: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/14.jpg)
Albuterol has me feeling all sorts of dizzy and weak this morning
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Proportion out of all
keeps
Keep
Proportion of posts token never Shows up in
None
Proportion out of all trash
Trash
albuterolalbuterol hasalbuterol has mehas mehas me feelingfeelingfeeling allfeeling all sortssortssorts of dizzydizzydizzy and weakweakweak thisweak this morningmorningmorning
1. Tokenization 2. Score calculation
For each token “w”
b(w) = (number of positives containing w)
(total number of positives)
g(w) = (number of negatives containing w)
(total number of negatives)
Classifier Performance
![Page 15: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/15.jpg)
PRO CONINDICATOR
SCORE
0.8208
Classifier Performance
![Page 16: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/16.jpg)
Positive Predictive Value or Precision
7 OF 10 posts contain adverse event information (0.68).
Can increase to 100% with manual curation (may vary by product).
If the algorithm says it is a Proto-AE, then 7 out of 10 times is actually is.
![Page 17: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/17.jpg)
Sensitivity or Recall
Automated tools identify 9 OF 10 adverse events
across all products, all time, all data sources (0.88).
The algorithm correctly identifies 9 out of 10 Proto-AEs from the pool of everything.
![Page 19: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/19.jpg)
Back Up Slides
![Page 20: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/20.jpg)
![Page 21: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/21.jpg)
DIGITAL
DRUG SAFE
TY
SURVEILLA
NCE
• Co-authored with FDA
• Evaluate similarity between Tw
itter posts mentioning AEs and
FAERS spontaneous data
• Collected and classified 6.9MM
Twitter posts mentioning 23 pr
oducts
• Served as validation study for
system and patients’ voice
• Groundwork (for) research in t
his area
Digital drug safety surveillance: monitoring
pharmaceutical products in twitter.
Freifeld CC, Brownstein JS, Menone CM, B
ao W, Filice R, Kass-Hout T, Dasgupta N. D
rug Safety. 2014 May;37(5):343-50. doi: 10
.1007/s40264-014-0155-x. PMID: 2477765
3
![Page 22: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/22.jpg)
Rank-order concordance observed at the MedDRA SOC level between Twi
tter and FAERS data
![Page 23: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/23.jpg)
WHAT WE KNOW
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Clinical Trials Sponsor AE Practice Data Social Media
PRO SeriousnessCompletene
ss
Social media can elucidate what
patients experience most frequently.
1. Patient-Reported Outcomes
Traditional PV focus on most serious
of events
may be less central in social media.
2. Seriousness
Social media posts may by less
complete.
Conversely, contain less sensitive
identifying
information than electronic health
records.
3. Completeness
Three Dimensions of Bias
Each source of product safety information provides different perspectives on patient
experienceHypothetical data for illustrative purposes only.
![Page 24: Overview of Social Listening Methods · doublevision couldn’t see visión doble blind googley eyed blurry vision cross eyed changes in vision seeing weird vision change blindness](https://reader034.vdocuments.us/reader034/viewer/2022042313/5ee021bbad6a402d666b5fee/html5/thumbnails/24.jpg)
AllPostsProto-AEs
MWS TOTALS
2016
Public posts from Facebook, Twitter
& patient forums, back to 2012
Posts collected
63,000,000
Curator inter-rater reliability 0.88
Manually classified
380,000
~2.8% of all product mentions
Proto-AEs
1,700,000