The effect of regulatory definitions on mining social media for adverse events:

A real-world examination

Michael A. Ibara, Pharm.D.

Head of Digital Healthcare CDISC*

*(Formerly Pfizer, Inc., during which time research was conducted.)

M.  Ibara1,  S.  Stergiopoulos2,  J.  Van  Stekelenborg3,  A.C.  Ianos4,  R.  Ruben5,  P.N.  Naik6,  R.  Boland7.    1CDISC  previously  Pfizer-­‐  Inc,  Pharmacovigilance,  New  York,  USA  2TuRs  Center  for  the  Study  of  Drug  Development,  Project  Management,  Boston,  USA  3Johnson  &  Johnsn,  Lead  Methods  and  Analysis,  New  York,  USA  4Pfizer-­‐  Inc.,  Safety  Risk  Management,  London,  United  Kingdom  5Independent  formerly  ParagonRx  InternaYonal-­‐  LLC,  Risk  Management,  Philadelphia,  USA  6Independent  formerly  TuRs  Center  for  the  Study  of  Drug  Development,  Research  Analyst,  New  York,  USA  7Janssen-­‐  PharmaceuYcal  Companies  of  Johnson  and  Johnson,  TranslaYonal  InformaYcs  &  External  InnovaYon  R&D  IT,  Philadelphia,  USA    

TuRs  CSDD  2013  Social  Media  and  Drug  Development  Study  •  Center  at  the  TuRs  University  School  of  Medicine  (h]p://

csdd.tuRs.edu)  

•  Independent,  academic  group  focusing  on  regulatory,  economic  and  management  aspects  of  drug  development  

•  Grant  funded  and  (cross-­‐industry)  sponsored  studies    

•  Results  have  informed  industry,  Congress,  FoundaYons,  Capital  Market  analysts  and  investors,  Regulatory  Agencies  

•  Working  group:  20  major  biopharma  and  CRO  companies  

•  Study  duraYon:  Conducted  March  through  December  2013  

•  5  Subcommi]ees,  each  with  key  acYviYes  

Tufts CSDD: Furthering the Cause of Pharmaceutical Innovation

           

Staff Presentations

Louis Lasagna Library of Drug Development

Electronic Databases &

Bibliographies

   Sponsorship

     

Surveys & Questionnaires

Information Services

Multi-Company Project Series

   

Manuscripts, White Papers & Book Chapters

 

Research and Publications

 

Industry Programs

Sponsored Research Projects

     

Commentaries & Editorials

Tufts CSDD Executive

Forum    

Public Hearings & Testimony

 

Tufts Center for the Study of Drug Development

Quarterly Updates

   

Bi-Monthly Impact Report

T U F T S U N I V E R S I T Y  Annual

Postgraduate Course

     

Annual Outlook Report

 

Subscription- Based Reports

 

Training & Workshops

Annual Leadership

Course      

Quarterly R&D Mgmt. Report

   

Webinars, Wkshps., & R&D

Roundtables

CSDD 2013 SM & DD Five Subcommittees General

Practices

Guidelines for company-owned sites

Guidelines on

resourcing

Program Planning

Best practices on

open innovation

Best practices on crowdsourc-

ing

Best practices on

patient reported

outcomes

Patient Recruitment

Effective global

recruitment strategies

Recomme-ndations for

future recruitment

Pharmaco-vigilance

Recomme-nded policy

guidance

Principles for identified AE

scenarios

Social Listening

Strategies behind

listening effectively

Operation-alizing social

listening

4

Mining Social Media for Adverse Events

u  There is growing interest and sophistication in mining social media for possible reports of AEs

u  Also growing interest in determining how such activities are interpreted in light of regulations on reporting AEs to regulatory authorities

u  Academic studies have been conducted to address the direct question of the utility of mining social media for AEs, but in biopharma, real world scenarios match up poorly with ideal research conditions

u  It is not always clear to what the extent operational definitions used in the detection and collection of AEs in social media affect the results of that work

5

Objectives

u  Determine the real-world ability to obtain reproducible results using a single definition of “reporter” to mine social media for possible AEs

u  Using available vendors

u  Allowing standard approaches within limits

u  Test whether the operational definition of “reporter” has a direct impact on results (i.e., counts of possible AEs)

u  Determine the extent to which varying the definition of “reporter” changes the counts of possible AEs

6

Design: 6 Vendors / 9 Drugs u  6 vendors recruited based on their stated capabilities of mining social media for possible AEs

u  Predetermine set of drugs was used for the investigation* None were marketed by companies whose members participated in the investigation:

u  Olanzapine (Lanzek, Zypadhera, Zyprexa, Symbyax)

u  Trazodone (Depyrel, Desyrel, Molipaxin, Oleptro, Trazodil, Trazorel, Trialodine, Trittico)

u  Lamotrigine (Lacmictal)

u  Natalizumab (Tysabri, Antegren)

u  Aripiprazole (Abilify, Aripiprex)

u  Esomeprzole (Nexium, Essocam)

u  Duloxetine (Cymbalta, Ariclaim, Xeristar, Yentreve, duzela, Dulane)

u  Nicotine (Nicotrol, Habitrol, Nicoderm, Nicorette, Nicotinell, Commit, Thrive)

u  Aspirin

*Comparable with Leaman, Robert, et al.2010; Nikfarjam, Azadeh, et al. 2011 7

Design: Retrospective Mining u  Retrospective mining of social media for possible AEs for each listed drug

u  From 1Jan2013 to ‘present’ to reach at least 400 hits per drug

u  Greater time period can be investigated if needed to meet hits requirement

u  Possible AE is defined as containing the “4 elements”

u  Identifiable event, patient, drug, reporter

u  Three definitions were standard

u  “Event” as defined broadly in regulations and defined operationally per vendor (multiple symptoms in a single post are defined as a single ‘hit’)

u  “Patient” defined broadly in regulations and in absence of vendor’s definition will be “Knowledge of an individual experiencing the event with at least one of the following patient qualifiers: a pronoun or noun implying a human; an age or age category; gender; initials; date of birth; name; or patient ID number

u  “Drug” as the drug in question using vendor’s procedures. 8

Identifiable Reporter – 4 Levels u  LENIENT

u  Post exists, i.e., no requirement to identify a reporter

u  Used in the initial data collection

u  LOW

u  Any type of information suggesting there is an actual reporter (e.g., acronym, pseudonym, proper name, email address)

u  STANDARD

u  At least one piece of identifiable information for the reporter - i.e,. local identifier for the platform that allows contacting (e.g., Facebook name) OR a validly formatted email address OR validly formatted phone number

u  STRICT

u  Must match standard criteria and in addition have additional piece of identifying information such as valid phone number, valid user name from site, mention of geographic location of the person in question 9

“Hits” u  For each drug a minimum of 400 ‘hits’ according to the Lenient

definition was the goal

u  Type of reporter identifying information was collected by vendors, but was masked to subteam investigators and supplied as counts

u  E.g., 10 cases with single valid mail addresses, 5 cases with both valid email addresses and user names

u  Search was limited to English

10

Design: Data Sources / Collection

u  Four standard data sources u  Facebook

u  Twitter

u  Dailystrength.org

u  Drugs.com

u  Vendors varied in terms of methods of collection and extent to which sources were utilized

u  As this was a real-world examination, no attempt was made to develop a single consistent dataset

u  No data was collected that was not publically available

11

Results: Real-world Variability

u  Began with 6 vendors claiming ability to independently complete request. Completed with 4 vendors

u  Significant differences across vendors in methods

u  Operational definitions of drugs, terms

u  Lexicon use, sophistication

u  Sourcing and amount of data collected

u  Algorithms used

u  Type and extent of curation

12

Results: Overall Findings

u  It is not possible to pool results or make direct comparisons given the variability in methods

u  Each set of results must be treated individually based on the unique set of methods used

13

Vendor Facebook (‘hits’ per examined) Twitter (‘hits’ per examined)

1 0 / 34 17 / 239

2 43 / 11,431 [human] 141 / 37,409 [human]

3 373 / 9,823 [human] 32 / 33,167 [human]

4 1,200 / 85,220 1,318 / 105,018

Example: ‘Hits’ on Facebook and Twitter by Vendor for Olanzapine

“Identifiable Reporter”… irrelevant?

u  Very little difference between bottom three categories. The only real change was seen with “strict” definition, which dramatically reduced hits.

u  For Facebook and Twitter, a valid email address is a requirement, thus rendering a distinction between “lenient” and “low” meaningless.

u  This may point to the discord between a definition created before the world wide web existed, applied to social media today.

14

“Identifiable Event”…subject to change

u  Vendors varied in their operational definition of an “event” based on their specific assumptions and working models.

u  In one instance the vendor used definitions already developed and used in their daily business – with the effect that the precision was high but sensitivity poor.

u  What events qualified as hits also varied with lexicon use and sophistication, and with application of curation.

15

Each Platform has unique considerations

u  While it may seem obvious upon reflection, it is important to realize that searching “social media” reduces to a collection of specific methodologies for each platform: Facebook vs Twitter vs Drug Info Sites vs Patient engagement sites, etc.

u  Platforms influence was found not only in how data was collected, but how much could be collected (directly from API, from 3rd parties, stored by vendor)

u  Twitter provided the most initial data to review, but it also provided fewer hits per unit number examined

16

Curation / Machine approach Tradeoff

u  In instances where human curation was used, the precision and specificity improved (although in two different vendors the curation method was not comparable)

u  However, drugs with a very large initial hit rate (e.g., aspirin, nicotine) broke the human curation steps, and required various workaround both in machine and human approaches

u  There is as of yet no solely-machine-based approach that approximates human curation, but there is no human curation approach that can handle very large numbers in a cost-effective manner

u  This is an area ready for innovation (e.g., a machine-human-based solution that takes advantage of crowd-sourcing?)

17

Five suggested areas of methodological focus

u  Operational definitions of drugs, terms

u  Lexicon use, sophistication

u  Sourcing and amount of data collected

u  Algorithms used

u  Type and extent of curation

18

Recommendations

u  For organizations required to report on safety of drugs/devices to regulatory authorities, more transparency and sharing of methods is highly desirable

u  An industry-agreed “good social-media / PV research practices” that goes beyond the current general recommendations, and begins to address those areas raised here, would be timely

u  A re-examination of the regulatory definitions of the “4 elements” is needed, to ground them in modern concepts that include social media and the internet – a greater specificity in regulatory definitions will ease the burden on the reporter and lead to greater standardization in methods

19

Thank You