Proceedings of the 2nd Workshop on Noisy User-generated Text,pages 56–63, Osaka, Japan, December 11 2016.

Analysis of Twitter Data forPostmarketing Surveillance in Pharmacovigilance

Julie Pain and Jessie Levacher and Adam QuinquenelINSA Rouen

Avenue de l’Universite, Saint-tienne-du-Rouvray, France{julie.pain,jessie.levacher,adam.quinquenel}@insa-rouen.fr

Anja BelzComputing, Engineering and Mathematics

University of Brighton, Lewes Road, Brighton, UKa.s.belz@brighton.ac.uk

Abstract

Postmarketing surveillance (PMS) has thevital aim to monitor effects of drugs af-ter release for use by the general pop-ulation, but suffers from under-reportingand limited coverage. Automatic meth-ods for detecting drug effect reports, es-pecially for social media, could vastly in-crease the scope of PMS. Very few auto-matic PMS methods are currently avail-able, in particular for the messy text typesencountered on Twitter. In this paper wedescribe first results for developing PMSmethods specifically for tweets. We de-scribe the corpus of 125,669 tweets wehave created and annotated to train and testthe tools. We find that generic tools per-form well for tweet-level language iden-tification and tweet-level sentiment anal-ysis (both 0.94 F1-Score). For detectionof effect mentions we are able to achieve0.87 F1-Score, while effect-level adverse-vs.-beneficial analysis proves harder withan F1-Score of 0.64. Among other things,our results indicate that MetaMap seman-tic types provide a very promising ba-sis for identifying drug effect mentions intweets.

1 Introduction

Postmarketing surveillance (PMS) is a vital part ofpharmacovigilance, taking place during the final,post-licensing phase of drug development whenthe effects on larger numbers of users, who mayhave other conditions and take other medicinesthan those included in pre-approval tests, can be

assessed. PMS is implemented in passive na-tional reporting schemes such as the Yellow CardScheme in the UK and MedWatch in the US; it isalso implemented as active surveillance, e.g. or-ganisations such as the MHRA in the UK and theFDA in the US conduct post-approval studies andpostmarketing surveys.

Existing PMS methods either rely on healthpractitioners and patients to report adverse effectsto which only a small (self-selected) proportionof patients in particular will contribute (report-ing schemes), or they involve single products andsmall numbers of participants (surveys). Moregenerally, such methods are more likely to iden-tify (i) very serious problems, and (ii) problemsrelating to newly released drugs and drugs alreadyunder continuous surveillance.

The ultimate goal of our work is to develop textanalysis techniques to facilitate automatic, contin-uous and large-scale monitoring of adverse drugreactions (ADRs), and more generally of effects,beneficial or otherwise, reported for given drugs.There is currently huge interest in such methodsin the pharmaceutical industry, in particular wherethey can be used to monitor what is being said onsocial media about specific drugs. Automatic PMSmethods are also of interest to national regulatorybodies.

The potential benefits of high-precision auto-matic ADR detection methods for social media aregreat: such methods would ameliorate the recog-nised problem of under-reporting of ADRs via ex-isting channels (Lardon et al., 2015), and could in-form the design of post-approval studies. Studieshave already demonstrated that analysis of socialmedia contents (manual analysis so far) can leadto the discovery of serious side effects, e.g. Abou

56

Taam et al. (2014) retrospectively identified a se-vere side effect on the basis of user content postedmonths before the drug in question was withdrawnbecause of the same side effect. The challenge isto automate this process so that suitably large por-tions of social media can be scanned.

2 Related Research

The main interest has so far been in discoveringADRs, defined by the World Health Organisationas a “a response to a medicinal product which isnoxious and unintended and which occurs at dosesnormally used in [humans] for the prophylaxis, di-agnosis or therapy of disease or for the restoration,correction or modification of physiological func-tion” (WHO, No date).

While studies involving manual analysis haveconfirmed the usefulness of social media contentin ADR identification (see previous section), man-ual analysis is necessarily limited to minute frac-tions of available online content. Automatic anal-ysis of online content for ADR detection remains ahuge challenge, in particular when applied to Twit-ter, due to the messy and ‘un-language-like’ natureof tweets, and it remains a small field.

Last year, Lardon et al. (2015) reported the re-sults of a very thorough scoping review of (i) man-ual and computer-aided identification of ADRs,and (ii) semi-automatic and automatic identifica-tion of ADRs, from social media. The authorsfound just 13 papers on the latter, of which justone paper focused on Twitter data (the remainderused content from online health forums).

That one paper (Bian et al., 2012), among thevery earliest on this precise topic, reported the cre-ation and annotation of a corpus of drug-relatedtweets, and results for training SVM classifiersto detect (a) whether the tweeter was reportingtheir own experience, and (b) whether a givenown-experience tweet contained a mention of anADR. The study targeted five drugs that werethe subject of a pre-approval clinical study duringa known time window, and analysed tweets thatwere posted during the same time window, on theassumption that study participants were likely totweet about their experience. 489 tweets by 424users were identified in this way, reduced to 239users after removal of re-tweets and non-Englishtweets. Tweets by the same user were concate-nated and annotated manually. An SVM classifierwas trained to classify texts into (i) users report-

ing their own experience of taking the drug, and(ii) user reporting someone else’s experience; theclassifier used 171 features, some based on key-words and word types, hashtags and user names;others based on a MetaMap (Aronson, 2001) anal-ysis of the texts yielding UMLS meta codes. Clas-sification into first-hand vs. second-hand reportingachieved an average accuracy of 0.74, and classi-fication of first-hand-reporting tweets into ADR-mentioning and not ADR-mentioning was also re-ported as achieving 0.74 accuracy.

Ginn et al. (2014) created a corpus of annotatedtweets and results for training classifiers to de-tect ADR mentions. Two members of the projectteam with “medical or biological science back-ground” annotated 10,822 tweets related to 74drugs for mentions of an ADR, indication or ben-eficial effect, and each ADR/indication/beneficialeffect with corresponding UMLS concepts. The74 drugs included some for which ADRs arewell established and some for which they arenot. Product names, common names and mis-spellings were used for each drug; tweets withURLs were removed (deemed to be mostly ad-verts). Inter-annotator agreement (IAA) waskappa = 0.69. Ginn et al. trained Naive Bayes(NB) and SVM classifiers on the binary annota-tions (presence/absence of effect mentions), usingbag-of-words feature vectors, text normalisationand lemmatisation. Three versions of the corpuswere created with different levels of imbalance be-tween positive and negative examples. Best Preci-sion for detecting effects was 0.89 (Recall = 0.695;F-Score=0.78; Accuracy = 0.746) for NB and theexactly balanced version of the corpus; Accuracywas around 0.75 for NB across the three differ-ently balanced corpus versions.

Two members of the Ginn et al. team subse-quently reported a larger set of results (Sarker andGonzalez, 2015), using the Twitter data above, butalso data from an online health forum, and an ex-isting corpus of clinical reports. They describe theearlier results as having been obtained by “classi-fication via under-sampling, which yields higherADR F-scores at the expense of overall accuracy”(Footnote 22). Sarker & Gonzalez decided not touse under-sampling or balancing, and report ADRF-Scores of 0.538 and Accuracy 86.2 for the Twit-ter data with a different type of SVM than in theearlier work, going up to 0.597 and 90.1, respec-tively, when the Twitter data is supplemented by

57

the data from the online health forum.In summary, the tasks for which tools have so

far been built under the heading of automatic ADRdetection in tweets are all binary (note that the an-notations in the data sets include a wider varietyof information):

• for a given tweet, decide whether it is in En-glish or not (Bian et al., 2012);

• for a given set of tweets by the same Twitteruser and a given drug, decide whether or notthe user themself is taking (or has taken) thedrug (Bian et al., 2012);

• for a given tweet, decide whether or not itcontains a mention of an ADR (Bian et al.,2012; Ginn et al., 2014; Sarker and Gonza-lez, 2015).

In this paper, we address (i) the first task above(Section 5), (ii) a more general case of the last taskabove (mentions of all drug reactions, adverse ornot, Section 7), as well as two new, non-binarytasks (for this domain): (iii) for a given tweet,decide what its sentiment is (Section 6); and (iv)for a tweet that has been identified as containing amention of a drug reaction (as in ii above), decidewhether the drug reaction is beneficial, adverse orneither (Section 8).

3 Automatic Drug Effect Detection:Task, Annotation, Tools

Automatic drug effect detection for Twitter issomething of a worst case scenario for NLP: notonly are tweets one of the messiest, most abbrevi-ated forms of short text, for which standard NLPtools do not tend to work well; but automatic ADRdetection is also extremely hard, even for very wellbehaved types of text, due among other factors tothe large variety of ways in which the same drugeffects can be referred to, and to the complex rela-tionships between drugs and effects.

The research reviewed in the previous sectionfocused on detecting adverse drug effects (the an-notations in some cases also include other kindsof effects, but the tools were for ADR detec-tion). There are three reasons why we addressa wider set of effects: (a) the related researcharea of automatic drug discovery/development, in-cluding our industrial advisory partner, is inter-ested in identifying all effects claimed for a givendrug/compound; (b) post-marketing drug monitor-ing, especially by pharmaceutical companies, is

also interested in a wider set of effects; and (c) itmay make the detection task harder if one specificsubtype of drug effects is targeted only.

To elaborate on that last point, ADRs can beseen as a special case of a more general binarydrug-effect relation has effect(drug, effect). Ourcorpus (Section 4) contains diverse examples, in-cluding the following:1

has effect(“glucosamine”, “got me feelingsome typa way”)has effect(“azathioprine”, “think I’ve got#shingles”)has effect(“daunorubicin”,“made his weered”)

Reports of effects, whether positive, negative, orneither, are likely to have similar patterns and cuephrases (e.g. ‘I’ve got’, ‘got me feeling’). It maybe the case that it is easier to identify all reports ofeffects, not just negative ones, and then classifythose into adverse, beneficial, mild, severe, andother dimensions.

We construe drug-effect detection as a knowl-edge extraction task, where the objective is toidentify mentions of drug effects and to fill relationtemplates such as has effect(drug, effect). In thisgeneral form such mentions can be incorporatedinto knowledge graphs and combined with otherkinds of relations about drugs and health. Suchknowledge graphs can be easily visualised, andare used for example in automatic drug discoveryresearch where mentions of drug effects from so-cial media could provide a useful complementarysource of information.

In this paper, we report first results towards theabove knowledge extraction task. We use the term‘drug effect’ to refer to both cases where the ef-fect is specified and cases that might more intu-itively be described as ‘properties’ where thereis no specified effect (see also Section 8 below).Section 4 describes how we collected our set oftweets, and the text cleaning and normalisation weperform. Sections 5, 6, and 7 report results fromour experiments on language identification, senti-ment analysis and drug effect detection in tweets,respectively, while Section 8 reports results fromclassifying drug effects into beneficial, adverseand other.

1The text would be mapped to e.g. UMLS codes beforebeing incorporated into knowledge bases, as below.

58

4 Data Collection

We compiled a list of drug names by extractingall names of approved drugs from DrugBank,2

1,999 in total. Next, for each drug name we col-lected the HTML files of all tweets returned forthe search “Drug Name” OR #DrugName (e.g.“Acetic Acid” OR #AceticAcid), up to a maximumof 100 tweets per drug name, and used BeautifulSoup3 to extract the information we needed (tweettext, tweet id, timestamp, etc.) from the HTMLfiles containing the search results (tweets).

The results was a corpus4 of 125,669 tweets,with an average of 62.87 tweets collected for eachdrug. Figure 1 gives a more detailed picture ofnumber of tweets per drug: most drugs have 70-95 tweets, 19 have none, and 15 have 100 ormore. Note that we did not expand the set of drugnames with misspelt variants, generic or commonnames as done in some related research (Ginn etal., 2014).

We used three versions of the tweet texts: (i) intheir raw form; (ii) cleaned, with URLs and usernames replaced by tags, and hash tags with the ‘#’removed; and (iii) with hashtags additionally con-verted to likely strings of words, using a tool madeavailable on StackOverflow.5

5 Language Identification

In order to be able to analyse tweets in a meaning-ful way, moreover using text analysis tools trainedfor English, it is important that we can reliably fil-ter out non-English tweets. Twitter tags tweets forlanguage, so part of our first set of experimentswas to test the reliability of the Twitter languagetags, as well as to see how it compared againstexisting language identification tools. For the lat-ter we chose the three tools that were, combinedas an ensemble method, identified by Lui & Bald-win (2014) as the best language identification toolsfor tweets: langid.py (Lui and Baldwin, 2012),LangDetect,6 and CLD2.7

We randomly selected a subset of 300 tweets(test set A) and manually annotated each for En-glish vs. other. Table 5 shows results for eachof the methods tested, as well as for combining

2http://www.drugbank.ca/drugs3https://www.crummy.com/software/BeautifulSoup4We will publish the corpus along with this paper.5By anonymous user Generic Human.6Y. Nakatani: http://www.slideshare.net/shuyo/language-

detection-library-for-java7Language identification in Google Chrome.

Twitt

er

lang

id.p

y

CL

D2

Lan

gDet

ect

Maj

ority

vote

P 0.995 0.991 0.986 0.973 0.995R 0.861 0.889 0.877 0.898 0.861F1 0.923 0.937 0.928 0.933 0.923

Table 1: Language identification results for Twit-ter tags and three language identification tools ontest set A of 300 manually annotated tweets (Re-call, Precision, F1-Score, for the English class).

langid.py, CLD2 and LangDetect in majority vot-ing. langid.py outperforms the others, includingTwitter, slightly on the raw tweets (see end of Sec-tion 4). First cleaning/normalising tweets led to aslight improvement for CLD2, and a slight deteri-oration for languid.py; additionally parsing hash-tags led to slight improvement for languid.py, andslight deteriorations for CLD2 and LangDetect.The final best result is an F1-Score of 0.94 forlangid.py. This confirms (for this domain) previ-ous results that good language identification toolsoutperform Twitter language tags (Lui and Bald-win, 2014).

6 Sentiment Analysis

Following the language identification experi-ments, we filtered out the non-English tweets us-ing langid.py, which left us with 93,697 tweets andan average of 46.87 tweets per drug name. In theexperiments in this section, we aim to predict theoverall sentiment of an (English) tweet, indepen-dently of whether a drug effect is being reported.

For this purpose we selected a different test setof 300 random tweets (test set B) from our corpusand annotated each tweet for positive, negative, orneutral. We tested the 18 sentiment analysis (SA)tools made available in the ifeel package:8 Afinn,Emolex, Emoticons, EmoticonDS, HappinessIn-dex, MPQAAdapter, NRCHashtagSentimentLex-icon, OpinionLexicon, Panas-t, Sann, SASA,SenticNet, Sentiment140Lexicon, SentiStrength-Adapter, SentiWordNet, SoCal, UmigonAdapter,Vader, and a majority vote by all methods. Theifeel team have reported comparative results formany of these tools (Nunes Ribeiro et al., 2010;Ribeiro et al., 2016). For our task, the methods

8http://blackbird.dcc.ufmg.br:1210/ (accessed: 06/2016)

59

Number of tweets

Figure 1: Number of drugs (y) that have a given number of tweets (x). E.g. there are 19 drugs with 0tweets, and 51 drugs with 88 tweets.

achieved an average F-Score of 0.792, and Ac-curacy of 0.753. The best three were, by a con-siderable margin: Emoticons, Panas-t and SANNSA, across all three variants of our corpus. Verybriefly, these use the following approaches.

Emoticons: Given lists of positive and neutralemotions, if a text contains an emoticon it is as-signed the polarity of the first one, else a text isdeemed neutral (Goncalves et al., 2013a).

Panas-t: Provides association strengths betweenwords and eleven moods including surprise, fear,guilt, joviality, attentiveness, fear, etc. (Goncalveset al., 2013b).

SANN SA: A dictionary-based rule-based senti-ment classifier using the MPQA polarity lexicon(Pappas et al., 2013).

Results for these three classifiers and the majorityvote by all 18 systems in ifeel are shown in Ta-ble 2; we are only showing results for the raw ver-sion of the corpus (see end of Section 4), becauseresults were identical over the three corpus ver-sions, i.e. cleaning, normalising and parsing hash-tags did not help with sentiment analysis.

Em

otic

ons

Pana

s-t

SAN

N

Maj

ority

vote

:

(all

18)

F1-measure 0.941 0.929 0.920 0.665Accuracy 0.936 0.932 0.932 0.576

Table 2: Tweet-level sentiment analysis results for3 best sentiment analysis tools on test set B of 300manually annotated tweets (weighted-average F1-score over POS, NEG and NEU labels; Accuracy).

Out of the 300 tweets in test set B, 259 wereannotated as neutral, 22 as positive, and 19 as neg-ative. This gives a very high most-frequent-labelbaseline of 0.863 Accuracy which the three bestclassifiers, however, outperform comfortably.

7 Detection of Drug Effect Mentions

We additionally annotated test set B for drug ef-fects with three mention labels: mention of an ef-fect, mention of a property (without specified ef-fect), and none. 81 of the 300 tweets containedan effect mention (sometimes more than one), 10

60

tweets a property mention, 4 tweets both an ef-fect and a property mention, and 213 contained nomentions.

The experiments in this section were aimed atdistinguishing tweets that mention drug effectsand properties from tweets that do not. As astarting point we process each of our tweets withMetaMap (Aronson, 2001), a tool for recognis-ing UMLS9 concepts in text. For a given inputtext (tweet in our case) MetaMap produces a setof concept vectors as the output, each scored forrelevance.

One of the elements in concept vectors indi-cates ‘semantic type’ (semtype); semtypes are aset of broad subject categories over all conceptsrepresented in the UMLS metathesaurus. Thereare 133 different semtypes which fall into sixmajor groupings corresponding to concepts re-lated to organism, anatomical structure, biologi-cal function, chemical object, physical object, andidea/concept. For example, a semtype much morecommonly seen with tweets that do contain an ef-fect mention is ‘Pathologic Function’ (83%); ex-amples of semtypes much more commonly seenwith tweets that do not contain an effect men-tion include ‘Plant’ (96%), ‘Receptor’ (95%), and‘Laboratory Procedure’ (93%).

In this first experiment aimed at detecting effectmentions, we use all 133 semtypes, resulting in bi-nary feature vectors of length 133. We used thesepaired with the corresponding effect/property la-bels (+ME = mention of an effect/property, and-ME = none) as training data to train classifiersfor this task, using 10-fold cross-validation in test-ing. The most-frequent-label (-ME) baseline forthis task is 0.71 Accuracy.

Table 3 shows results for the three classifiers wetested (Multinomial Naive Bayes, SVM, and Lo-gistic Regression) in terms of overall as well asper-class Recall, Precision and F1-Score. For allclasses, SVM performed best with a weighted av-erage F1-Score of 0.873. For the +ME class, thelogistic regression classifier did best, with an F1-Score of 0.914. On the -ME class, SVM was bestwith F1=0.823. Results in Table 3 are for the rawversion of the corpus (see end of Section 4); resultswere identical for the raw and cleaned versions,while additionally parsing hashtags worsened re-

9The Unified Medical Language System (UMLS)“integrates and distributes key terminology, classifica-tion and coding standards, and associated resources”(https://www.nlm.nih.gov/research/umls/).

sults very slightly for the NB classifier only.Looking at the Precision scores for +ME, which

is arguably the most important measure fromthe perspective of incorporating information intoknowledge graphs, all three classifiers performedextremely well. Especially considering we had atiny training set, this indicates that the MetaMapsemtypes form a highly reliable basis for identify-ing tweets that mention drug effects/properties.

8 Adverse vs. Beneficial Effects

In test set B, we also annotated those tweets withmentions of an effect or property with a further la-bel encoding whether the effect/property was ad-verse, beneficial or neutral. 23 tweets were la-beled as having an effect/property that is adverse,57 tweets having a beneficial one, and 11 a neutralone, with 5 tweets having more than one of these.

In the experiments in this section, for the sub-set of tweets which were identified as contain-ing a drug effect/property mention, we wanted tosee whether any of the sentiment analysers wouldbe able to predict whether the effect/propertymentioned was an adverse, beneficial or neutralone. We tested the same 18 sentiment analy-sers against the effect sentiment labels (setting ad-verse=negative, beneficial=positive, and neutral).Some of the sentiment analysers, despite not be-ing designed for this task, did reasonably well atit; the four best ones were the following:

EmoticonsDS: Uses a large sentiment-scoredword list based on the co-occurrence of each to-ken with emoticons in a corpus of over 1.5 billionmessages (Hannak et al., 2012).

SenticNet: A semantic and affective resourcefor concept-level sentiment analysis, modellingthe polarities of common-sense concepts and re-lations between them (Cambria et al., 2014).

SentiWordNet: Lexical SA tool based on Word-Net using polarity scores associated with WordNetsynsets (Baccianella et al., 2010).

AFINN: Twitter based sentiment lexicon pro-viding emotion ratings for words (Nielsen, 2011).

Table 4 shows the results for these four tools. In-terestingly, there is no overlap with the three toolsthat did best at the standard SA task (Table 2), infact those three methods were the three worst onesat this task.

61

Both classes +ME class -ME classClassifier R P F1 R P F1 R P F1Multinomial NB 0.847 0.852 0.849 0.848 0.981 0.909 0.961 0.688 0.794Logistic Regression 0.85 0.851 0.850 0.855 0.984 0.914 0.969 0.689 0.799SVM 0.855 0.892 0.873 0.8 1.0 0.887 1.0 0.71 0.823

Table 3: Results for detection of effect mentions (ME) on test set B of 300 manually annotated tweetswith 10-fold cross-validation (Recall, Precision, F1-scores for all classes, ME class, and not ME class).

Em

otic

onsD

S

Sent

icN

et

Sent

iWor

dNet

AFI

NN

Maj

ority

vote

(all

18)

F1 0.638 0.621 0.616 0.615 0.627Acc 0.592 0.595 0.592 0.597 0.604

Table 4: Effect-level sentiment analysis results for4 best sentiment analysis tools on test set B of 300manually annotated tweets (weighted-average F1-score over ADV, BEN and NEU labels; Accuracy).

9 Conclusions and Further Work

In this paper we described our new corpus of125,669 tweets for 1,999 drug names. The cor-pus includes one randomly selected subset (A) of300 tweets which is annotated for language, andanother set (B) of 300 random tweets which isannotated for overall sentiment and drug effectmentions. The corpus represents many more drugnames than in comparable existing corpora, but sofar has only a small number of annotated tweets.

We reported results for language identificationand sentiment analysis. One of the language iden-tification tools tested (langid.py) outperforms theTwitter language tags in this domain. Tweet-levelsentiment analysis achieved a best result of 0.94weighted average F1-Score (Emoticons method).

Perhaps the most surprising result we report isthat a straightforward approach to training a clas-sifier to distinguish tweets that mention drug ef-fects from those that do not, achieves an overallF1-Score of 0.873 and perfect precision for thepositive class (+ME). This indicates that MetaMapsemtypes are a strong basis for predicting drug ef-fect mentions. However, using sentiment analy-sis tools for the new task of predicting the polar-ity of an effect (adverse vs. beneficial vs. neutral)achieved best results of just 0.64 weighted averageF1-Score (EmoticonsDS).

In future work, we are planning to expand our

annotations to allow more effective training ofclassifiers in particular to address the latter task,as well as generally to expand our corpus to in-clude tweets containing common, colloquial andgeneric names and common misspellings of ourdrug names. In terms of methodology, we are cur-rently working on drug effect extraction, i.e. iden-tifying the span of words in a tweet that describesthe effect.

ReferencesM Abou Taam, C Rossard, L Cantaloube, N Bous-

caren, G Roche, L Pochard, F Montastruc, A Herx-heimer, JL Montastruc, and H Bagheri. 2014. Anal-ysis of patients’ narratives posted on social mediawebsites on Benfluorex’s (mediator R©) withdrawalin france. Journal of Clinical Pharmacy and Thera-peutics, 39(1):53–55.

Alan R Aronson. 2001. Effective mapping of biomed-ical text to the umls metathesaurus: the metamapprogram. In Proceedings of the AMIA Symposium,page 17. American Medical Informatics Associa-tion.

Stefano Baccianella, Andrea Esuli, and Fabrizio Sebas-tiani. 2010. Sentiwordnet 3.0: An enhanced lexicalresource for sentiment analysis and opinion mining.In LREC, volume 10, pages 2200–2204.

Jiang Bian, Umit Topaloglu, and Fan Yu. 2012. To-wards large-scale twitter mining for drug-related ad-verse events. In Proceedings of the 2012 interna-tional Workshop on Smart Health and Wellbeing,pages 25–32. ACM.

Erik Cambria, Daniel Olsher, and Dheeraj Rajagopal.2014. Senticnet 3: a common and common-senseknowledge base for cognition-driven sentiment anal-ysis. In Proceedings of the twenty-eighth AAAI con-ference on artificial intelligence, pages 1515–1521.AAAI Press.

Rachel Ginn, Pranoti Pimpalkhute, Azadeh Nikfarjam,Apurv Patki, Karen OConnor, Abeed Sarker, KarenSmith, and Graciela Gonzalez. 2014. Mining twit-ter for adverse drug reaction mentions: a corpusand classification benchmark. In Proceedings ofthe fourth workshop on building and evaluating re-sources for health and biomedical text processing.

62

Pollyanna Goncalves, Matheus Araujo, Fabrıcio Ben-evenuto, and Meeyoung Cha. 2013a. Comparingand combining sentiment analysis methods. In Pro-ceedings of the first ACM conference on Online so-cial networks, pages 27–38. ACM.

Pollyanna Goncalves, Fabrıcio Benevenuto, and Meey-oung Cha. 2013b. Panas-t: A pychometric scalefor measuring sentiments on twitter. arXiv preprintarXiv:1308.1857.

Aniko Hannak, Eric Anderson, Lisa Feldman Barrett,Sune Lehmann, Alan Mislove, and Mirek Riede-wald. 2012. Tweetin’ in the rain: Exploringsocietal-scale effects of weather on mood. InICWSM. Citeseer.

J. Lardon, R. Abdellaoui, F. Bellet, H. Asfari, J. Souvi-gnet, N. Texier, M.C. Jaulent, M.N. Beyens, A. Bur-gun, and C. Bousquet. 2015. Adverse drug re-action identification and extraction in social media:A scoping review. Journal of Medical Internet Re-search, 17:171.

Marco Lui and Timothy Baldwin. 2012. langid.py: Anoff-the-shelf language identification tool. In Pro-ceedings of the ACL 2012 system demonstrations,pages 25–30. Association for Computational Lin-guistics.

Marco Lui and Timothy Baldwin. 2014. Accuratelanguage identification of twitter messages. In Pro-ceedings of the 5th Workshop on Language Analysisfor Social Media (LASM)@ EACL, pages 17–25.

Finn Arup Nielsen. 2011. A new anew: Evaluationof a word list for sentiment analysis in microblogs.arXiv preprint arXiv:1103.2903.

Filipe Nunes Ribeiro, Matheus Araujo, PollyannaGoncalves, Fabrıcio Benevenuto, and MarcosAndre Goncalves. 2010. A benchmark compari-son of state-of-the-practice sentiment analysis meth-ods. ACM Transactions on Embedded ComputingSystems, 9(4).

Nikolaos Pappas, Georgios Katsimpras, and EfstathiosStamatatos. 2013. Distinguishing the popularity be-tween topics: A system for up-to-date opinion re-trieval and mining in the web. In International Con-ference on Intelligent Text Processing and Computa-tional Linguistics, pages 197–209. Springer.

Filipe N Ribeiro, Matheus Araujo, PollyannaGoncalves, Marcos Andre Goncalves, and FabrıcioBenevenuto. 2016. Sentibench-a benchmark com-parison of state-of-the-practice sentiment analysismethods. EPJ Data Science, 5(1):1–29.

Abeed Sarker and Graciela Gonzalez. 2015. Portableautomatic text classification for adverse drug reac-tion detection via multi-corpus training. Journal ofbiomedical informatics, 53:196–207.

WHO. No date. World Health Organiza-tion English Glossary. Online document,http://www.who.int/medicines/areas/coordination/English Glossary.pdf. Accessed: 27 July 2016.

63