can i hear you sentiment analysis on medical forums
TRANSCRIPT
Can I hear you? Sentiment Analysis on Medical Forums
TANVEER ALI DAVID SCHRAMM MARINA SOKOLOVA DIANA INKPEN
UNIVERSITY OF OTTAWA &
CHILDREN'S HOSPITAL OF EASTERN ONTARIO, OTTAWA, CANADA
Extract opinions from medical forums about hearing problems. Classify the opinionated messages into positive, negative and neutral.
Introduction
Neutral/UnknownNow, you'll hear some people saying that compression is bad, and linearity is good, especially for music.
NegativeSomeone with 50 DB hearing aid gain with a total loss of 70 DB may not know that the place is producing 107 DB since it may not appear too loud to him since he only perceives 47 DB.
PositiveThis has the beneficial effect of making the quieter sounds audible but not blowing your head off with the louder sounds.
Task
Factual Statements
Opinion Statements
Dataset
We collected 893 individual posts from 34 threads from three different threads of Hearing loss problems.
We filtered the initial 34 threads by removing the threads in which people did not discuss Hearing Aids. The final data contained 607 posts in 26 threads.
Manual AnnotationA total of 3515 sentences from 26 threads were manually annotated by two annotators.
Threads Posts Average posts per person
www.hearingaidforums.com 7 185 2.9
www.medhelp.org 9 105 2.77
www.alldeaf.com 10 317 1.93
Total 26 607 2.53
Filtered dataset collection statistics
Annotator 2
Annotator 1
Pos Neg Neut No Label Total
Pos 226 329
Neg 214 296
Neutral 117 162
No Label 2720 2728
Total 230 218 128 2939 3515
Annotations statistics of Sentences between the two annotators
Data Annotation
The overall percentage agreement between the annotators for the balanced dataset was 82.01% and kappa
was 0.65.
Annotator 1 and Annotator 2 did not label a large number of sentences, i.e., 2939 and 2728, respectively;
therefore these sentences are removed from the dataset.
We consider only those sentences labeled as positive and negative. Since the data is already balanced, no
balancing is performed.
We used the subjective lexicon built by Wilson, Wiebe, and Hoffman (2005) as a resource to extract features for supervised learning classification.
Subjectivity Lexicon
Positive Negative Neutral Both Total PercentAdjective 1171 1838 235 5 3249 39.52Noun 677 1346 144 3 2170 26.40Verb 380 869 68 8 1325 16.12anypos 362 676 104 5 1147 13.95Adverb 128 183 19 0 330 4.01Total 2718 4912 570 21 8221 100Percent 33.06 59.75 6.93 0.26 100
Distribution of prior polarities in the Subjectivity Lexicon Strong Subj Weak Subj Total Percent
Adjective 2006 (61.74%) 1243 (38.25%) 3249 39.52
Noun 1440 (25.85%) 730 (33.6%) 2170 26.40
Verb 861 (15.46%) 464 (35.01%) 1325 16.12
Anypos 1043 (18.72%) 104 (9.06%) 1147 13.95
Adverb 219 (3.93%) 111 (33.6%) 330 4.01
Total 5569 2652 8221 100
Percent 67.74 32.26 100
Distribution of subjectivity clue in the Subjectivity Lexicon
Example: type=weaksubj len=1 word=ability pos=noun stemmed=n prior_polarity=positive
Sentence splitting, tokenization, part of speech tagging, lemmatization for nouns and verbs.
Computed the features and then using the classifiers to classify into positive, negative, and neutral sentences.- Lexicon matching features simply count how many words in a sentence match in the lexicon.- All features were on individual tokens, except pronoun phrases which were calculated using different patterns, e.g., I wouldassume, I feel as, I could sympathize.
STRONGSUBJ # of words found as strong subjective in current sentence
WEAKSUBJ # of words found as weak subjective in current sentence
ADJECTIVE # of adjectivesADVERBS # of adverbsPRONOUN # of pronounsPOSITIVE # of words found having prior polarity as positiveNEGATIVE # of words found having prior polarity as negative
polarity as negativeNEUTRAL # of words found having prior polarity as neutralPRP_PHRASE # of phrases containing pronouns found in current
sentence
Final features considered for the experiments
Methodology
The most common features were pronouns, followed by weak subjective clues, adjectives, and adverbs.
Comparison of performance between different features among three classifiers (no lemmatization)
No lemmatization
Naïve Bayes SVM Logistics-R
P R F-1 P Re F-1 P R F-1
Polarity features only 0.656 0.65 0.644 0.661 0.641 0.625 0.649 0.645 0.641
All features 0.595 0.584 0.565 0.641 0.618 0.596 0.657 0.657 0.656
Baseline 0.540 0.541 0.539 0.586 0.586 0.586 0.585 0.584 0.584
Including lemmatization
Naïve Bayes SVM Logistic-R
P R F-1 P Re F-1 P R F-1
Polarity features only 0.644 0.625 0.607 0.636 0.607 0.578 0.688 0.686 0.685
All features 0.589 0.580 0.560 0.627 0.600 0.570 0.671 0.670 0.670
Baseline 0.540 0.541 0.539 0.586 0.586 0.586 0.585 0.584 0.584
Comparison of performance between different features among three classifiers (after lemmatization)
Experiments For the baseline, the feature vector of bag of words is used. We have not considered the unique words for the bag of words.
We used 10-fold cross validation with three classifiers i.e. Naïve Bayes, Support Vector Machine and Logistics Regression.
More words matched with the lexicon’s positive words than the lexicon’s negative words. This led to classifier’s performance to
become slightly better for positive in the experiments.
Analysis
The bag-of-word representation (BOW) is a high baseline that is hard to beat in many textsclassification problems.
The Subjectivity Lexicon clues for polarity such as positive, negative and neutral have shownsignificant improvement with 4.2% on average among the three classifiers.
The combination of Subjectivity Lexicon clues with other features, i.e., the number of pronoun phrases,number of adjectives, etc., decreased the classification performance.
The classification for semantic orientation depends heavily on the quality of the lexicon used, ratherthan the size of the lexicon, as the classification of the sentences into positive and negative reached 70%by using only the polarity clues for individual words within the lexicon.
Conclusion and Future Work
We used several lexicon-based features together with rule-based features like pronoun phrases forour classification of the medical forums dataset for detecting semantic orientation.
The dataset of 3515 sentences from 26 threads were manually annotated by two annotators andachieved 82.01% overall agreement with kappa 0.65.
Evaluations have been made for the classification of the data using three different supervised learning-based classifiers.
It is shown that our proposed features outperformed the baseline of bag-of-word features by10.5% with Naïve Bayes, 3.9% with SVM and 5.7% with logistic-R.
In future work, some techniques and features such as intensification and negation presented in(Taboada et al., 2011), (Kennedy and Inkpen, 2006) can be used for the semantic orientation orpolarity detection of the sentences.
References
Yi, J., Nasukawa, T., Bunescu, R., &Niblack, W. (2003, November). Sentiment analyzer: Extractingsentiments about a given topic using natural language processing techniques. In Data Mining,2003.ICDM 2003. Third IEEE International Conference on (pp. 427-434).
Sokolova, M., &Bobicev, V. (2011). Sentiments and Opinions in Health-related Web messages.InRecent Advances in Natural Language Processing (pp. 132-139).
Kennedy, A., & Inkpen, D. (2006).Sentiment classification of movie reviews using contextual valenceshifters. Computational Intelligence, 22(2), 110-125.
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., &Stede, M. (2011).Lexicon-based methods for sentimentanalysis. Computational linguistics, 37(2), 267-307.
