why sentiment analysis is a market for lemons … and how to fix it
TRANSCRIPT
Language Intelligence
Why Sentiment Analysis is a Market for Lemons … and How to Fix it
Robert Munro
With thanks!
Gary King & Jana Thompson:
<- other Idibon people here:Michelle Casbon & Nick Gaylord
What is a market for lemons?
• Information asymmetry between buyers and sellers, leaving only "lemons" behind. George Akerlof • Buyers cannot distinguish good
from bad products• Prices are equally low for all
products• The buyer's price adverse
selection problem drives the high-quality products from the market
Competition is not increasing accuracy• 100+ companies
offering some form of sentiment analysis• Accuracy hovering
around 70% for real-world applications for almost a decade
The most honest sentiment analysis results you will see
Accuracy
F-Score Recall Precision F-Score
PositiveNegativ
e NeutralPositiv
eNegativ
e NeutralPositiv
eNegati
ve NeutralSemantria 0.59 0.59 0.56 0.47 0.78 0.68 0.80 0.45 0.62 0.59 0.57MonkeyLearn 0.50 0.38* 0.84 0.54 0.00 0.45 0.60 0.00 0.59 0.57 0.00MetaMind 0.66 0.66 0.68 0.46 0.88 0.78 0.88 0.50 0.73 0.60 0.64Idibon Public 0.68 0.67 0.76 0.75 0.49 0.66 0.69 0.72 0.71 0.72 0.58
• Even within the best results for one domain, there is no clear leader when broken down by category• All systems could have best results in other domains• All could adapt here: Monkey Learn had errors with the ‘Neutral’
category, but we are sure they could update their models
Source: Sentiment 140 corpus, 3-way sentiment on social data:http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip
Data beats algorithms; feedback beats data
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.457 0.473
0.615
0.948precisionrecallF-value
Distinguishing the correct ‘Ford’
Distinguishing “Ford” the company from people called “Ford”
Consumers are uncertain• When consumers try out-
of-domain analysis, they lose confidence from the poor results.• Domain-dependence
means that even bad models will be accurate in some areas• Consumers can only
evaluate anecdotally or by precision, not recall • Uncertainty prevails
Market forces are not breeding innovation• Can’t innovate
through code alone• More training data! • But low price-points
means low margins • Lack of capital to
find & label enough training data
The Solution
• A different economic models for useful sentiment analysis: • Data-sharing for more
accurate training data • Protecting sensitive data
from public release
Machine learning
Optimization
Human annotation
Cloudprediction
engine
Actionable intelligence
On-site prediction
engine
Copy & Sync Models
App Requests
Ambiguous, Novel & Interesting Items
Internal Data Flow
Hybrid Model Data Flow
Application Data Flow
firewall
The Benefits• Multiple organizations can share in the benefits of better
sentiment analysis, without sacrificing privacy• Single point of human-contact: no expensive duplicate
manual labeling of data• Keeps lemons out of the market
Idibon Public: our implementation
• Free product, offered in addition to our enterprise Idibon Studio and Idibon Terminal solutions
Applies to NLP and Machine Learning more broadly
Every human communication
• Any task can be bundled this way• Allows margins for use cases that
were not otherwise viable• … including the full diversity of
languages, priced out when everyone started in English
Language Intelligence
Why Sentiment Analysis is a Market for Lemons … and How to Fix it
QUESTIONS?Robert Munro