using sentiment analysis to fill in the gaps in user surveys€¦ · build your own (can use...
TRANSCRIPT
Using Sentiment Analysis to Fill in the Gaps in User
SurveysThat’s a really long title
YOW! Data 2017My lack of expertise
Confusing setup
Flaunted my poor understanding of underlying tech
Audience understood less than when they went in
Dance number
TodayLower expectations
Confusing setup
Poor explanation of the technology
Ambiguous results
Silent weeping
Confusing Setup
SuperannuationWe all love superannuation
Can we measure just how much we love it?
Two SourcesPhone calls with consultants Web chat logs
Phone callsSpeech-to-text
Poor quality, single-channel
Comical results
Much funnier presentation
Transcript
“hope hi there John my name is Spider-Man advise of the strange episode advice can you please get some of that your separation under some fantastic now we all in 20 minutes according to that and so that I can include details Yu-Gi-Oh counterpart of a conversation”
Web chat logsAgent: Hello, I am totally a human being. My name is Agent.
Customer: Yes, indeed. May I just say how much I love Super?
Agent: Me too, after all, it is called “super” for a reason.
Customer: How can I get more Super into me?
Agent: Would you be interested in our new “Super Super”?
Customer: Here is all my money, it is yours.
Web chat logs13 months JSON data
100k conversations
1 million lines of dialogue
Metadata
Web chat logsMeaningful Connection Score
Agent-supplied rating
-100 to +100
Almost every chat has one
Exit Survey
User-supplied rating
10% get asked
10% of those will do it
~1000 conversations have one
0 to 10
Web chat logsNobody wants to fill in a survey
Can we use Sentiment Analysis as an indicator?
Poor Technical Explanation
Sentiment AnalysisClassification problem
First ProblemHumans only agree on the sentiment of text about 80% of the time
ComplicatedI like cheese.
I do not like cheese.
I like cheese, but I’d rather eat biscuits.
I used to like cheese.
SarcasmWell, that’s great.
Vs.
Well, that’s great.
ModelsLexical - word frequencies
Supervised - Naive Bayes, SVM
Unsupervised - Latent Dirichlet Allocation, word2vec
Neural Networks
LevelsDocument
Paragraph
Sentence
Entity
Google Natural Language APIParser + NN
Sentence and entity level
Sentiment score and magnitude
Cheap compared to writing your own
Other services are available
Build your own (can use SyntaxNet if you want)
Ambiguous Results
MCS vs Sentiment
Exit Scores vs Sentiment
Entity Sentiment
Silent Weeping
Please stop nowSentiment analysis is a reasonable indicator of customer satisfaction
Sentiment analysis is slightly more accurate than astrology
Probably best to train your own model for entity sentiment
Refunds available from the organisers
Questions?Please don’t.
Nobody wants this to go on any longer than it has to.
@nomiddlename