using sentiment analysis to fill in the gaps in user surveys€¦ · build your own (can use...

Using Sentiment Analysis to Fill in the Gaps in User

SurveysThat’s a really long title

YOW! Data 2017My lack of expertise

Confusing setup

Flaunted my poor understanding of underlying tech

Audience understood less than when they went in

Dance number

TodayLower expectations

Confusing setup

Poor explanation of the technology

Ambiguous results

Silent weeping

Confusing Setup

SuperannuationWe all love superannuation

Can we measure just how much we love it?

Two SourcesPhone calls with consultants Web chat logs

Phone callsSpeech-to-text

Poor quality, single-channel

Comical results

Much funnier presentation

Transcript

“hope hi there John my name is Spider-Man advise of the strange episode advice can you please get some of that your separation under some fantastic now we all in 20 minutes according to that and so that I can include details Yu-Gi-Oh counterpart of a conversation”

Web chat logsAgent: Hello, I am totally a human being. My name is Agent.

Customer: Yes, indeed. May I just say how much I love Super?

Agent: Me too, after all, it is called “super” for a reason.

Customer: How can I get more Super into me?

Agent: Would you be interested in our new “Super Super”?

Customer: Here is all my money, it is yours.

Web chat logs13 months JSON data

100k conversations

1 million lines of dialogue

Metadata

Web chat logsMeaningful Connection Score

Agent-supplied rating

-100 to +100

Almost every chat has one

Exit Survey

User-supplied rating

10% get asked

10% of those will do it

~1000 conversations have one

0 to 10

Web chat logsNobody wants to fill in a survey

Can we use Sentiment Analysis as an indicator?

Poor Technical Explanation

Sentiment AnalysisClassification problem

First ProblemHumans only agree on the sentiment of text about 80% of the time

ComplicatedI like cheese.

I do not like cheese.

I like cheese, but I’d rather eat biscuits.

I used to like cheese.

SarcasmWell, that’s great.

Vs.

Well, that’s great.

ModelsLexical - word frequencies

Supervised - Naive Bayes, SVM

Unsupervised - Latent Dirichlet Allocation, word2vec

Neural Networks

LevelsDocument

Paragraph

Sentence

Entity

Google Natural Language APIParser + NN

Sentence and entity level

Sentiment score and magnitude

Cheap compared to writing your own

Other services are available

Build your own (can use SyntaxNet if you want)

Ambiguous Results

MCS vs Sentiment

Exit Scores vs Sentiment

Entity Sentiment

Silent Weeping

Please stop nowSentiment analysis is a reasonable indicator of customer satisfaction

Sentiment analysis is slightly more accurate than astrology

Probably best to train your own model for entity sentiment

Refunds available from the organisers

Questions?Please don’t.

Nobody wants this to go on any longer than it has to.

@nomiddlename