Analyzing Analysts

Dustin McIntosh

Mode Analytics

Understanding the Queriers Behind the Queries

Goal: Understand user base and how they interact with the product

NovicesPower Users

● Who are the users?

● Which SQL errors do they make?

● How can we make users' experience better?

● Why do users churn?

● Can we classify users' ability level based on their queries?

Goal: Understand user base and how they interact with the product

● Who are the users?

● Which SQL errors do they make?

● How can we make users' experience better?

● Why do users churn?

● Can we classify users' ability level based on their queries?

NovicesPower Users

Time

Churn – Are users getting frustrated?

Churn – Are users getting frustrated?

Churn – Are users getting frustrated?

No!

Churning Users = Tutorial Users

Recommendation: prompt tutorial users to connect their own data at tutorial conclusion

Classifying Users Based on the Content of their Queries

Bag of SQL

keywords

Formatting based:

– White space

– Parentheses

Keyword diversity

Is it an Error?

Error type

Random Forest

Classifying Users Based on the Content of their Queries

Bag of SQL

keywords

Formatting based:

– White space

– Parentheses

Keyword diversity

Is it an Error?

Error type

Random Forest

AUC = 0.66

The differences between experts and novices

Deliverable: model that classifies users' ability based on queries

Dustin McIntosh

EFavDB.com

Extra Slides

What errors do they make?

Recommendation: prominently display or auto-correct table/column names

Top-heavy nature of SQL

Non-tutorial users also not getting frustrated

All churned users Churned users that have connected their own data

Correlations of the Features

Compare Using top 5 features

All features

Top 5 features

Histograms of important variables

novicesexperts