why everything is an a/b test at pinterest
TRANSCRIPT
Krishna Gade
Why everything is an A/B test at Pinterest
Confidential
2
What is Pinterest?
A visual bookmarking tool
Discover an inspiring ideaSave it to a board
Go do it
Pinterest is a data product
Data Flow
App
events
Kafka
SecorSkyline
Pinball
Redshift
Pinalytics
Features
Qubole (Hadoop)
Singer
Confidential
Metrics
7
Confidential
User Lifecycle
New Users
Active Users
Dormant Users
Acquisition
Activation
Churn
Churn Resurrection
Confidential
Terminology
● MAU – Monthly Active User● WAU – Weekly Active User● DAU – Daily Active User● WAR – Weekly Active Repinner● WAC – Weekly Active Clickthrougher● WARC – Weekly Active Repinner or Clickthrougher● 1d7 – User who comes back at-least once in 7 days after
signup● 1rc7 - User who repins/clicks at-least once in 7 days
after signup
Confidential
MAUs
● Current progress to quarterly goal● MAU forecast● MAUs by app/gender/country● MAU Accounting
Confidential
MAU Accounting
Confidential
Acquisition Metrics
● Total Signups● Signups by App● Signups by Referrer● Invites Sent● Unique Invite Senders● Invite Signups
Confidential
Activation Metrics
Activation Rate = 1rc7/signups
● Activation Rate by App/Gender/Referrer● Overall signups to 1rc7● Signup engagement funnel● Signups to WAU 35 days after signup
Confidential
Engagement Metrics● *AU ratios
● # of Emails sent● # of Emails opened● # of Emails clicked● # of Notifications sent● # of Notifications clicked
Confidential
Resurrection Metrics
● # of Users who weredormant for 28+ daysbut came back, split byplatform, referrer etc
Confidential
Takeaways
● Emphasize on activation (process of converting a new user to MAU)
● Activation critical to long-term growth● Deeply understand how different
segments of users interact
Confidential
Automated Metrics
17
Confidential
Confidential
Chart creation
Confidential
Reuse existing query result
Confidential
Metrics creation
Confidential
A/B Experimentation
22
Confidential
Categories of Experiments
● A/B testing new features● Copy testing● SEO changes● Infrastructure updates
Confidential
A/B Testing Framework
Confidential
Confidential
Confidential
Confidential
A/B Testing Dashboard
Confidential
Confidential
p-value (via chi-squared test)
Statistical tests for validation
Confidential
Impact of Badging
Confidential
Impact of Badging
● 7% lift in DAU● 2.5% lift over long term● Effective at increasingengagement levels
Confidential
Copy Testing
Definition: Testing different variations of text we display to the user.
● A/B Testing copy was a tedious process● Even when we did this we were optimizing
only for English
Confidential
New Approach
● Ability to run a copy test w/o changing a line of code.
● Support for translating text dynamically● Use Multi-armed bandits to converge to showing
best performing text
Confidential
Multi Armed Bandits
Confidential
SEO
36
50+ Billion Pinscategorized by people
into more than
1 Billion Boards
Confidential
37
What makes SEO so challenging?
What is the Impact of the new SEO feature?
Will this new change affect SEO?
Uhhh...Dunno...
Uhhh...Ask Google.
Confidential
KEEPCALM
and
RUN ANSEO EXPERIMENT
Confidential
What is an SEO experiment?
39
Regular AB Experiment SEO Experiment
Metrics User behavior Search engine referred traffic
Unique id for hashing userid / sessionid page url
Confidential
What is an SEO experiment?
Launch
● Measure the gap ● Before vs After launch
Confidential
Dashboard
41
DIFF = enabled - control
Traffic Gain
Confidential
42
Decision Making 1 2
3
4
SHIP!
SHIP!
SHUT DOWN!VALIDATE
Confidential
43
Experiment Driven SEO
What is the Impact of the new SEO feature?
We got XX% of traffic growth from it! Can we use more resources to make bigger impact?
No, the experiment result is flat, so let’s ship it.
SEO experiment
SEO experiment
Is my UI change affecting SEO negatively?
Confidential
Why you should be A/B testing even your infrastructure
Confidential
Infrastructure Changes
● 2013: A new web framework● 2014: Pyapns● 2015: User service
Confidential
Experiment results for replacing PyAPNs
Confidential
Conclusions
● Figure out what success means● Define your metrics● Automate metric computation● A/B test everything!
Confidential
Thank You!