groupon_controlled experimentation_panel_the hive
TRANSCRIPT
!Rajesh Parekh!
Controlled Experimentation to Guide Product Innovation!
Controlled Experimentation (A/B Testing)!• Method to study effects of a treatment#
• Concept:!- Randomly split users into two groups#
� A : Control#
� B: Treatment#- A and B are identical to each other except
for the treatment being evaluated#- Collect performance metrics from the
experiment#- Run statistical tests to determine if
differences between A and B are purely by chance#
2
Randomly)Divide)
A)(Control)) B)(Treatment))
Measure)&)Evaluate)
Controlled)Experimenta=on)Panel)
Why Run Controlled Experiments?!• Commonly used approach in clinical trials!- What is the effect of a particular drug / treatment?#
• Systematically validate hypotheses with data!!• Concurrently run the treatment and control!- The difference (if any) is#� Because of the treatment OR#� Due to random chance#
• Determine if a treatment is causal in nature!- E.g., Making the search box bigger causes increase in queries / user#
3 Controlled)Experimenta=on)Panel)
Controlled Experimentation: Use Cases! #
4 Controlled)Experimenta=on)Panel)
A"B"Stract"Widget"Company"
_________________)_________________)
_________________)_________________) BUY)NOW)
A"B"Stract"Widget"Company"
_________________)_________________)
_________________)_________________) BUY"NOW"
Website)Variants)
Controlled Experimentation: Use Cases! #
5 Controlled)Experimenta=on)Panel)
Free)Trial) Play)Now)
Mobile)Call)to)Ac=on)
Controlled Experimentation: Use Cases! #
6 Controlled)Experimenta=on)Panel)
Top)deal)highlighted)
Email)Template)Design)
Controlled Experimentation: Use Cases! #
7 Controlled)Experimenta=on)Panel)
Backend)changes)(e.g.,)Personaliza=on)Algorithm))
8
Controlled Experimentation: Use Cases!
# • Follow-up message for users who previously clicked on an ad#
• Incentive campaign to re-engage lapsed users#
• Think of this as placing filters / guards on a randomly chosen user population#
Controlled)Experimenta=on)Panel)
Custom)Defined)User)Segments)
9
Key Components of an Experimentation Platform!Hashing function!!
!
!
!
!
Logging!!
!
!
!
Metrics – suite of KPI!!
!
!
!
!
Dashboard!
F())))))))))))))
Group)0)
Group)1)
Group)NU1)
Time"Spent"
Revenue"
Click>Through"Rate"
Session"Length"
Abandonment"
Purchase"Rate"
• Metric)improvements)and)Sta=s=cal)Significance)in)a)central)place)
• Detailed)logging)of)all)user)interac=ons)
Controlled)Experimenta=on)Panel)
Ensure Identical Control and Treatment!
• Custom Segments#
10
CONTROL" TREAMENT"
Gender""
Male)
Female)
CONROL" TREATMENT"
Region"Size"
Small)
Medium)
Large)
CONROL" TREATMENT"
Prior"Exposure"
No)
Yes)
δ%"
Controlled)Experimenta=on)Panel)
• Frequency Distribution#
• Large Difference in Prior Exposure Rate violates assumptions#
A/A Tests!• Run an experiment with two identical variants#
• Helps to determine if:#- Users are being split uniformly at random#- Correct data is being logged#- Variance between identical populations of users is acceptable#
• Challenge:!- Few purchases of high value deals render statistically significant
difference between treatment and control#
11
SPAIN"TRIP"$1,999"
Controlled)Experimenta=on)Panel)
Monitor Each Variant!• Place yourself in each variant
to validate the experience#!
!
12
Carefully)inspect)each)variant)
Controlled)Experimenta=on)Panel)
• Wrong sort order!!
!
Objective Function!#
#
Conversion" Revenue"
P(conversion)))
• Favors)lower)price)deals)
E(rev))=)P(conversion))*)price))
• More)expensive)deals)can)dominate)
Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves))
13 Controlled)Experimenta=on)Panel)
Measure Overall Impact!• Test focuses on#- A particular area of
the website#- A sub-population of
users#
• Measure!- Improvement on the
sub-segment AND#- Entire population!#
14
Measure)overall)impact)to)guard)against)cannibaliza=on)
Controlled)Experimenta=on)Panel)
Panel Discussion: Questions!
15 Controlled)Experimenta=on)Panel)
Thanks to many talented individuals at Groupon I am privileged to work with!#• Data Science#• Engineering#• Marketing / Market Research#
Acknowledgements#
16 Controlled)Experimenta=on)Panel)
