![Page 1: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/1.jpg)
Continuous Experimentationwith Continuous
Deployment
Steve MardenfeldEtsy
Monday, November 5, 2012
![Page 2: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/2.jpg)
Monday, November 5, 2012
![Page 3: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/3.jpg)
World’s Handmade Market
Monday, November 5, 2012
![Page 4: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/4.jpg)
Continuous Deployment
Monday, November 5, 2012
![Page 5: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/5.jpg)
Monday, November 5, 2012
![Page 6: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/6.jpg)
Monday, November 5, 2012
![Page 7: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/7.jpg)
Metrics
Monday, November 5, 2012
![Page 8: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/8.jpg)
Graph Factory
Monday, November 5, 2012
![Page 9: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/9.jpg)
Oops
Monday, November 5, 2012
![Page 10: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/10.jpg)
Oops
Monday, November 5, 2012
![Page 11: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/11.jpg)
Good Enough?
Monday, November 5, 2012
![Page 12: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/12.jpg)
What is Better?
Monday, November 5, 2012
![Page 13: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/13.jpg)
Sometimes it’s Obvious
Monday, November 5, 2012
![Page 14: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/14.jpg)
Most Times it’s Not
Monday, November 5, 2012
![Page 15: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/15.jpg)
Most Times it’s Not
Cart Page Restyled
Monday, November 5, 2012
![Page 16: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/16.jpg)
Most Times it’s Not
Etsy Mentioned on Oprah
Internet Goes Down in Brooklyn
Justin Beiber Sneezes
New Cat Meme
Cart Page Restyled
Monday, November 5, 2012
![Page 17: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/17.jpg)
A/B Testing?Monday, November 5, 2012
![Page 18: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/18.jpg)
Monday, November 5, 2012
![Page 19: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/19.jpg)
Monday, November 5, 2012
![Page 20: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/20.jpg)
Test Them Both
Monday, November 5, 2012
![Page 21: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/21.jpg)
Statistics?
Monday, November 5, 2012
![Page 22: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/22.jpg)
First Pass
HADOOP
Monday, November 5, 2012
![Page 23: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/23.jpg)
First Pass
CASCADING
Monday, November 5, 2012
![Page 24: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/24.jpg)
First Pass
R
Monday, November 5, 2012
![Page 25: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/25.jpg)
First Pass
RHADOOP
CASCADING
Monday, November 5, 2012
![Page 26: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/26.jpg)
Automate It
A/B Analyzer
Monday, November 5, 2012
![Page 27: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/27.jpg)
A/B Analyzer
•Framework for looking at experiments
•Done for every experiment beforehand
Monday, November 5, 2012
![Page 28: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/28.jpg)
A/B Analyzer
Monday, November 5, 2012
![Page 29: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/29.jpg)
A/B Analyzer
Monday, November 5, 2012
![Page 30: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/30.jpg)
Monday, November 5, 2012
![Page 31: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/31.jpg)
3 Types of Metrics
•Co-occurence
•visits that did X (searched)
•average number of X (searches)
•Funnels
•Special Cases*
Monday, November 5, 2012
![Page 32: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/32.jpg)
Co-Occurence
•Co-occurence of all actions with the variant
•i.e. - how many people saw this variant and then purchased
•i.e. - average number of purchases per variant
Monday, November 5, 2012
![Page 33: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/33.jpg)
Funnels
BuySearch Listing Buy
Monday, November 5, 2012
![Page 34: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/34.jpg)
Funnels
Buy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
![Page 35: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/35.jpg)
Funnels
BuyBuy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
![Page 36: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/36.jpg)
Special Cases
•Exiting pages
•Bounce rate
•Visit length
Monday, November 5, 2012
![Page 37: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/37.jpg)
Pipeline
HDFS Events Visits MySql
A/B
PROCESSING
Client
Logs
DB
COLLECTION
Monday, November 5, 2012
![Page 38: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/38.jpg)
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
![Page 39: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/39.jpg)
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
![Page 40: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/40.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 41: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/41.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 42: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/42.jpg)
Idea
Monday, November 5, 2012
![Page 43: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/43.jpg)
Changing Search
Monday, November 5, 2012
![Page 44: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/44.jpg)
Related Searches?
Monday, November 5, 2012
![Page 45: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/45.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 46: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/46.jpg)
Be Specific
•What does an improvement look like?
•More clicks?
•Less searches?
•Shorter visits?
•Longer visits?
Monday, November 5, 2012
![Page 47: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/47.jpg)
Bet Driven Development
Monday, November 5, 2012
![Page 48: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/48.jpg)
Is this even worth it?
Monday, November 5, 2012
![Page 49: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/49.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 50: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/50.jpg)
Code It Up
Monday, November 5, 2012
![Page 51: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/51.jpg)
Code
Monday, November 5, 2012
![Page 52: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/52.jpg)
Turn it On
Monday, November 5, 2012
![Page 53: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/53.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 54: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/54.jpg)
Realtime?
Monday, November 5, 2012
![Page 55: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/55.jpg)
Wait
Monday, November 5, 2012
![Page 56: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/56.jpg)
Process
•Idea
•Hypothesis
•Code
•Wait
•Analysis
Monday, November 5, 2012
![Page 57: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/57.jpg)
Making Sense of ResultsMonday, November 5, 2012
![Page 58: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/58.jpg)
Back To Related Searches
Monday, November 5, 2012
![Page 59: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/59.jpg)
Back To Related Searches
Monday, November 5, 2012
![Page 60: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/60.jpg)
Monday, November 5, 2012
![Page 61: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/61.jpg)
Monday, November 5, 2012
![Page 62: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/62.jpg)
Final Thoughts
•Bet driven development - be explicit about your changes
•Easy experimental setup and quick analysis makes for more experiments
•An experiment is successful if you learned something, not if you launched
Monday, November 5, 2012
![Page 63: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/63.jpg)
Monday, November 5, 2012
![Page 64: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.us/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/64.jpg)
http://www.slideshare.net/powerlexis/retro-slides-from-ibm
Monday, November 5, 2012