scaling deployment at etsy
DESCRIPTION
How we keep deployments fast at Etsy presented at Scaleconf 2013TRANSCRIPT
![Page 2: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/2.jpg)
Thursday, April 18, 13
![Page 3: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/3.jpg)
the marketplace we make together
Thursday, April 18, 13
![Page 4: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/4.jpg)
February 2013
• 1.49 billion page views
• 4,215,169 items sold
• $94.7 million of goods sold
• 22+ million members
• 800,000+ active shopshttp://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
Thursday, April 18, 13
![Page 5: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/5.jpg)
LAMMP
Item by TheBackPackShoppe
Thursday, April 18, 13
![Page 6: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/6.jpg)
Item by FrankelPhotos
Monolithic App
Thursday, April 18, 13
![Page 7: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/7.jpg)
No Branching
Item by NurseryWallArt
Thursday, April 18, 13
![Page 8: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/8.jpg)
Deploy Frequency
Thursday, April 18, 13
![Page 9: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/9.jpg)
First Day
Item by flowersandfleurons
Thursday, April 18, 13
![Page 10: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/10.jpg)
Thursday, April 18, 13
![Page 11: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/11.jpg)
IRC
Thursday, April 18, 13
![Page 12: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/12.jpg)
Thursday, April 18, 13
![Page 13: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/13.jpg)
Developer VMs
• KVM
•Dev version of full Etsy stack
•Chef
•DevTools
Thursday, April 18, 13
![Page 14: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/14.jpg)
Thursday, April 18, 13
![Page 15: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/15.jpg)
Item by codecards
Thursday, April 18, 13
![Page 16: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/16.jpg)
% review -r dschauenberg
Thursday, April 18, 13
![Page 17: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/17.jpg)
automatically assignedautomatically assigned
Thursday, April 18, 13
![Page 18: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/18.jpg)
Try
Item by CSSDesign
Thursday, April 18, 13
![Page 19: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/19.jpg)
Actually ...
Thursday, April 18, 13
![Page 20: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/20.jpg)
CI - The Old Way
• ~10 buildtest servers, single SSD
• running multiple Jenkins executors
• tests pegging IO
• Jenkins is much better at running single executors
Thursday, April 18, 13
![Page 21: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/21.jpg)
The Bobs
Item by Signz
Thursday, April 18, 13
![Page 22: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/22.jpg)
The Bobs
• LXC containers on buildtests
•Multiple SSDs
• Labels for heavy/any execution
•One heavy executor per disk
Thursday, April 18, 13
![Page 23: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/23.jpg)
CI/Try
• ~200 Bobs
•Mostly for try
•Constant monitoring for slow tests
Thursday, April 18, 13
![Page 24: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/24.jpg)
push train
Item by decomodwalls
Thursday, April 18, 13
![Page 25: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/25.jpg)
#push
• IRC channel to organize push trains
• Join a train if you want to deploy changes
• Schedule is planned via the channel topic
• First in the train is the driver (controls the deploy)
•Opening hours: 7am - 10pm NYC time
Thursday, April 18, 13
![Page 26: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/26.jpg)
#push
<prod> kseever* + jameslee | jpaul | DanielConvissor (c)
Thursday, April 18, 13
![Page 27: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/27.jpg)
#push
bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c)
Thursday, April 18, 13
![Page 28: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/28.jpg)
pushbot
• .join
• .in
• .good
• .done
Thursday, April 18, 13
![Page 29: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/29.jpg)
pushbot
Thursday, April 18, 13
![Page 30: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/30.jpg)
https://github.com/etsy/PushBot
Thursday, April 18, 13
![Page 31: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/31.jpg)
Item by EsalonPhotography
Thursday, April 18, 13
![Page 32: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/32.jpg)
Deployinator
Thursday, April 18, 13
![Page 33: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/33.jpg)
Thursday, April 18, 13
![Page 34: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/34.jpg)
Thursday, April 18, 13
![Page 35: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/35.jpg)
stale commits
Thursday, April 18, 13
![Page 36: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/36.jpg)
version checks
buttons disabledbuttons disabled
Thursday, April 18, 13
![Page 37: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/37.jpg)
version checks
Thursday, April 18, 13
![Page 38: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/38.jpg)
lock down deploys
Thursday, April 18, 13
![Page 39: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/39.jpg)
https://github.com/etsy/deployinator
Thursday, April 18, 13
![Page 40: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/40.jpg)
Dashboards
deploy linesdeploy lines
Thursday, April 18, 13
![Page 41: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/41.jpg)
https://github.com/etsy/dashboard
Thursday, April 18, 13
![Page 42: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/42.jpg)
Supergrep
Thursday, April 18, 13
![Page 43: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/43.jpg)
https://github.com/etsy/supergrep
Thursday, April 18, 13
![Page 44: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/44.jpg)
Thursday, April 18, 13
![Page 45: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/45.jpg)
Summary
•Current setup has scaled to ~150 people
•Constantly trying to improve the speed of deployment
• Find weak parts in the process and make them more robust/faster
• Bring Dev closer to Prod
•Not being able to deploy has the same status as the site being down
Thursday, April 18, 13
![Page 46: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/46.jpg)
codeascraft.etsy.comwww.etsy.com/codeascraft/talks
etsy.github.comwww.etsy.com/careers
Thursday, April 18, 13
![Page 47: Scaling Deployment at Etsy](https://reader034.vdocuments.us/reader034/viewer/2022042814/555c3299d8b42a09438b543e/html5/thumbnails/47.jpg)
Questions?
Thursday, April 18, 13