experiences from devops production: deployment, performance, failure
DESCRIPTION
In his All Your Base talk, David Mytton (founder of Server Density) will talk you through our experiences in handling large scale MongoDB deployments.TRANSCRIPT
![Page 1: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/1.jpg)
Experiences from productionDeployment, performance, failure
David MyttonAll Your Base - Oct 2014
blog.serverdensity.com
![Page 2: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/2.jpg)
David Mytton
![Page 4: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/4.jpg)
Slides: twitter.com/davidmytton
![Page 5: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/5.jpg)
Agenda
● Performance
● Architecture
● Downtime
● Preparation
● Where to host?
![Page 6: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/6.jpg)
Server Density Architecture
![Page 7: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/7.jpg)
Server Density Architecture
● ~100 servers - Ubuntu 12.04
![Page 8: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/8.jpg)
Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
![Page 9: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/9.jpg)
Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data
![Page 10: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/10.jpg)
Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data
● Nginx, Python, MongoDB
![Page 11: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/11.jpg)
Server Density Architecture
● ~100 servers - Ubuntu 12.04
● 50:50 virtual/dedicated
● 200TB/m processed data
● Nginx, Python, MongoDB
● Softlayer > 1TB RAM, 5TB SSDs
![Page 12: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/12.jpg)
Two choices for deployment
![Page 13: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/13.jpg)
Two choices for deployment
● Virtualized
● Bare metal
![Page 14: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/14.jpg)
Advantages of virtualization
● Easy to manage
![Page 15: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/15.jpg)
Advantages of virtualization
● Easy to manage
● Fast boot
![Page 16: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/16.jpg)
Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate
![Page 17: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/17.jpg)
Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate
● Templating/snapshots
![Page 18: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/18.jpg)
Advantages of virtualization
● Easy to manage
● Fast boot
● Easier to resize/migrate
● Templating/snapshots
● Containment
![Page 19: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/19.jpg)
Disadvantages of virtualization
● Another layer
![Page 20: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/20.jpg)
Disadvantages of virtualization
● Another layer
● Hypervisor overhead
![Page 21: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/21.jpg)
Disadvantages of virtualization
● Another layer
● Hypervisor overhead
● Host contention
![Page 22: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/22.jpg)
Disadvantages of virtualization
● Another layer
● Hypervisor overhead
● Host contention
● i/o performance
![Page 23: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/23.jpg)
Advantages of bare metal
● Dedicated resources
![Page 24: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/24.jpg)
Advantages of bare metal
● Dedicated resources
● Direct access to hardware
![Page 25: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/25.jpg)
Advantages of bare metal
● Dedicated resources
● Direct access to hardware
● Customisable specs
![Page 26: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/26.jpg)
Advantages of bare metal
● Dedicated resources
● Direct access to hardware
● Customisable specs
● Performance
![Page 27: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/27.jpg)
Disadvantages of bare metal
● Build/deploy time
![Page 28: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/28.jpg)
Disadvantages of bare metal
● Build/deploy time
● More difficult to resize
![Page 29: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/29.jpg)
Disadvantages of bare metal
● Build/deploy time
● More difficult to resize
● Difficult to migrate/snapshot
![Page 30: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/30.jpg)
Disadvantages of bare metal
● Build/deploy time
● More difficult to resize
● Capex/lifetime
● Difficult to migrate/snapshot
![Page 31: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/31.jpg)
Performance problems?
![Page 32: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/32.jpg)
Performance problems?
Easy answer: move to bare metal!
![Page 33: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/33.jpg)
Key performance factors
● Network
![Page 34: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/34.jpg)
Key performance factors
● Network
● EC2: Cluster compute, high memory, high i/o, high storage
● GCE: Higher CPU instances
![Page 35: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/35.jpg)
Key performance factors
● Network
![Page 36: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/36.jpg)
Key performance factors
● Network
Location Ping RTT LatencyWithin USA 40-80msTrans-Atlantic 100msTrans-Pacific 150msEurope-Japan 300ms
![Page 37: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/37.jpg)
Networking performance
AWS
GCE
bit.ly/googlevsamazon
![Page 38: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/38.jpg)
Key performance factors
● Memory
![Page 39: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/39.jpg)
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
![Page 40: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/40.jpg)
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
![Page 41: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/41.jpg)
Key performance factors
● Memory is expensive
![Page 42: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/42.jpg)
Key performance factors
● Disk
● SSDs!
![Page 43: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/43.jpg)
Key performance factors
● Disk
● SSDs!
GCE: 256GB = $83.20/m
EC2: 256GB = $35.32/m
SL: 200GB = $81/m
![Page 44: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/44.jpg)
Why cloud?
● Flexible
![Page 45: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/45.jpg)
Why cloud?
● Flexible
● Unlimited resources
![Page 46: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/46.jpg)
Why cloud?
● Flexible
● Unlimited resources
● Cheap to get started
![Page 47: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/47.jpg)
Why cloud?
● Flexible
● Unlimited resources
● Cheap to get started
● Other products
![Page 48: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/48.jpg)
Why colo?
![Page 49: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/49.jpg)
Why colo?
● Vastly cheaper
![Page 50: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/50.jpg)
Why colo?
● Vastly cheaper
● Complete control
![Page 51: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/51.jpg)
Let’s talk about downtime
![Page 52: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/52.jpg)
2013 Spend: ~$5bn
![Page 53: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/53.jpg)
2013 Spend: ~$6bn
![Page 54: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/54.jpg)
2013 Spend: ~$4bn
![Page 55: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/55.jpg)
You will have downtime
How much do you spend?
![Page 56: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/56.jpg)
Preparation
![Page 57: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/57.jpg)
Preparation - On Call
● Rotations
![Page 58: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/58.jpg)
Preparation - On Call
● Off call
● Rotations
![Page 59: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/59.jpg)
Preparation - On Call
● Off call
● Rotations
● Work the next day?
● Reachability - Train, 3G/4G (edge?!), Do Not Disturb mode, system updates
![Page 60: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/60.jpg)
Preparation - On Call
● Off call
● Rotations
● Work the next day?
● Reachability - Train, 3G/4G (edge?!), Do Not Disturb mode, system updates
● Work the next day?
![Page 61: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/61.jpg)
Preparation - Documentation
![Page 62: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/62.jpg)
Preparation - Documentation
● Searchable
![Page 63: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/63.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
![Page 64: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/64.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
![Page 65: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/65.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
● Up to date
![Page 66: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/66.jpg)
![Page 67: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/67.jpg)
Unexpected failures
![Page 68: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/68.jpg)
Unexpected failures
● Communication systems
![Page 69: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/69.jpg)
Unexpected failures
● Communication systems
● Network connectivity
![Page 70: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/70.jpg)
Unexpected failures
● Communication systems
● Network connectivity
● Access to support
![Page 71: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/71.jpg)
ALERT!
![Page 72: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/72.jpg)
ALERT!
1. Load up incident response checklist
![Page 73: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/73.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
![Page 74: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/74.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
![Page 75: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/75.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
4. Public status post
3. Log into Ops War Room
![Page 76: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/76.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
4. Public status post
5. Initial investigation
3. Log into Ops War Room
![Page 77: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/77.jpg)
Key response principles
![Page 78: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/78.jpg)
Key response principles
● Log everything
![Page 79: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/79.jpg)
Key response principles
● Log everything
● Frequent public status updates
![Page 80: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/80.jpg)
Key response principles
● Log everything
● Frequent public status updates
● Gather the team
![Page 81: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/81.jpg)
Key response principles
● Log everything
● Frequent public status updates
● Gather the team
● Escalate!
![Page 82: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/82.jpg)
Summary
● Performance
● Architecture
● Downtime
● Preparation
● Where to host?
![Page 83: Experiences from DevOps production: Deployment, performance, failure](https://reader034.vdocuments.us/reader034/viewer/2022052316/557d60c4d8b42abf3d8b512f/html5/thumbnails/83.jpg)
どもありがとうございます
@davidmytton
blog.serverdensity.com
serverdensity.com/allyourbase