best practices for running mongodb on aws - aws may 2016 webinar series
TRANSCRIPT
![Page 1: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/1.jpg)
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rahul Bhartia, Partner Solution Architect
24th May 2016
MongoDB on AWSAn introduction and best-practices for AWS
![Page 2: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/2.jpg)
MongoDB : Basics
![Page 3: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/3.jpg)
Document Model
RDBMS
{ first_name: ‘Paul’, surname: ‘Miller’, city: ‘London’, location: [45.123,47.232], cars: [ { model: ‘Bentley’, year: 1973, value: 100000, … }, { model: ‘Rolls Royce’, year: 1965, value: 330000, … } ]}
MongoDB
![Page 4: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/4.jpg)
Features
Rich Queries Find Paul’s carsFind everybody in London with a car between 1970 and 1980
Geospatial Find all of the car owners within 5km of Trafalgar Sq.
Text Search Find all the cars described as having leather seats
Aggregation Calculate the average value of Paul’s car collection
Map Reduce What is the ownership pattern of colors by geography over time (is purple trending in China?)
![Page 5: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/5.jpg)
MongoDB: New features in 3.2
![Page 6: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/6.jpg)
New Storage engines
In-memory
Run your most demanding, real-time apps with
the in-memory storage engine (beta)
Encrypted
Secure your data-at-rest with the encrypted
storage engine
![Page 7: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/7.jpg)
Data Governance: Document Validation
Implement data governance without sacrificing
the agility that comes from a dynamic schema
![Page 8: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/8.jpg)
Dynamic Lookup
Combine data from multiple collections with left
outer joins for richer analytics & more flexibility
in data modeling
![Page 9: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/9.jpg)
Richer In-Database Analytics & Search
New Aggregation operators extend options for
performing analytics with lower developer
complexity
Array Operators
Math Operators Text
$slice,
$arrayElemAt,
$concatArrays,
$filter, $min,
$max, $avg,
$sum, and
more.
$stdDevSamp,
$stdDevPop,
$sqrt,$abs,
$trunc, $ceil,
$floor, $log,
$pow
$exp, and
more.
Case sensitive text search and support for languages such as Arabic, Farsi, Chinese, and more.
![Page 10: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/10.jpg)
MongoDB Connector for BI
Visualize and explore multi-structured data using
SQL-based BI platforms.
Connector for BIProvides Schema
Translates QueriesTranslates Response
![Page 11: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/11.jpg)
Compass: The GUI for MongoDB
Visually explore your data and schema.
Run ad hoc queries in seconds
Make smarter decisions about indexing,
document validation, and more.
No command line needed
![Page 12: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/12.jpg)
MongoDB: High Availability & Scalability
![Page 13: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/13.jpg)
Replica Sets
Replica set – 2 to 50 copies
Replica sets make up a self-healing ‘shard’
Replica sets address - High availability, Maintenance (e.g., HW swaps) and Disaster Recovery
Application
Driver
Primary
Secondary
Secondary
Replication
![Page 14: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/14.jpg)
Replica Sets – Workload Isolation
Replica sets enable workload isolation
Example: Operational workloads on the primary
node, analytical workloads on the secondary
nodes
Application
PrimaryIn-memory
SecondaryWiredTiger
User DataSessions, Cart,Recommendations
SecondaryWiredTiger
PersistedUser Data
![Page 15: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/15.jpg)
Elastic Scalability: Automatic Sharding
Increase or decrease capacity as you go
Automatic load balancing
Three types of sharding - Hash-based, Range-based and Tag-aware
Shard 1
Shard 2
Shard 3
Shard N
Horizontally Scalable
![Page 16: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/16.jpg)
Query Routing
Multiple query optimization models
Each of the sharding options are appropriate for different apps / use cases
![Page 17: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/17.jpg)
MongoDB: AWS Deployment Architectures
![Page 18: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/18.jpg)
AWS EC2 Instance Types
General Purpose - M3, M4• Start with General Purpose instances and EBS GP2
Compute-optimized - C3, C4• WiredTiger - Write performance due to document-level concurrency control
Memory-optimized - R3• In memory storage engine: NEW!• Larger working set - MMAPv1 with read-intensive applications
Storage-optimized - I2, D2• Local instance store but lost when instance is stopped or terminated• Always use with higher replication
https://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#deployment-notes
![Page 19: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/19.jpg)
Amazon EBS Volumes
EBS GP2 for general workloads and EBS PIOS for consistent performancehttps://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#storage-considerations
EBS-optimized Instances provides additional/dedicated bandwidthhttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html
EBS snapshots for recovery and backupshttps://docs.mongodb.com/ecosystem/platforms/amazon-ec2/#backup-restore-verify
![Page 20: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/20.jpg)
AWS Global Footprint
![Page 21: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/21.jpg)
High Availability: Intra-Region
![Page 22: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/22.jpg)
High Availability: Intra-Region
![Page 23: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/23.jpg)
High Availability: Inter-Region
![Page 24: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/24.jpg)
Disaster Recovery: Inter-Region
![Page 25: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/25.jpg)
Disaster Recovery: Inter-Region
![Page 26: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/26.jpg)
Go global
Deploy Global, Access Local
Replicate Data Near Users
![Page 27: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/27.jpg)
MongoDB - Management
![Page 28: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/28.jpg)
The Best Way to Run MongoDB
Ops Manager and Cloud Manager allows you
leverage and automate the best practices.
10x-20x more efficient operations
Complete performance visibility
Protection from data loss
Performance optimization
![Page 29: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/29.jpg)
How they work
Ops or Cloud Manager
mongod mongodmongod
Agent Agent Agent
![Page 30: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/30.jpg)
Amazon QuickStart
https://aws.amazon.com/quickstart/
VPC with private and public subnets
Instance role with fine-grained permissions.
Security groups
A fully customized MongoDB cluster - Replica sets, shards, and config servers and customized EBS storage
![Page 31: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/31.jpg)
Monitoring & Alerting
Over 100+ database-related metrics
Dozens of optimized charts
Custom alerts so incidents do not become emergencies
![Page 32: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/32.jpg)
Do-It-Yourself
Set of utilities distributed with MongoDB – mongostat, mongotop
Database commands – serverStatus, dbStats, collStats etc
Self-hosted – Ganglia, mtop, munin, nagios
Hosted (SaaS) – New Relic, Datadog, Server Density
https://docs.mongodb.com/manual/administration/monitoring/
![Page 33: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/33.jpg)
Backup with Point-in-time Recovery
Cluster-wide snapshots of sharded clusters
Restore to precisely the moment you need, quickly and safely with point-in-time restores
![Page 34: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/34.jpg)
Manual Backup Considerations
Consider a hidden member in a Replica set
Consider EBS Snapshots
Consider Journaling (Write Ahead Log), and allow for DB durability in case of a fault
Ensure consistency by using db.fsyncLock()
https://docs.mongodb.com/ecosystem/tutorial/backup-and-restore-mongodb-on-amazon-ec2/
![Page 35: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/35.jpg)
Visual Query Profiler
Identify the slow-running queries across your cluster with just the click of a button
Index suggestions to improve your query performance
Automate rolling index builds to reduce operational overhead and the risk of failovers
![Page 36: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/36.jpg)
Resources
MongoDB on AWS best practices:http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
MongoDB production Noteshttp://docs.mongodb.org/manual/administration/production-notes/
MongoDB Documentationhttp://docs.mongodb.org
![Page 37: Best Practices for Running MongoDB on AWS - AWS May 2016 Webinar Series](https://reader034.vdocuments.us/reader034/viewer/2022051521/5873ecb51a28abb1528b4755/html5/thumbnails/37.jpg)
Thank you!