mongodb in aws (mongodb as a dbaas) - the …return back to cloudformation as custom resource •...

MongoDB in AWS (MongoDB as a DBaaS)

Jing Wu Zhang Lu

April 2017

2

Goals

• Automatically build MongoDB cluster

• Flexible scaling options

• Automatically recover from resource failures

3

Utilizing CloudFormation Template• CloudFormation : Infrastructure As

Code

• CloudFormation template: JSON Formatted file

• Cloudformation Stack: A collection of AWS resources created by AWS CloudFormation.

4

Utilizing CloudFormation Template

• In CloudFormation template, there are no loop/math functions available

• CloudFormation templates are mainly for stateless applications, thus cannot be used directly for building scalable database servers, which are stateful

5

Utilizing Lambda Function

• AWS Lambda lets you run code without provisioning or managing servers. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration

• The code you run on AWS Lambda is uploaded as Lambda function

6

How Lambda Function is used

• Allow users to specify number of shards and nodes per shard as input parameters. The total instances in Cluster will be calculated in Lambda function and return back to CloudFormation as custom resource

• Assign instance initial roles in the Cluster for building process

• Save roles information in DynamoDB table

7

Auto Scaling Group (ASG)

• Monitor the health of database nodes and keep number of healthy nodes constant

• Two ASGs are defined in CFN template. One is for mongod nodes and the other one is for config nodes in the cluster

8

DynamoDB table• Store Cluster metadata information (ip, subnet,

volume ) for all instances provisioned by AWS CFN template

• Also store additional information like memberid, roles in MongoDB cluster, build stage and status

• Provide role information during initial cluster build

• Replacing node will reuse resources from old node stored in DynamoDB to save recovery time

9

Build and Recovery Processes

10

Initial Build of MongoDB Cluster• Lambda function to calculate total instances for

mongod and pre-assign roles and save into DynamoDB table

• ASG for mongod to launch instances based on the number return back form Lambda function

• ASG for config to launch instances based on input parameter for config servers

• Each newly provisioned instance goes through build stage 1-6 to establish a cluster

11

MongoDB Cluster Build Stage 1• Get roles information from DynamoDB table

• Create EBS volumes and configure filesystems

• Create ENI and attach to instance

• Create config files for mongo processes and startup scripts

• Bring up mongo processes

• Update status in DynamoDB table for build stage 1 to complete

12

MongoDB Cluster Build Stage2• For mongod secondary nodes, wait for status to be

updated to stage 5 complete

• For monod primary nodes, initialize replSet, and update status of all nodes in the same replSet to stage 2 complete

• For config node, the 1st config node will be used as primary, initialize the config replSet and update status of all config nodes to stage 2 complete

• The other config nodes, wait for status to be updated to stage 5 complete

13

MongoDB Cluster Build Stage 3

• All mongod and config instances do nothing except pulling DynamoDB table for stage status changes

• On 1st config node, create additional config file and startup scripts for mongos, then start mongos process

• The build process of mongos updates status of all nodes to stage 3 complete.

14

MongoDB Cluster Build Stage 4

• The build process of mongos pulls each replSet info from DynamoDB table, add each replSet into Cluster.

• All mongod and config instances doing nothing except pulling DynamoDB table for stage status changes

• Update status of all nodes in DynamoDB to stage 4 complete.

15

MongoDB Cluster Build Stage 5

• The build process for mongod primary node of each shard creates all logins and update status of all nodes in the same replSet to stag 5 complete

• The build process for mongos creates all logins and update status of all config nodes to stage 5 complete in DynamoDB table

• The build processes of mongod secondary nodes and config nodes wait for stage status change.

16

MongoDB Cluster Build Stage 6

• All nodes update their own config files to enable authentication and SSL.

• All nodes restart their processes.

• Update status of DynamoDB table to stage 6 complete

• Return success signal to CloudFormation

17

18

What Happens When Node Is Down?• ASG will bring up a new instance to replace the failed

instance

• Each new instance will look at DynamoDB table to find the instance to be replaced

• If they are in the same AZ, will re-attach the volumes and ENI, bring node back into cluster

• if it’s in different AZ, cleanup old volumes and ENI, rebuild node, bring it back to cluster and update DynamoDB table

19

Other AWS services used • KMS: to encrypt and decrypt passwords

• ENI: static IPs, re-attach to the new instance. This is to let inter cluster communication to use dedicated channels and also avoid re-config replSets

• SNS/SQS: Database notification and monitor

• IAM: EC2 instance profiles and roles

• S3: Installation scripts EBS : Database storage

20

21

Other Components And Services Used

• Consul: DNS service for registry/discovery to provide one end point for applications

• Enable SSL for MongoDB Cluster during build

• Setup backup using ec2-consistant snapshot

22

Cross Region Replication

• Adding nodes on different regions using the CFN Template for Cross-region

• Cross Region template will access DynamoDB table in primary region for Cluster information, then add new secondary in different region

• The metadata for new secondary will be saved in DynamoDB table in primary region

23

Template Parameters

24

Template Parameters (continued)

25

Template Parameters (continued)

26

Template Parameters (continued)

27

Template Parameters (continued)

28

Q/A

Jing Wujing.wu@teamaol.com

Zhang Luzhang.lu@teamaol.com

Contacts