mongodb on aws in 5 min

MongoDB on AWS

david@davidmturner.com


David TurnerI look after the internet

125

david@davidmturner.comdavid@davidmturner.com124

david@davidmturner.com123 david@davidmturner.com

david@davidmturner.com120


M102: MongoDB for DBAs


100%

116


M101P: MongoDB for Programmers


Less than 100%

114


What is MongoDB?

112


Prebaked AMI

109

MongoDB with 1000 or 4000 PIOPS


Up and running in minutes

107


Not ready for production

106


Approach

1. configure AWS objects

2. instantiate instances using AWS CLI tools

3. scripted install with user data bash script

4. initialise the replica set

102


Configure VPC

101


Subnets

100


$ aws ec2 create-subnet --vpc-id vpc-xxxxxxxx --cidr-block 10.0.1.0/24 --availability-zone eu-west-1a

99


$ aws ec2 create-subnet --vpc-id vpc-xxxxxxxx --cidr-block 10.0.2.0/24 --availability-zone eu-west-1b


Route Table

97


Assuming you have a NAT instance already

96


$ aws ec2 create-route-table --vpc-id vpc-xxxxxxxx

95


$ aws ec2 create-route --route-table-id rtb-xxxxxxxx --destination-cidr-block 0.0.0.0/0 --instance-id i-xxxxxxxx

94


$ aws ec2 associate-route-table --route-table-id rtb-xxxxxxxx --subnet-id subnet-xxxxxxx1

93


$ aws ec2 associate-route-table --route-table-id rtb-xxxxxxxx --subnet-id subnet-xxxxxxx2


Security Group

91


$ aws ec2 create-security-group --group-name MongoDB --description MongoDB --vpc-id vpc-xxxxxxxx

90


Ingress

89


$ aws ec2 authorize-security-group-ingress --group-id sg-xxxxxxxx --protocol tcp --port 27017 --source-group sg-xxxxxxxx

88


$ aws ec2 authorize-security-group-ingress --group-id sg-xxxxxxxx --protocol tcp --port 27017 --source-group sg-yyyyyyyy

87


$ aws ec2 authorize-security-group-ingress --group-id sg-xxxxxxxx --protocol tcp --port 22 --source-group sg-zzzzzzzz

86


Egress

85


$ aws ec2 authorize-security-group-egress --group-id sg-xxxxxxxx --protocol tcp --port 27017 --source-group sg-xxxxxxxx

84


$ aws ec2 authorize-security-group-egress --group-id sg-xxxxxxxx --protocol tcp --port 80 --cidr 0.0.0.0/0

83


$ aws ec2 authorize-security-group-egress --group-id sg-xxxxxxxx --protocol tcp --port 443 --cidr 0.0.0.0/0


$ aws ec2 authorize-security-group-egress --group-id sg-xxxxxxxx --protocol udp --port 123 --cidr 0.0.0.0/0


Network ACLs

80


Lazy. Default ALLOW.

79


Placement Groups

77


$ aws ec2 create-placement-group MongoDB-a

76


$ aws ec2 create-placement-group MongoDB-b


IAM

74


TrustPolicy.json: { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

73


MongoDBPolicy.json

{ "Version": "2012-10-17", "Statement": [ { "Action": ["ec2:DescribeVolumes", ec2:DescribeTags, "ec2:DescribeInstances", "ec2:DescribeInstanceAttribute"], "Resource": ["*"], "Effect": "Allow" }, { "Action": ["ec2:DeleteSnapshot", ec2:CreateSnapshot", "ec2:DescribeSnapshots"], "Resource": ["*"], "Effect": "Allow" }, { "Action": ["ec2:CreateTags"], "Resource": ["*"], "Effect": "Allow" } ] }

72


Role

71


$ aws iam create-role --role-name MongoDB --assume-role-policy-document file://TrustPolicy.json

70


$ aws iam put-role-policy --role-name MongoDB --policy-name MongoDB-Policy --policy-document file://MongoDBPolicy.json

69


Instance Profile

68


$ aws iam create-instance-profile --instance-profile-name MongoDB

67


$ aws iam add-role-to-instance-profile --instance-profile-name MongoDB --role-name MongoDB

66


fire up instances with a bash userdata script

65


$ data=$(cat ./user-data.sh)

64


# volume sizes in GB data_size=500 log_size=15 journal_size=25

63


$ aws ec2 run-instances --region eu-west-1

62


--security-group-ids sg-xxxxxxxx

61


--key-name TopSecretKeyPair

60


--iam-instance-profile {Arn:"arn:aws:iam::012345678901:instance-profile/MongoDB_Instance_Profile"}'

59


--instance-type r3.large

(2 vCPU, 15.25GB RAM)

58


--block-device-mapping [{ "DeviceName": "/dev/xvdf", Ebs: {"VolumeSize":'$data_size', "VolumeType": "io1", "Iops": 1000}}, {DeviceName": "/dev/xvdg", "Ebs": {VolumeSize":'$data_size', VolumeType: "io1", "Iops": 1000}}, {"DeviceName": "/dev/xvdh", Ebs": {VolumeSize":'$journal_size', "VolumeType": "io1", "Iops": 250}}, {DeviceName": "/dev/xvdi", "Ebs": {VolumeSize":'$log_size', "VolumeType": "io1", "Iops": 150}}]

57

(data)

(journal)

(log)


--placement AvailabilityZone=eu-west-1a,GroupName=MongoDB

56


--disable-api-termination

55


--image-id ami-a10897d6

Amazon Linux AMI 2015.03 (HVM), SSD Volume TypeRoot device type: ebs Virtualization type: hvm

54


--subnet-id subnet-xxxxxxxx

53


user-data $data

52


count 2

51

(then same again for AZ eu-west-1b)


yum -y update

50


mdadm --verbose --create --name=mongo /dev/md0 --level=0 --chunk=256 --raid-devices=2 /dev/xvdf /dev/xvdg

mdadm --detail --scan | tee -a /etc/mdadm.conf

49


mkdir /mnt/data /mnt/journal /mnt/log

mkfs.ext4 /dev/md0 mkfs.ext4 /dev/xvdh mkfs.ext4 /dev/xvdi

48


uuid=`blkid -o value -s UUID /dev/md0`

echo "UUID=$uuid /mnt/data ext4 defaults,auto,noatime,noexec 0 0 /dev/xvdh /mnt/journal ext4 defaults,auto,noatime,noexec 0 0 /dev/xvdi /mnt/log ext4 defaults,auto,noatime,noexec 0 0" >>

/etc/fstab mount -a

47


ln -s /mnt/journal /mnt/data/journal

46


echo "[MongoDB] name=MongoDB Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 enabled=1" >> /etc/yum.repos.d/mongodb.repo

45


yum install -y mongodb-org-2.6.1 mongodb-org-server-2.6.1 --exclude mongodb-org, mongodb-org-server

service mongod stop

44


yum install -y sysstat gcc python27 python27-pip python27-devel

pip-2.7 install pymongo boto

43


chown mongod:mongod /mnt/data chown mongod:mongod /mnt/journal chown mongod:mongod /mnt/log

42


echo "dbpath=/mnt/data logpath=/mnt/log/mongodb.log logappend=true fork=true replSet = AWSUserGroup " > /etc/mongod.conf

41


echo "mongod soft nofile 64000 mongod hard nofile 64000 mongod soft nproc 32000 mongod hard nproc 32000" > /etc/security/limits.d/90-mongo.conf

40


echo 'ACTION=="add", KERNEL=="md*", ATTR{bdi/read_ahead_kb}="16"' >> /etc/udev/rules.d/85-ebs.rules echo 'ACTION=="add", KERNEL=="xvdf", ATTR{bdi/read_ahead_kb}="16"' >> /etc/udev/rules.d/85-ebs.rules echo 'ACTION=="add", KERNEL=="xvdg", ATTR{bdi/read_ahead_kb}="16"' >> /etc/udev/rules.d/85-ebs.rules echo 'ACTION=="add", KERNEL=="xvdh", ATTR{bdi/read_ahead_kb}="16"' >> /etc/udev/rules.d/85-ebs.rules echo 'ACTION=="add", KERNEL=="xvdi", ATTR{bdi/read_ahead_kb}="16"' >> /etc/udev/rules.d/85-ebs.rules

39


echo "/mnt/log/mongodb.log { daily rotate 30 compress dateext missingok notifempty sharedscripts prerotate grep query /mnt/log/mongodb.log | logger -t mongodb -p warn endscript postrotate /bin/kill -SIGUSR1 \$(/bin/cat /mnt/data/mongod.lock) rm -f /mnt/log/mongodb.log.[0-9][0-9][0-9][0-9]-* endscript } " > /etc/logrotate.d/mongod

38


echo "net.ipv4.tcp_keepalive_time = 120" > /etc/sysctl.d/01-mongod.conf

37


chkconfig mongod on

36


reboot35


Route53

34


Skipping it for time

33


Configure MongoDB

31


rs.initiate()

30


rs.add(x.x.x.x)

29


rs.add(y.y.y.y)

28


rs.status()

27


Backups

26


Hidden Member

24


EBS Snapshots

23


Hourly cronjob

22


Python Script

Lock the database

Lock the filesystem

EBS Snapshot

Unlock filesystem

Unlock database

Trim snapshots

21


if __name__ == "__main__": conn = boto.ec2.connect_to_region(REGION) client.admin.command("fsync", lock=True) p = subprocess.Popen( '/usr/bin/sudo /sbin/fsfreeze -f /mnt/data', shell=True) p.wait() create_snapshots(conn, get_volume_ids(conn)) p = subprocess.Popen( '/usr/bin/sudo /sbin/fsfreeze -u /mnt/data', shell=True) p.wait() client.admin[$cmd'].sys.unlock.find_one()

conn.trim_snapshots(hourly_backups=3, daily_backups=7, weekly_backups=4, monthly_backups=True)

20


Development instances

19


--block-device-mapping [ { "DeviceName": "/dev/xvdf", "Ebs": {"SnapshotId": snap-xxxxxxxx, "VolumeSize":'$data_size', "VolumeType": gp2"} },{"DeviceName": "/dev/xvdg", "Ebs": {"SnapshotId": snap-xxxxxxxx", "VolumeSize":'$data_size', "VolumeType": gp2"} }

18


Development instance in less than 15 min

17


Havent Shown

16


EBS Encryption and KMS

15


Multi-region

14


Sharded configuration

13


Log management

12


Monitoring

11


Would like to meet...


Images4: http://i.ytimg.com/vi/-xcecNrpChQ/maxresdefault.jpg

15: http://elginsweeper.com/portals/0/Images/Application_Photos/Runway.gif

17: http://arabhardware.net/wp-content/uploads/2014/12/ram-04.jpg

18: http://www.public-domain-image.com/full-image/objects-public-domain-images-pictures/an-old-west-style-padlock-in-old-town-san-diego.jpg-free-stock-photo.html

20: http://orbital.comp.nus.edu.sg/wp-content/uploads/2013/02/STS-133_Discovery_Lift_Off_Launch_Pad_39A_KSC.jpg

23: http://www.layman.org/wp-content/uploads/2013/10/face.jpg

26: http://blog.mongodb.org/post/4982676520/mongodb-on-ec2-best-practices

122: https://dogchow.com/media/28148/how_do_i_stop_my_dog_from_begging_istock_000012144475small.jpg

124: http://www.salus-wellness.com/wp-content/uploads/2010/11/Business.jpg

126: http://www.twinfinite.net/wp-content/uploads/2015/03/Lego.jpg

mongodb on aws in 5 min

Software

program introduction and overvie€¦ · module 9:...

full-stack web development with mongodb, node.js and aws

mongodb on aws - amazon web services · pdf filemongodb on...

running mongodb 3.0 on aws

webinar: serverless architectures with aws lambda and...

2015 05-09 deploy mongodb replicaset in aws

mongodb in aws (mongodb as a dbaas) - the …return back to...

securing mongodb to serve an aws-based, multi-tenant,...

mongodb world 2016: mongodb & ibm

mongodb on the aws cloud: quick start reference deployment

nosql database in the loud: mongod on aws -...

aws re:invent 2016: how thermo fisher is reducing mass...

mongodb training | mongodb online training | mongodb...

lezione aws mongodb -...

mongodb on the aws cloud · mongodb atlas creates a new...

automate mongodb with mongodb management service

mongodb europe 2016 - mongodb 3.4 preview and introduction...

mongodb: what, why, when. solutions architect, mongodb inc....

maximizing mongodb performance on aws

mongodb and aws: integrations