mongodb europe 2016 - mongodb, ops manager & docker at sncf

MONGODB, OPS MANAGER & DOCKER @ SNCF

Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON

FEEDBACK ON HOW TO DEPLOY MONGODB REPLICASET IN LESS THAN 5 MINUTES ! (PROOF OF CONCEPT)

MONGODB EUROPE 2016 - NOV 15th - LONDON

Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON

AGENDA

01. MONGODB , PRESENTATION OF THE CONTEXT @ SNCF : Needs & Goals 02. INGREDIENTS / PRE-REQUISITES 03. RECIPES & ORCHESTRATION : how to combine Docker & Ops Manager for mongodb replicaset ? 04. CONCLUSION: Next steps & Improvements





ABOUT SNCF BUSINESS PROFILE


Transport Group offering B to B and B to C services in several fields :




ABOUT SNCF IT @ SNCF


“Production IT” is the operator serving the IT System Departments of the group.

DC #3

DC #1a DC #1b

DC #2

Physical servers

Smartphones

Workstations Fixed telephone line

Operated applications




NEEDS WANTS

HIGH AVAILABILITY

DELIVERY TIME

INDUSTRIALIZE

OPERATIONS

SIMPLIFICATION

SHARED SERVICES

AFFORDABLE PRICE

GOALS

RESILIENT FASTER

CHEAPER

NEEDS AND GOALS SERVICE OFFER



TRANSLATING THE REQUIREMENTS



CHEAPER

FASTER

MORE RESILIENT

SHARED INFRASTRUCTURE

AUTOMATED DEPLOYMENT

BACKUP, ALERTING POLICY



SELECTING INGREDIENTS

SHARED INFRASTRUCTURE

HARDWARE CLUSTER DOCKER

3 large servers (256 GB/6 TB)

3 Data Center rooms



CHEAPER

Lightweight containers

Full autonomy

Very fast startup

FASTER AUTOMATION



DOCKER


Type of container : Small/Medium/Xlarge

Server details




SELECTING INGREDIENTS

FASTER RESILIENT AUTOMATION BACKUP, ALERTS

OPS MANAGER

MongoDB’s Management

Solution



OPS MANAGER

MANAGEMENT

•  MONITORING •  BACKUP •  AUTOMATION

•  FULLY API-DRIVEABLE



INGREDIENTS


1. OPS MANAGER

FASTER AND MORE RELIABLE DEPLOYMENTS

3 nodes for HA 2 MongoDB Enterprise Replica Sets

OPS 1

OPS Manager

HTTP Service

Backup database

(secondaire)

Application database (primaire)

Backup Daemon

Head DB

Head DB

Backup database (primaire)

Application database

(secondaire)

OPS 2

OPS Manager

HTTP Service

Backup Daemon

Head DB

Head DB

Application database

(secondaire)

OPS 3

OPS Manager

HTTP Service

Backup Daemon

Head DB

Head DB

Backup database

(secondaire)

Loadbalancer VIP

Ops Manager




INGREDIENTS


2. HARDWARE CLUSTER

CHEAPER DEPLOYMENT OF

MONGODB Linux Ubuntu 16.04 64bits / 256 GB RAM Pool of 30 IPs pre-allocated per server (10Gb network interface) Local SSD (6TB) storage per server

And some basic orchestration : bash, python script ….




INGREDIENTS


3. DOCKER ENGINE (V1.11)

CHEAPER, FASTER DEPLOYMENTS Standardised MongoDB deployment: 4 containers

And some basic orchestration : bash, python script ….

Mongodb replicaset




PUTTING IT ALL TOGETHER


1 2

3




Our guideline: simple, simple, simple = only 2 commands (maybe 3 !)

l  create_replicaset.py-groupsncf-test-sizesmall–name

sncf1-nb3–filepasswdfile.csv-backupdefault-alertingdefault-envprod–version3.4-dryrun

l  remove_replicaset.py-namereplicaset-name

RECIPE & ORCHESTRATION


Inthefuture:upgrade_replicaset.py–namereplicaset-name–sizelarge–nbnb-

replicaset-member–envprod




create_replicaset.py–help-groupgroup_name-sizecontainer’ssizeS/M/XL–namereplicaset-name-nbnodes(3/3a/5/5a)-filepasswordfile.csv-backuppolicy-alertingpolicy-envprod/preprod-version3.2.10-dryrun(reportingonly)-helpThishelpmessage


Ops Manager API – Create Group

Docker – Create Image

Capacity Planning





r=requests.post(host+"/api/public/v1.0/groups",auth=HTTPDigestAuth(user,key),headers=headers,data=json.dumps(payload))j=r.json()group_id=j["id"]agent_api_key=j["agentApiKey"]





Capacity Planning







Capacity Planning

Dockerfile:FROMubuntu-sncf:16.04MAINTAINERext.osmozium.david.tsang-hin-sun@sncf.frRUNapt-getupdateRUNapt-getinstall-ynet-toolsRUNapt-getinstall-yvim-tinyRUNapt-getinstall-yaptitudeRUNapt-getinstall-ylibsasl2-2RUNapt-getinstall-yssl-certca-certificatesopensslRUNapt-getinstall-ymunin-nodeADDmongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.deb/tmp/mongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.debRUNdpkg-i/tmp/mongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.debADDautomation-agent.config/etc/mongodb-mms/automation-agent.configRUNchownmongodb:mongodb/etc/mongodb-mms/automation-agent.configRUNchmod600/etc/mongodb-mms/automation-agent.configRUNmkdir/data&&chown-Rmongodb:mongodb/dataADDrunautomationagent.sh/runautomationagent.shENTRYPOINT/runautomationagent.sh&&bash



Start containers

Create and mount LVM Volumes

Reserve IP addresses





Start containers

Create and mount LVM Volumes

Reserve IP addresses



dockerrun--name$replica-node-$n\--restart=always-d-hAAA.BBB.CCC.DDD-m2G\--cpu-shares1024--blkio-weight=300\-pAAA.BBB.CCC.DDD:27017:27017-ti\-v/data/replicaset1/AAA.BBB.CCC.DDD:/data:rw–v\/home/docker/vol/replicaset1/AAA.BBB.CCC.DDD:/var/lib/mongodb-mms-automation/:rwsncf/ubuntu-replicaset1:16.04/bin/bash



Ops Manager Backup API

Ops Manager Alerts API

create_replicaset.py–help-groupgroup_name-sizecontainer’ssizeS/M/XL–namereplicaset-name-nbnodes(3/3a/5/5a)-filepasswordfile.csv-backuppolicy-alertingpolicy-envprod/preprod-version3.2.10-dryrun(reportingonly)-helpThishelpmessage

"replicaSets":[{"_id":"testReplicaSet","members":[{"_id":0,"arbiterOnly":false,"hidden":false,"host":"testReplicaSet_0","priority":10,"slaveDelay":0,"votes":1},{"_id":1,"arbiterOnly":false,"hidden":false,"host":"testReplicaSet_1","priority":1,"slaveDelay":0,"votes":1},{"_id":2,"arbiterOnly":true,"hidden":false,"host":"testReplicaSet_2","priority":1,"slaveDelay":0,"votes":1}]}

Ops Manager Automation API








{"clusterId":"CLUSTER-ID","dailySnapshotRetentionDays":7,"groupId":"BACKUP-ID","monthlySnapshotRetentionMonths":13,"pointInTimeWindowHours":24,"snapshotIntervalHours":6,"snapshotRetentionDays":2,"weeklySnapshotRetentionWeeks":4}








{"eventTypeName":"MONITORING_AGENT_DOWN","groupId":"GROUP-ID","notifications":[{"delayMin":0,"emailEnabled":true,"intervalMin":60,"smsEnabled":false,"typeName":"GROUP"}],"typeName":"AGENT",}





Notify user





Scalability ? - Add more physical hosts Security? - limit use of Global Owner role in Ops Manager API Availability? - Use of 3 Distinct DATACENTERS Software upgrades? - Docker 1.12 swarm enabled ? Docker API ? - Graphical interface for true As A Service experience?

IMPROVEMENTS TO THE RECIPE




CONCLUSION



CHEAPER

FASTER

MORE RESILIENT

Several applications hosted on

shared services

End to end industrialization

Streamline architectures

Availability rate

Recovery point objective

Less than one hour data loss



NEXT STEPS & IMPROVEMENTS

Integrate to a software factory

Integrate into the internal

cloud

Create a Disaster

Recovery Plan



Next steps ….

…. and improvements

THANKS MERCI



mongodb europe 2016 - mongodb, ops manager & docker at sncf

Data & Analytics