operationalizing docker at scale: lessons from running microservices in production

M M / D D / Y Y

YOUR T ITLE HERE

P R E P A R E D F O R :

P L A C E L O G O

H E R E

Operationalizing Docker at ScaleLessons from running microservices in production

V e n k a t Thiruvengadam, ZenefitsM a x i m e P e t a z z o n i , S i g n a l F x

Introductions

Venkat Thiruvengadam• Principal Engineer @ Zenefits

• Creator of Project Duplo, a hosting platform for microservices

Maxime Petazzoni• Software Engineer @ SignalFx

• Creator of MaestroNG, a container orchestrator for Docker environments

M M / D D / Y Y

YOUR T ITLE HERE

P R E P A R E D F O R :

P L A C E L O G O

H E R E

Micro-services @ Zenefits

Micro-services@ZenefitsThiruvengadamVenketesan

PrincipalEngineer

ZenefitsEngineering

• 200Developers• MonolithDjango ApplicationonMySQLBackend• TransitioningtoSOA

• Dozensofservices,fewHundredcontainers• HostedinAWS(Noonpremiseservers)• Duplo:Self-serviceHostingplatformforMicroservices

• NoDirectAWSaccessfordevelopers• NodedicatedDevops

MonolithvsMicroservices

• Monolith• Advantages

• SingleCodeBase.Easytoread/debug/deploy• Nointer-servicecontracts,compatibilityorauthenticationconcerns.• Easytogetstarted.Greatforsmallteams.

• Disadvantages• Singlepointoffailure• Slowiterationwithlargeteams(Managedbycentralopsteam)• Singleprogramminglanguage&technologypattern

• Microservices• Independentdeploymentandhencefasteriteration• DistributedFunctionality,nosinglepointoffailure• ProgrammingLanguage&technologyagnostic

Micro-services:TheDarkSide

ContainerManagement

ServiceDiscovery

ConfigurationManagement

Authentication

ManagementComplexityw/oright

tools

AWSResourceManagement

ResourceUtilization Monitoring

FindYourTippingPoint

Microservices

Monolith

TeamandProductSize

Productivity

@zenefits 3years,>millionlinesofcodeand200devs,wedecidedtomovetoMicroservices

OriginalapproachtoMicro-services@Zenefits

Developers

InfraAdmin

1

TranslatetoAWSConfiguration2

ApplyConfigurationusingTerraform\Otherautomation

3

SaveInfrastructureconfigurationlikeVPC,Subnet,SecurityGrp,IAM,EC2

4

5

SaveApplicationconfigurationlikenamecredentialsinappcode;builddocker image

6

7

DeploytoECS8

- Authenticationtokens,servicediscovery,configurationmanagementfollowsimilarpattern- Infrabecomesbottleneck.Notscalablewithlargenumberservicesandusers

Anonself-serviceinfrastructuredefeatsthepurpose

IGetSelf-service,DoIneedMulti-Tenancy?

Multi-Tenant:Eachapplication(orservice)isatenant.Usershaveaccesstoonlytheirrespectiveservices

Multi-tenancy(appteam)isnecessaryifyourorganizationneeds:- Resourceandsecurityisolationbetweenservices- Isolationofconfigurationmistakes- Audittrails- Varyingresourceprivilegesbetweentenants- VaryingDevelopersecurityawareness- Largenumberofdevelopers

Notenancy

TeamSize

Mis-config andSecurityRisksMulti-tenancy

How can I do CICD in Duplo?

Let’sgiveyouAWSaccess,createyourservice

WhatisthisSecuritygroup,subnet,IAM?Whydoesmydjangoappneedthis?Willallow*work?

Hmm…..

ShallwecreateanAWSusernameforeveryone?

WeNeedaHostingPlatform

TranslatetoAWSConfiguration

2

- ApplyConfigurationusingAWSandDockerAPIs- InjectconfigurationinappENV&discoveryservice

3

ApplicationrequirementsandDockerImageviaUI/API

1

Developers

Admin

- SelfService.Deployandmanageat-will.- Multi-tenant.Eachserviceorapplicationteamhavetheirownaccounts,resourcepool,IAMrole,SGetc.- ServicesisolatedexceptexplicitAPIinterfaces.- ProgrammaticInfrastructure,consistentconfigurationgeneration

- CreatebaseinfraConfigurationinAWSusingTerraform- InstallDuplo.Setupstaticbaseinfrastructurepolicies.- CreatestaticTenantpoliciesandlimitsa.k.a.“Plans”

0

DUPLO

DEMO:DEPLOYINGAMICRO-SERVICE

DuploMicro-servicesPlatform

DUPLO

AWSOrchestrationandabstraction

ContainerManagement

CI/CD

EcosystemOrchestrationAuthentication

ConfigurationManagement

ServiceDiscovery

HowDuploWorks

BaseInfrastructure:VPC,Subnets,NAT,AdminSecurityGroups,DNSDomainName,SSLWildcharCertTerraform

DUPLOAdminPortalNewServiceRegistration

IAMRole

SecurityGroup

Autocreateperservice

SecurityGroup

IAMRoleNewServiceRegistration

AwsResources

TenantPortal

DUPLO

CreatedbytenantactionsAws

Resources

DeployandManageService

TheNextStep:ContinuousIntegrationandDelivery

DuplocandeployimageswithdesiredAWSResources.It’sselfservice.WhydoIneedCI\CD?

• OfficialBuildImagesasagainstLaptoptests

• Testsuitesthataredocumented,enforcedwithpasscriterion

• Integrationenvironmentforintegrationtestswithotherservices

• DeploymentPipelineDevsandboxtostagetoprod.

WhatisspecialaboutaCI\CDsolutionforMicroservices,canIjustuseJenkins?

- MonolithCI/CDisonlyaboutcodecorrectnessandbuildartifacts- TheycanruninanInfratopologyseparatethantherealdeployment

- Beyondcode,MicroservicesCI/CDisaboutreplicatingtherealdeploymentinfratopology- ItneedstotesttheinterfaceoftheservicewithitscomplexinfrastructurelikeIAM,Securitygroupsetc.

- ServicesCI/CDneedstointegrateandbuildontopofthehostingplatform

CI\CDDEMO:GITPRToDeployment

KatKit:CI/CDExtensiontoDuplo

• After-allbuildscanbeaMicroservice too,theyarejustshortlived!Let’sreuseDuplotolaunchandterminatebuilds.

• Let’sbuildaworkfloworchestrationontopofduplo forCI/CD.

• WegetCI/CDandHostingPlatform“integration”forfree

SUMMARY

• Determinethetippingpoint.

• SelfserviceisP0.Determineifmulti-tenancy(isolation)isnecessary

• Hostingplatformisnecessary

• AWSbyitselfisnotasufficientsolution,needahigherlayerplatformtoprovideabstraction,containermanagementandCI/CD

• CI/CDsolutionhastointegratewiththehostingplatformandtestitsinteractionwiththeservicecode.

M M / D D / Y Y

YOUR T ITLE HERE

P R E P A R E D F O R :

P L A C E L O G O

H E R E

Q&A

For more information

• Duplo Platform• https://engineering.zenefits.com/2016/03/duplo-aws-orchestration-container-

management-and-paas-for-microservices/

• https://engineering.zenefits.com/2016/07/duplo-cicd-extensions-for-microservices/

• https://www.linkedin.com/pulse/duplo-vs-kubernetes-mesos-thiruvengadam-venketesan

• Monitoring Docker containers• https://signalfx.com/blog/monitoring-docker-containers-take-get-started/

• https://signalfx.com/blog/monitoring-docker-containers-signalfx-monitors-containerized-infrastructure/

• https://signalfx.com/blog/monitoring-docker-at-scale-with-signalfx/

M M / D D / Y Y

YOUR T ITLE HERE

P R E P A R E D F O R :

P L A C E L O G O

H E R E

THANK YOU!

S IGN UP FOR A TR IAL AT:

s i gna l f x . com