container orchestration: which conductor?

ContainerOrchestration:WhichConductor?

ContainerConEurope,Berlin,Oct2016

MikeBright, @mjbright

HaikelGuemar, @hguemar

MarioLoriedo, @mariolet

First...Alittlebitofhistory

Solet'sfirstlookatrecentcontainerhistory...

Unix

con

tain

ers,

Unik

erne

lsLi

nux

cont

aine

rs(L

XC),

Mes

osLX

C(C

loud

Foun

dry,

Dot

Clou

dPa

aS)

Dock

er

Cont

aine

rOrc

hest

ratio

nOp

tions

PaaS

ado

ptio

nso

fDoc

ker

Dock

er"S

war

mm

ode"

,OCI

D...

Mic

ro-O

Ses,

Swar

m,R

kt,L

XD

Dock

erb

uysU

nike

rnel

Sys

tem

s

...2009 2013 2014 2016...2015

@hguemar@mjbright@mariolet

History μ-OSesManyvendorsaredevelopingμ-OSes,smallOS(mainlyLinux-based)tobethebasisforcontainerenginehostswhethertheybebare-metalorvirtualhostmachines.

They'resmall,withfaststartup,usefewresourcesandhaveasmallattacksurfaceandoften"atomic"softwareupdates.

OS VendorCoreOS - (CoreOS)ProjectAtomic - (RedHat)RancherOS - (RancherLabs)Photon - (VMWare)NanoServerOS - (Microsoft)UbuntuSnappyCore - (Canonical)

...Unikernels

μ-Services

μ-services Frommonolithstoμ-services

Rememberwhenhighavailabilitymeantthis...?

Active StandbyAppApp

ServersrunningmonolithicapplicationsinActive-Standbymodes,as1+1,N+1,orN+Morsplitacross3tiers.

Scalingmeantto"scaleup"byaddingCPU,RAM,disk.Butthere'salimittothis...thenyouhaveto"scaleout"@hguemar@mjbright@mariolet

μ-services Frommonolithstoμ-services

Thencameμ-services..

Astheindustrymovedtovirtualizedmicro-servicesthisallowedtoobtaingreaterefficiencies(higherutilisationofresources)andtheredesignofapplicationsallowstoscaleoutandachievehighavailability.

Containersfacilitatethismove,allowingfasterscalingandevengreaterefficiencieswithlessredundancy(noOStoreproduce).

@hguemar@mjbright@mariolet

μ-services Howcontainershelp?

ContainersolutionssuchasDockergobeyondtheisolationcapabilitiesofLXCbyprovidingsimpletousetoolstoenablepackagingofappswiththeirdependenciesallowingportableapplicationsbetweensystems.

Containersarelightweight

Versionedimagescontainingalldependanciescanbeshared

Containersallowtousethesameapplicationbinariesondevelopment,testandproductionsystemswhetherthatbeonalaptop,serverorinthecloud.

It'sanobrainerfordevelopers,whocanbuildandsharetheirownimages

@hguemar@mjbright@mariolet

μ-services Frommonolithstoμ-services

But1000'sofnodesareunmanageable...aren'tthey?

Wecan'ttakecareofour

,

sowehavetotreatthemlike

that'scloudnative!

@hguemar@mjbright@mariolet

Soweneedcontainerorchestration

OrchestrationWhatwasContainerOrchestrationagain?

Architecture-Composition&StitchingWorkflows&Policiesto

Scalein/out(maybeautomatically)Placeworkloadsfor

loadbalancing,faulttolerance,resourceoptimization

Adapttofaults

....

....

Master

Worker Worker Worker

....App1 AppNDistrib

Cluster

Config

(quorum)

Master Master

WorkerTasks:

OrchestrationGettingto"DesiredState"Tomanage100's,1000's,10,000'sofnodesweneedtoexpress"desiredstate"ratherthan"dothis".

Imperative DeclarativeTellsystem Dothis desiredstate

"startanewnode" "3mysqlnodes".

Intelligence Operator OrchestrationEngine.

Flexibility Best Least.

Itisnotlongerfeasibleforanoperatorto

knowtheresourcesavailable(e.g.SSD/HDD,GPU,...)reacttofailure,knowwhentoscale...

@hguemar@mjbright@mariolet

Choiceisgreat-whenyouknowwhatyouwant...

Orchestration

TheBig3-MainOrchestrationChoices

DockerSwarm("SwarmMode")

ApacheMesos

Kubernetes

Orchestration

TheBig3-MainOrchestrationChoices

DockerSwarm("SwarmMode")

ApacheMesos

Kubernetes

...moreChoices...

Rancher(RancherLabs)

Fleet(CoreOS)

Nomad(HashiCorp)

Kontena

OpenStackMagnum@hguemar@mjbright@mariolet

OrchestrationTheBig3-WhatdoesGoogleTrendssay?

ClearlyKuberneteshasaleadinGoogle"searchtrends"

Butwecanexpect"DockerSwarm"tomakequickprogressthankstothenew"swarmmode"

DockerSwarm

DockerSwarm

Dec2014 ...DockerSwarmisannounced

OrchestrationusingDockerCompose

Jun2016 ... SwarmToolkitreleasedOpenSourceOrchestrationToolkit

Jun2016 ... SwarmModeannouncedOrchestrationintegratedintoDockerEngine

Docker1.12isthefirstreleasetointegrate"SwarmMode"TheoriginalDockerSwarmismaintainedforlegacyuse.

"SwarmMode"isarevolutionbringing:

OrchestrationdirectlyintheDockerEngineAdvancednetworkingfeatures

meshnetwork,vxlanLoadbalancingServiceDiscoveryDockertraditionaleaseofuse

docker.com @hguemar@mjbright@mariolet

DockerSwarm

Architecture

....

....

Master

Worker Worker Worker

....App1 AppN DistribClusterConfig(quorum)

Master Master

WorkerTasks:

GossipNetwork

DockerEngine DockerEngine DockerEngine DockerEngine

gRPC

docker.com @hguemar@mjbright@mariolet

DockerSwarm

UsingDocker"SwarmMode"Createanewswarmbycreatingthemasternode:

$dockerswarminit--advertise-addr192.168.2.100Swarminitialized:currentnode(dxn1zf6l61qsb1josjja83ngz)isnowamanager.

JoinanewWorkernodetotheswarm:

$dockerswarmjoin--tokenTOKEN192.168.2.100:2377

JoinanewMasternodetotheswarm:

$dockerswarmjoin-tokenmanager

docker.com @hguemar@mjbright@mariolet

DockerSwarmDemo

DockerSwarm

DockerSwarmDemoCreationofa3nodecluster

Runaserviceontheclusterandscaleitto3replicas

Makearollingupdateoftheservice

Drainanodeofthecluster

docker.com @hguemar@mjbright@mariolet

DockerSwarm

Gettingstarted

BrunoCornecs'Docker101LabNewtoDocker?Cometohttp://sched.co/7oHf

Docker101Lab,9am-Friday7thOctober

https://github.com/bcornec/Labs/tree/master/Docker

Followedby..

JeromePetazzoni'sOrchestrationWorkshopormoreadvanced?Cometohttp://sched.co/7oHx

OrchestratingContainersinProductionatScalewithDockerSwarm,Friday7thOctober

https://github.com/jpetazzo/orchestration-workshop

docker.com @hguemar@mjbright@mariolet

Kubernetes

FromtheGreek:"Steersman,helmsman,sailingmaster"

Kubernetes

Googlecreatedbasedonextensiveexperiencerunningcontainersinternally~billionsofcontainersayear

StartedOct2014,reachedv1.0inJuly2015,nowatv1.4

ManagedbytheCloudNativeComputingFoundationhttps://cncf.io/

CommercialofferingsfromCoreOS(Tectonic)andCanonical

Integratedin:

GKE(GoogleContainerEngine)OpenStackaboveKubernetes

Stackanetes(CoreOS,usesTectonic)SelfhealingOpenStackdemo

Mirantis(OpenStackCI/CDbasedonKolla)VariousPaaS:

RedHatOpenShiftCPHPEStackatov.40Deis

kubernetes.io @hguemar@mjbright@mariolet

Kubernetes

Architecture

....

....

Master

Worker Worker Worker

....App1 AppN etcd

Master Master

WorkerPods: 192.168.100.10

ServicesLabels,Selectors

kubernetes.io @hguemar@mjbright@mariolet

ApacheMesos

ApacheMesos

Themostprovenorchestratortoday,existssince2009.

Canscaleto~10,000nodes.

Usedinproductionby:

UberTwitterPaypalHubspotAirbnbeBayGrouponNetflix

SupportsContainerizerstoisolatetasks

mesos.apache.org

ApacheMesos

Architecture

....

....

Master

Worker(agent)

....Spark AppNDistrib

Cluster

Config

(quorum)

Master Master

WorkerTasks:

ResourcesAvailable

Scheduler

SparkExecutor SparkExecutor Executor

Offer

AcceptRegister

Worker(agent) Worker(agent)

mesos.apache.org

ApacheMesos

MesosisusedinconjunctionwithFrameworkssuchas

Forlongrunningtasks:

Marathon(Mesosphere),AuroraorSingularity

Forjoborchestration:

Chronos"cron",Jenkins

ForBigDataProcessing:

Hadoop,Spark,StormCassandra,ElasticSearch,...

mesos.apache.org@hguemar@mjbright@mariolet

Soisn'tittimewetoldyouwhattochoose?

...let'sjustcomparethem...

What'scommon

DockerSwarmandKubernetesarecreatingrichOrchestrationstackswithintegratedruntimes.

They'removingincrediblyquicky...

Theyareaddingfeaturessuchasnetworkingcapabilities,loadbalancing,services,labels.

Theyhaveamore'declarative'approach

Theysupportorarelookingtosupportdifferentruntimeengines(*)

@hguemar@mjbright@mariolet

Whatadvantages?Docker"Swarm

Mode"Simpletouse(despiteunderlyingcomplexity)

All-in-onecontainerengineplusorchestration

UsesDockerAPIandfamiliardockercommands

Advancednetworking

-meshnetworking-LoadBalancingandServiceDiscovery

Replication

@hguemar@mjbright@mariolet

Whatadvantages?Kubernetes

Richconceptualmodel

Podsasgroupingsofcontainers

LabelsandSelectors(forallcomponents)

Largeecosystem

Networking

-LoadBalancingandServiceDiscovery

Replication

@hguemar@mjbright@mariolet

Whatadvantages?ApacheMesos

Mostmature

Battletestedbymanyserviceproviders

Scalesto10,000nodes

DataCenterOS-appearsas1resource

Notjustcontainers

Manyframeworksavailable

Difficultramp-up

@hguemar@mjbright@mariolet

Handson...

@hguemar@mjbright@mariolet

Hands-on Comealong

Thisafternoon'stutorialsessionledbyMario:Tuesday,October4-15:30-16:20

5Containersfor5Languages:PatternsforSoftwareDevelopmentUsingContainers-MarioLoriedo,RedHat

Tomorrow'slabsessionledbyHaikel:Wednesday,October5-11:00-12:50

ContainerOrchestrationLab:Swarm,Mesos,Kubernetes-HaïkelGuémar,FedoraProject

Labsetupinstructionshere

DockerSwarmKubernetesApacheMesos

@hguemar@mjbright@mariolet

Questions?Thankyou

@hguemar@mjbright@mariolet

Resources

@hguemar@mjbright@mariolet

ResourcesBooksPublisher Title AuthorOReilly DockerCookbook SébastienGoasguen

OReilly DockerUp&Running

KarlMatthias,SeanP.Kane

OReilly UsingDocker AdrianMouat

OReilly[EarlyAccess]KubernetesUp&Running

KelseyHightower

Manning [MEAP]CoreOSinAction MattBailey

Manning [MEAP]KubernetesinAction MarkoLukša

@hguemar@mjbright@mariolet

ResourcesArticles/OrganismsCloudNativeComputingFoundation-Kubernetes,Prometheushttps://cncf.io/

"KubernetestheHardWay,KelseyHightower"-https://github.com/kelseyhightower/kubernetes-the-hard-way

"KubernetesUserGuide,Walkthrought"-http://kubernetes.io/docs/user-guide/walkthrough/

@hguemar@mjbright@mariolet

ResourcesVideosJune2016-ContainerOrchestrationWars,KarlIsenberg,Mesosphere

Mar2016-ContainerOrchestrationwithKubernetes,DockerSwarm&Mesos-Marathon-AdrianMouat,ContainerSolutions

Jan2016-Docker,Kubernetes,andMesos:Compared.,,AdrianOtto,SouthernCaliforniaLinuxExpo

Repos

@hguemar@mjbright@mariolet

Kubernetes

DocumentationGettingstartedguides

CreatingaKubernetesClusterportKubernetestoanewenvironment

inGettingStartedfromScratchUserdocumentation

torunprogramsonanexistingKubernetescluster

KubernetesUserGuide:ManagingApplications

theKubectlCommandLineInterfaceisadetailedreferenceonthekubectlCLIUserFAQ

kubernetes.io @hguemar@mjbright@mariolet

Kubernetes

Documentation-2

ClusteradministratordocumentationforpeoplewhowanttocreateaKubernetesclusterandadministeritintheKubernetesClusterAdminGuide

DeveloperandAPIdocumentationtowriteprogramsusingtheKubernetesAPI,writepluginsorextensions,ormodifycorecodeKubernetesDeveloperGuidenotesontheAPIAPIobjectdocumentation,adetaileddescriptionofallfieldsfoundinthecoreAPIobjects

Walkthroughsandexampleshands-onintroductionandexampleconfigfilesintheuserguideinthedocs/examplesdirectory

ContributionsfromtheKubernetescommunity

inthedocs/contribdirectory

kubernetes.io @hguemar@mjbright@mariolet

Kubernetes

Documentation3Designdocumentationanddesignproposals

tounderstandthedesignofKubernetes,andfeatureproposalsKubernetesDesignOverviewandthedocs/designdirectorydocs/proposalsdirectory

Wiki/FAQthewikitroubleshootingguide

Community,discussion,contribution,andsupport

ConsiderjoiningtheCloudNativeComputingFoundation.Fordetailsaboutwho'sinvolvedandhowKubernetesplaysarole,readtheirannouncement.

kubernetes.io @hguemar@mjbright@mariolet