build applications like google · kubernetes glossary (part 1) master: the managing machine, which...

Build applications like Google: Using containers, Kubernetes and Mesos

CloudOpen 2014, DüsseldorfBernd Mathiske, MesosphereNat Welch, Google

1. Keep programs simple

2. Prefer smaller, stateless programs

Images by Connie Zhou

A 2000-machine cluster will have >10 machine crashes per day

DRAM errors (1% AFR)Disk failures (2-10% AFR)Machine crashes (~2/year)OS upgrades (2-6/year)

Agenda● Hardware => GCE

● Containers => Docker

● Cluster management => Mesos

● Container management => Kubernetes

● Platform stacks for large distributed apps

● How to get started

Google Compute Engine

● VMs as service

● Europe, Asia, US

● Live migration

Problems with raw VMs

● Expensive to turn up (time)

● "Hard" to make repeatable

● Difficult to replicate locally for testing

● High possibility of failure

Container

● Lightweight Linux execution environment○ libcontainer, LXC, LMCTFY

● Static application composition○ Reliable deployment

● Unit of resource isolation○ Multi-tenancy without heavyweight VMs

Docker● Open source

● Configurable layers

● Reproducible

● Version-controlled

● Lots and lots of other peoples containers○ https://registry.hub.docker.com/

Run everything in containers!

Mesos facts● Supports Docker

● Scales to 10,000s of nodes

● Top-level Apache project

● Twitter, Airbnb are major users and contributors

● APIs for C++, Python, JVM-languages, Go, ...

● Pluggable CPU, memory, IO isolation

● Highly configurable, with easy-to-use defaults

● Packages and support through Mesosphere

How Mesos worksApplication

Framework

Mesos Slave

Executor

Task

Task

TaskExecutor

Scheduler

Mesos Master

Mesos MasterMesos

Master

Zookeeper

Mesos Master

Kubernetes● Employs and promotes Docker

● Simple, portable, extensible modular

framework for container-based service

app management

● Patterned after internal systems in Google that

manage internet-scale workloads

● Written in Go

github.com/GoogleCloudPlatform/kubernetes

Kubernetes glossary (part 1)

● Master: the managing machine, which oversees one or more minions.

● Minion: a slave that runs tasks as delegated by the user and Kubernetes master.

● Pod: an application that runs on a minion.

Kubernetes glossary (part 2)● Replication controller: Verifies everything

that should be running is running.● Label: an arbitrary key/value pair that the

Replication Controller uses for service discovery

● kubecfg: the config tool● Service: an amalgamation of similar pods

https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/master/docs/architecture.png

Kubernetesoverview

Connection points

● Kubernetes Pod → Mesos TaskGroup● Pod Labels → Mesos TaskGroup Labels, Task Labels● LabelSelectors ↝ Mesos Label Service

● ReplicationController Marathon● Kubernetes Service ↝ Marathon service discovery

Kubernetes-Mesos framework overview

Kubernetes-Mesos master and scheduler

Kubernetes-Mesos executors and tasks

What Mesos contributes● Multi-framework: weighted fair sharing, roles, preemption*, node drain*

● Run pods alongside other popular frameworks

(e.g. Spark, Rails, Hadoop, ...)

● Run services and batch apps in the same cluster

● Advanced scheduling: resources, constraints, global view

● High resource availability, cluster self-healing

● Proven at scale, battle-tested in production

● GUI/CLI cluster management console

(* in development)

What Kubernetes contributes● Pods: tasks to co-locate/deploy/isolate/replicate

● Label Service: arbitrary metadata for tasks/pods

● Tightly integrated service discovery solution

● Ability to run Kubernetes API workloads

+ + +

HardwareFault tolerance

PackagingUpdatesIsolation High availability

Service discoveryService aggregation

ScalabilityElasticity

Multi-tenancyBatch jobsUtilization

Google’s Hardware + Linux

Omega Mesos

Kubernetes

Google-like computing for everybody

Batch

Service Batch Service

GCE

Some possible stack variations

Google’s Hardware + Linux

GCE

Mesos

Marathon

Service

BatchService

Kubernetes

More Possible Stack Variations

Other Hardware or Cloud + Linux or CoreOS

Kubernetes Mesos

Service

Service

Portability => no vendor lock-in

Kubernetes

Mesos

Marathon

Service

Batch

google.mesosphere.com

Summary● Complete stack for large distributed apps

● Multi-tenancy

● Resource optimization

● Easy to deploy

● No vendor lock-in

Useful Links● GCE

○ cloud.google.com/compute

● Kubernetes○ github.com/GoogleCloudPlatform/kubernetes

● Mesosphere○ mesosphere.com

● Kubernetes - Mesos○ github.com/mesosphere/kubernetes-mesos

● Docker○ docker.com

Photo Credits● https://secure.flickr.com/photos/icco/7990117774/ ● https://secure.flickr.com/photos/icco/9956577145/ ● http://www.google.co.uk/about/datacenters/gallery/#/all/20● http://www.google.co.uk/about/datacenters/gallery/#/all/12 ● https://secure.flickr.com/photos/icco/14349351930/