Multi-Cluster Kubernetes:Planning for Unknowns

Dan Wilson, Principal Architect, Concur

Agenda

• Why k8s?• Why CoreOS?• K8s @ Concur• Q & A

What is Concur?

6500+ employees

Offices:AustraliaChinaCzech RepublicFranceGermanyHolland

India JapanPhilippinesSingaporeSwedenUKUS

HQ: Bellevue

70m transactions50b+ USD T&E spend annually

€59mNew cloud bookingsQ3 2015

Cloud DNA 99

Established1993 > 60%

Fortune 500

2011 2012 2013 2014 2015 2016 20182017

Expense transactions

Bookings growth

2013 2015

55million

77+million

Provisioned users

35+million2015

Registered mobile users

2013 2015

2.1million

5+million

Concur growth

A little about me

Principal Architect• Concur since 1998 in various roles (DBA, Ops Manager

& Architect, etc.)• Corking w/ k8s since mid 2015• contributor to k8s and participant in federation & service

catalog SIGs• Email danwilson on github, k8s slack & gmail

Why Kubernetes?

• Largest community support

• Gifted to Linux Foundation

• Protection from shifts in container technology

• Pluggable API

• Capable of isolation between namespaces

• Declarative syntax for the complete make up of services

• Built in logging, service discovery, etc.

• Networking design eliminates port conflicts of containers

Why CoreOS?

• Simple updates by channel• Designed to run as a cluster• Low overhead• Best documentation for k8s

Why CoreOS?

• Simple updates by channel• Designed to run as a cluster• Low overhead• Best documentation for k8s

CoreOS

High Availability

• All things fail, yes everything!• K8s HA guide: http://kubernetes.io/docs/admin/high-availability/

• cluster etcd – 3, 5 or 7 nodes• replicated + load balanced api servers• many minions

High Availability

If cluster == alwaysOnline { glog.V(1). Infof(“nothing to do here!”) }

staging prod

N clustersCI

K8s @ Concur

Location US EUROPE

Zone A B C A B C

Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c

K8s @ Concur

Elastic (logging)

Prometheus (monitoring)Load Balancers

etcd Nodes

etcd Nodes

K8s @ Concur

K8s @ Concur• kube2cnqr

• Golang docker container• Watches the k8s API for updates to services and worker nodes• Calls a Concur internal API to add\remove load balancer entries as

needed• Handles connecting multiple k8s clusters to a single load balancer

endpoint• Extras

• External-IP set on service objects• Annotation w/ DNS name set on service objects• Handles multiple load balancer tiers• Allow services to specify DNS name

Sample project

• https://github.com/concur/kubegowatcher• A sample golang program that leverages kubernetes watch endpoints

using the client-go library.• Designed to run as a container on k8s• example of setting annotation on the service• example of checking event delay• Apache 2.0 license• Add your own business logic• Contribute back improvements to make it better

Location US EUROPE

Zone A B C A B C

Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c

K8s @ Concur – k8s upgrades

Location US EUROPE

Zone A B C A B C

Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c

Cluster v.1.3 us13a us13b us13c eur13a eur13b eur13c

K8s @ Concur – k8s upgrades

K8s @ Concur

Deployment Challenges• Stale kubectl scripts• Stale kubectl config• Handling credentials for each ci system• Many commands for each cluster• Handling proxy

K8s @ Concur

Deployment API

CIDeploy petshop:v1 to test

test14a, b & c

K8s API calls for service & deployment

Deployment API

CIDeploy petshop:v1 to prod

US14a, b & c

EMEA14a, b & c

APAC14a, b & c

K8s API calls for service & deployment

Config API

Add/remove cluster

EMEA14a, b & c

Setup API

curl /setup

Output kubectl config commands for all cluster zones requested

| bash

K8s @ Concur

K8s @ Concur

Add features to kubernetes federation• v1.6 – deploying to clusters based on

label #29887

Other potentials…• Location specific overrides• Global overrides• Feature limitations (for example, don’t

allow pods/rc’s without a deployment)

Docs: https://github.com/concur/skipper

K8s @ Concur

Q & A

Email danwilson on github, k8s slack & gmail