osdc 2014: jochen lillich - dynamic infrastructure orchestration

52
freistil.it Dynamic Infrastructure Orchestration Jochen Lillich

Upload: netways

Post on 06-May-2015

408 views

Category:

Software


1 download

DESCRIPTION

Getting Configuration Management in place is a big step in the direction of infrastructure automation. Chef, Puppet and Co. replace error-prone manual changes with periodic system convergence runs controlled by a central database. Even with Puppet’s exported resources and Chef’s search capabilities, the weakness of this approach is that it is rather static. In situations where we need to propagate information quickly, handle failure detection, or tolerate network partitions, other tools might offer better solutions. In this talk, I’m going to present some of these alternatives (e.g. serf, etcd) and how they can be used to allow for more dynamic configuration changes.

TRANSCRIPT

Page 1: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

freistil.it

Dynamic Infrastructure Orchestration

Jochen Lillich

Page 2: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Jochen Lillich

@geewiz

Sysadmin since 1993

CEO of freistil IT

freistilbox: PaaS for Drupal & WordPress

Page 3: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Config Management

Elimination of manual changes

Consistent configuration

Single source of truth

DRY

Page 4: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Weaknesses

Periodic convergence runs

Central database

Page 5: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Orchestration

Perform changes quickly

Handle failure

Tolerate network partitions

Page 6: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

serf

Page 7: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Basics

Cluster communication tool

Developed by Hashicorp

Simple binary, written in Go

Gossip protocol

Page 8: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

serf node 1$ serf agent -node=node1 -bind=srv1

==> Starting Serf agent...

==> Starting Serf agent RPC...

==> Serf agent running!

Node name: 'srv1'

Bind addr: '192.168.0.11:7946'

==> Log data will now stream in as it occurs:

2014/04/09 16:05:51 [INFO] agent: Serf agent starting

2014/04/09 16:05:51 [INFO] serf: EventMemberJoin: node1 192.168.0.11

2014/04/09 16:05:52 [INFO] agent: Received event: member-join

Page 9: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

serf node 2

$ serf agent -node=node2 -bind=srv2

$ serf join srv1

Successfully joined cluster by contacting 1 nodes.

$ serf members

node2 192.168.0.12:7946 alive

node1 192.168.0.11:7946 alive

Page 10: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Membership

Cluster management

Service configuration

Page 11: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Failure detection

Detects failure quickly

Notifies other nodes

Executes handler scripts

Page 12: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Eventsmember-join

member-leave

member-failed

member-update

member-reap

user

query

Page 13: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

User events

Deploy application

Perform convergence run

Query information from the cluster

Page 14: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Example handler

#!/bin/bash

!

echo

echo "New event: ${SERF_EVENT}. Data follows..."

while read line; do

printf "${line}\n"

done

Page 15: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

User event

$ serf event deploy-app

!

2014/04/09 17:06:32 [INFO] agent: Received event: user-event: deploy-app

Page 16: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

etcd

Page 17: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Basics

HA key-value store

Developed by CoreOS

Written in Go

Raft consensus algorithm

Pure HTTP API

Page 18: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Key-value store

/release = 9607bcfeb48905d26db8f

/cluster

/cluster/node1 = node1.example.com

/cluster/node2 = node2.example.com

Page 19: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Quorum

Available

Page 20: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Quorum

Available

Page 21: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Quorum

Available

Page 22: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Quorum

Unavailable

Page 23: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Writing$ etcdctl set /message Hello

Hello

$ curl -L -X PUT \ http://127.0.0.1:4001/v2/keys/message \ -d value=“Test"

{"action":"set","node":{"key":"/message","value":"Test","modifiedIndex":15,"createdIndex":15},"prevNode":{"key":"/message","value":"Hello","modifiedIndex":2,"createdIndex":2}}

Page 24: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Reading

$ etcdctl get /message

Hello

$ curl -L \ http://127.0.0.1:4001/v2/keys/message

{"action":"get","node":{"key":"/message","value":"Test","modifiedIndex":15,"createdIndex":15}

Page 25: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Deleting

$ etcdctl rm /message

$ curl -L -X DELETE \ http://127.0.0.1:4001/v2/keys/message

Page 26: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Waiting

$ curl -L http://127.0.0.1:4001/v2/keys/release\?wait\=true

Page 27: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Waiting

$ curl -L http://127.0.0.1:4001/v2/keys/release\?wait\=true

{"action":"set","node":{"key":"/release","value":"9607bcfe","modifiedIndex":16,"createdIndex":16},"prevNode":{"key":"/release","value":"18512199","modifiedIndex":14,"createdIndex":14}}

Page 28: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Applications

Locking

Leader election

Database master

Elastic IP

Page 29: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

Page 30: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

AppDB?

Page 31: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

AppDB?

Nope.

Page 32: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

AppDB?

Nope.

I’ll wait.

Page 33: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

Page 34: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

Page 35: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB XMaster me!

Page 36: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB XMaster me!

Ok!

Page 37: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB X is master.

Page 38: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB X is master.

Page 39: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB X is master.

Page 40: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

Page 41: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB Y

Page 42: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB Y

Master me!

Page 43: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB Y

Master me!

No, DB X is.

Page 44: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App DB X

DB Y

Page 45: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB Y

Page 46: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB Y

Page 47: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB Y

Page 48: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB Y

You da master!

Page 49: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB YDB Y is master now.

Page 50: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Leader election

etcd

App

DB Y

Page 51: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Summary

Page 52: OSDC 2014: Jochen Lillich - Dynamic infrastructure orchestration

Thanks!

@geewiz

[email protected]