core os dna_automacon

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Inserting CoreOS DNA for Creating Docker Clusters.

Advanced Technology GroupSeptember 16th, 2015


Inserting CoreOS DNA for Creating Docker Clusters.

Why? What’s the goal of this presentation?


About the speaker● Patrick Galbraith

● HP Advanced Technology Group

● Has worked at Blue Gecko, MySQL AB, Classmates,

Slashdot, Cobalt Group, US Navy, K-mart

● MySQL projects: memcached UDFs, DBD::mysql,

Ansible HP switch drivers

● federated storage engine

● Family

● Outdoors

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

Enterprise workloads are migrating towards

Docker

Server

Host OS

Docker

bins+libs

Workload

B

Workload

Abins+libs

Docker

Server

Host OS

Hypervisor

Guest OSGuest OS

bins+libs

Workload

B

Workload

Abins+libs

Virtual

Machine

Docker is Great

• Lightweight “container” technology.

• Intelligent Packaging – Docker Images

• Intelligent Deployment – Docker

Containers

• Rapidly evolving ecosystem.

• Linux IS the API.

Docker has some gaps

• Scalability: Docker is a host application.

• Not Multi-Tenant.

• No comprehensive Host-to-Host

networking.

• Host OS maintenance is not included.

• No workload (Docker Container)

distribution.


Clustering Docker – What’s Important

CoordinationConfiguration + discovery for the base cluster, and

applications.

Deployment Deploy a Docker Image to some node in the cluster.

Scheduler Place containers efficiently on to the cluster.

Network Inter-Host networking is obscured in the default Docker model.The default model uses an internal only bridge.

MaintenanceInstall & update the base system in a scalable and effective

way.Note: Docker provides its own system: Images & Containers.


Existing Approaches to Clustering Docker

• Coreos

• Kubernetes

• Swarm

• Docker Machine

• Project Atomic

• Apache Mesos

• RancherOS


CoreOS DNA

The Clustered

Docker Proof of

Concept


Use Cases:

• In Cloud

• Moonshot

• Bare metal

Single Node – CoreOS DNA

Linux Kernel

etcd

fleetd

Base System

CoreOS DNA Node

dockerd

Docker Containers

systemd

App Container

1

App Container

…

Port: 4001

Fleetctl interacts with

fleetd by directly

changing values in

etcd.

etcdctl

Client(s)

fleetctl


CoreOS DNA Cluster

Cluster Configuration – CoreOS DNA

CoreOS DNA NodeDocker

Containers etcd

fleetddockerd

systemd


Containers etcd

fleetddockerd

systemd


Containers etcd

fleetddockerd

systemd…

etcd Discovery

Server

etcd

http://discovery.etcd.io

Docker

Registry

Images

Discovery

Control Node

(jump box)

etcdctl

fleetctlssh

1

2

3

1. Cluster Start

(etcd

discovery)2. Container Start

( fleetctl )

3. Docker

Download


Networking as deployed – CoreOS DNA

eth

0Linux Kernel

etcd

fleetd

CoreOS DNA Node

dockerd

Docker Containers

Container

Container

Container

Container

docker0

172.x.x.x

iptables

port

mapping

Netfilter

CoreOS

DNA Node

CoreOS

DNA Node

CoreOS

DNA Node

Neutron Router

10.x.x.

x

Public

Internet15.x.x.

xNA

T

172.x.x.x - Docker Internal

10.x.x.x - Host (private) Network

15.x.x.x - Public (NAT’ed)

Addresses


ELK Stack + Sinatra Worker Agents (ELK - Elasticsearch, Logstash, Kibana)

Example Application – CoreOS DNA

CoreOS DNA Cluster


Containers etcd

fleetddockerd

systemd


Containers etcd

fleetddockerd

systemd


Containers etcd

fleetddockerd

systemd…

Agent “@5001”

Agent “@5003”

Agent “@5002”

Agent “@5000”Nginx Logstash

Elasticsearch

+ Kabanna Docker

Images

Fleet Mapped

(scheduled into)

Docker

Containers


Building the POC


Building the cluster Using Ansible

Ansible Modules

• Nova_compute – to launch instances

• Nova_facts – used to build inventory files for launched instances

• Docker and docker_facts – used to run containers outside of fleet (testing) and

verification

• Docker_pull – pre-pull images on instances for faster launch by systemd (via fleet)

Using Ansible to provision etcd and build clusters

• query discovery URL

• write URL to local file ad set as a variable

• render etcd service file with the variable

• Build, configure, and run etcd

• Build, configure and run Fleet


The Special Sauce

Etcd

• Integral to cluster functioning.

• Fleet communicates with etcd to obtain key/values from etcd.

• Etcd also used by the sample ELK app to store key/value pairs used by confd to render

config files upon running containers (boot).

Confd

• Stored in each Docker container.

• Keeps an eye on files rendered.

• Can use etcd key/value pairs to interpolate what it rendered.

• Automatically keeps config files up to date with etcd information.


Sample App unit files

Systemd unit files

• Unit file directives

– ExecStartPre – pull image

– ExecStart – run container

– ExecStartPost– set IP of the container in etcd

– ExecStopPost – remove IP in etcd upon container stop

• ElasticSearch – sets its own public and private IPs in etcd for discovery by logstash

• Logstash – sets its own IP in etcd for discovery by logstash agents

• Sinatra app – sets title of app in etcd as well as IP addresses for discovery by nginx to

generate nginx conf using confd


Sample App container Dockerfiles

Docker file functionality

• ElasticSearch – install confd, install and configure elasticsearch, install kopf and kibana

plugins,expose port 9200, launch

• Logstash – Install confd, Install and configure logstash, run boot script

• Sinatra – Install sinatra, confd, place logstash agent, expose port 5000, run boot.sh

• Nginx – Install nginx, confd, run boot.sh


Sample App container CMD scripts

Boot Script

• Logstash – render logstash config (confd –onetime), generate SSL private key and cert,

stores in etcd, then run logstash

• Sinatra – render app.rb, SSL cert and keys, logstash forwarder config, start logstash

forwarder, start sinatra app (foreman)

• Nginx – render nginx.conf (-onetime) and start confd to check and update conf every 10

minutes, start nginx, tail nginx logs


Confd – resource file (nginx)

[template]

keys = [ "app/server", "elasticsearch/host" ]

owner = "nginx"

mode = "0644"

src = "nginx.conf.tmpl"

dest = "/etc/nginx/sites-enabled/docker_dns.conf"

check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"

reload_cmd = "/usr/sbin/service nginx reload"


Confd – template (nginx)

upstream app {

{{ range $server := .app_server }}

server {{ $server.Value }};

{{ end }}

}

…

upstream elasticsearch {

server {{ .elasticsearch_host }}:9200;

keepalive 15;

}


Looking at etcdubuntu@dod-01:~$ etcdctl ls --recursive

/elasticsearch

/elasticsearch/host

/elasticsearch/hostpublic

/logstash

/logstash/ssl_certificate

/logstash/ssl_private_key

/logstash/host

/app

/app/title

/app/server

/app/server/5000

/app/server/5001

/app/server/5002

/app/server/5003

ubuntu@dod-01:~$ etcdctl get /app/server/5000

10.0.0.58:5000


The CoreOS DNA Cluster + ELK Stack


ELK Stack + Sinatra Worker Agents (ELK - Elasticsearch, Logstash, Kibana)

Example Application – Application Architecture

HTT

P

Logstash

Service

HTT

P

Nginx

Port: 80

Key

Docker

Container

HTTP

Logstas

hPublic

Internet

Worker “@5001”

Worker “@5002”

Worker “@5003”

…

Worker

“@5000”Sinatra Service:

“Hello World”Logstash

Agent

H

T

T

P

etcd

/logstash

/logstash/host

…

/app/server/5000

/app/server/5001

…

/elasticsearch/host

…

etcd(federated)

Elasticsearc

hKibana

KopfPlugins:


Demohttps://youtu.be/pRtQ0AXYe6M


Questions, Comments & Feedback?

Dod Ansible Repository https://github.com/HPATG/DeCore

Sample app code https://github.com/HPATG/sample_a

pp

Marcel De Graaf’s blog post http://marceldegraaf.net/2014/05/05/

coreos-follow-up-sinatra-logstash-

elasticsearch-kibana.html


Thank You

Advanced Technology Group

Eric Gustafson [email protected]

Yazz Atlas [email protected]

Patrick Galbraith [email protected]

Special Thanks

Marcel De Graaf http://marceldegraaf.net/

Kelsey Hightower https://github.com/kelseyhightower

core os dna_automacon

Software