the cloud is my laboratory, adam friedman

The cloud is my laboratory Experimenting with Docker

Adam Friedman Dimension Data (ITaaS R&D) [email protected]

What this talk is about»

What this talk is about» Experimenting with Docker

What this talk is about» Experimenting with Docker (der)


» Using virtualisation to reduce cycle time when trying out ideas



» Using the cloud to achieve scale



» Using the cloud to achieve scale

» Without breaking the bank

What I'd like to leave you with1.

What I'd like to leave you with1. We learn more from our failures than our successes


2. Occam's razor


2. Occam's razor (prefer to minimise variables)


2. Occam's razor

» Aim for repeatability where possible


2. Occam's razor

3. Virtualisation + automation make it cheap to experiment


2. Occam's razor


4. If experimentation is cheap enough, you can try out almost any idea you can come up with


2. Occam's razor



» Think it's a silly idea? Prove it.


2. Occam's razor



» Don’t be afraid to experiment!

Why I love Virtualisation Virtualisation is the bedrock on which most things cloud are built.


It is the key to: »


It is the key to: » Economies of scale


It is the key to: » Economies of scale » Speed of provisioning


It is the key to: » Economies of scale » Speed of provisioning » Repeatability


It is the key to: » Economies of scale » Speed of provisioning » Repeatability

» The ability to readily and reliably duplicate an environment and its resources

Why I love the concept of CloudFor all its potential downsides, the cloud has vastly improved the process of experimentation:


» By selectively abstracting away levels of infrastructure, it enables us to focus on what we're trying to do


» By selectively abstracting away levels of infrastructure, it enables us to focus on what we're trying to do, rather than how to do it



» Economies of scale also come into play:




» You can’t fit 20 servers under your desk




» You can’t fit 20 servers under your desk, but your cloud provider has plenty to spare (for as long as you need them)




» Most providers now charge only for what you actually use



» Economies of scale also come into play

» Lower costs make it easier to explore a problem space, which helps to mitigate risk

Experimenting in the cloudThe trick to being productive when experimenting in the cloud:

» Get set up


» Get set up

» Do your experiment


» Get set up


» Clean up the evidence

Without breaking the bankThe trick to being productive when experimenting in the cloud:

» Get set up


» Clean up the evidence

» Before the mob arrives with pitchforks and flaming torches

Docker is great for experimenting

with new technologies

If you've heard of it, there's probably a Docker image of it.


Example: What’s Neo4J?


Example: What’s Neo4J?

docker run -p 7474:7474 neo4j


Ok, how about RabbitMQ?


Ok, how about RabbitMQ?

docker run -p 5672:5672 rabbitmq


Ok, smartarse


Ok, smartarse, but I bet there isn't one for cowsay


Ok, smartarse, but I bet there isn't one for cowsay

docker run mwendler/cowsay "Sorry."

________< Sorry. > -------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

That's super useful, but what if you're experimenting with docker itself?


docker-machine create docker1eval $(docker-machine env docker1)


docker-machine create docker1eval $(docker-machine env docker1)

Ok, but not very cloudy.


How about:


How about:

docker-machine create --driver amazonec2 docker1

docker-machine create --driver amazonec2 docker2


Or:


Or:

docker-machine create --driver digitalocean docker1

docker-machine create --driver digitalocean docker2

Docker Machine is nice

Docker Machine is nice, but it's opaque


Sometimes that’s what you want.


Sometimes that’s what you want (again, the focus may be on what you're trying to do).


Sometimes that’s what you want (again, the focus may be on what you're trying to do, not how).



Other times, not so much.



Other times, not so much.

So what other options are there?

Do it by hand

You could do this at least once - a useful learning experience (especially for when things go wrong).

Automate it!

I'm probably preaching to the converted here :)

Infrastructure with Terraform

Declarative configuration for infrastructure

» Multi-cloud / multi-provider

» Repeatable

» Easy to consistently create / destroy / re-create infrastructure

But Terraform is only half the story. Once the infrastructure has been created, how do you get Docker itself deployed and configured?

Software with Ansible

» Hardly the only option out there, but if you're experimenting with throw-away systems then it's a good choice because it's just SSH (no master / agents to deploy)

» Chef Solo (or Chef Zero) is an alternative, but may still require more work to bootstrap each node

» Quick to get started (just run commands or modules on target machines)

» Scales up to Roles + Playbooks for repeatability

» Not great for managing large numbers of machines

Software with Ansible

» Can be run from Terraform (via a plug-in provisioner) but it's a lot easier to run it separately while you're experimenting.

» There are Ansible inventory plugins that can read a Terraform state file (so it knows server roles, host names, IP addresses, etc).

» Ansible Galaxy has modules for everything, including Docker (hint, hint)

» Beware if you are deploying an OS with only Python 3.x (Ansible needs 2.x)

A quick detour: Docker from simple to complex» Stand-alone

» Docker

» Orchestrated

» Local

» Docker Compose

» Clustered

» Docker Swarm

» Clustered with GUI

» Cattle / Rancher

» Kubernetes

» Mesos / Marathon

» Kitchen Sink

» Mantl

Note that simple and easy-to-use are not the same thing :)

Cisco Mantl

» Mantl is a microservices platform based on Docker, Mesos / Marathon, Consul, Traefik, Kubernetes, Calico, Contiv, etc (pretty much everything but the kitchen sink).

» Because it already includes several popular systems that extend / orchestrate Docker, it’s a useful starting point if you’re stuck trying to work out how to deploy or integrate a particular component

Cisco Mantl

» Mantl is deployed using Terraform with Ansible, and can therefore be deployed on a variety of clouds by swapping out Terraform modules as required.

» Their Ansible inventory plugin understands which cloud provider was used to create the infrastructure and so Ansible playbooks can adjust their behaviour to suit (if required).

Build your own lab

Build your own (throw-away) lab

Build your own (throw-away) lab

» Digital Ocean is fast Add Terraform and you can repeatedly create and destroy environments in seconds, rather than minutes.

» Obviously you can use AWS / Azure, too (or Dimension Data Cloud Control) but for quick-and-dirty experiments I prefer the simplest thing that works).

» Create an Ansible playbook to install Docker

» Start small - put everything in a single playbook and then move stuff out to separate reusable roles when needed

Build your own lab

4 servers (1 master, 3 workers).

lab.tf:

resource "digitalocean_droplet" "master" {

count = 1

image = "ubuntu-14-04-x64"

name = "master-${count.index + 1}"

region = "nyc2"

size = "1024mb"

}

resource "digitalocean_droplet" "worker" {

count = 4

image = "ubuntu-14-04-x64"

name = "worker-${count.index + 1}"

region = "nyc2"

size = "1024mb"

}

Build your own lab

» terraform apply

Build your own lab

» terraform apply

» …

Build your own lab

» terraform apply

» …

» Profit!

Build your own lab

» terraform apply

» …

» Profit!

» terraform destroy

Build your own lab

» terraform apply

» …

» Profit!


» No pitchforks

Build your own lab

» terraform apply

» …

» Profit!


» No pitchforks (probably)

Why not use Rancher / Docker Cloud?

» If all you want to do is deploy containers then these are a much better choice



» Rancher uses docker-machine to create nodes for you




» But this is a good choice if you want to learn about:





» How to deploy docker






» Non-standard docker configurations






» Non-standard docker configurations

» It's useful to have a feeling for what's behind the UI

Links

» github.com/tintoy

» github.com/DimensionDataResearch

» mantl.io

» terraform.io

» ansible.com

» github.com/angstwad/docker.ubuntu

http://github.com/tintoy

http://github.com/DimensionDataResearch

http://mantl.io

http://terraform.io

http://ansible.com

http://github.com/angstwad/docker.ubuntu

Questions?

the cloud is my laboratory, adam friedman

Technology