stackengine demo - docker austin

Managing Docker Applications in Production

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged), Evangelist

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

History - BoydOracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

Working on the first Container Days Austin for late Feb

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Co-Organizer of DevOps Days Austin

Became involved in Docker Austin in August.

Working with Docker for the first Docker Days Austin

Keep an eye on @stackengine to see if we can make it happen!

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

Currently in private beta

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters

Funding announcement October

Currently in private beta

mike@stackengine.com

Common Ground

What is The Goal of Your

Company?

What is The Goal of Your

Company?Make Money!

So … What is DevOps?

Is DevOps a Process?

Is it an intersection of overlapping concerns?

Is DevOps a Culture?

DevOps:

DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

Presentation Prelude

Describe the problems StackEngine sees in using containers in production.

The Ideal Containerized Application

Netflix (unicorn)Service Oriented Architecture

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

BenefitsFeature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

Density - Reduce Costs

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Innovation velocity - Decreased build and test times make more time for trying that next great idea!

Density - Reduce Costs

Separation of Concerns - “Code to an interface” moves to the infrastructure level

Benefits

Taken together a company that can get to the end state has a significant and lasting competitive advantage

Today’s Docker Reality

Concerns

This is still very early days for Docker

Concerns

This is still very early days for Docker

It is even earlier for Rocket

Concerns

This is still very early days for Docker

It is even earlier for Rocket

Tools to support containerized applications are immature or do not exist

Concerns

This is still very early days for Docker

It is even earlier for Rocket

Tools to support containerized applications are immature or do not exist

Service oriented architecture is hard

ConcernsThis is still very early days for Docker

It is even earlier for Rocket

Tools to support containerized applications are immature or do not exist

Service oriented architecture is hard

Increasing developer throughput is only a local optima, thus a false economy

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

Laundry Ops: http://stackengine.com/laundryops-practical-devops-at-home/

SDLC

Developers are moving forward

SDLC

Developers are moving forward

QA sees production modeling, large test grids, more effective integration testing

SDLC

Developers are moving forward

QA sees production modeling, large test grids, more effective integration testing

Build engineers see parallelized build grids for faster feedback and better build quality

xkcd … er … SDLCOperations panics - no tools or instrumentation

SysAdminDon’t want this guy panicking

Information Architecture

Problem Description

Information Architecture

Problem Description

StackEngine’s work on that problem today

Information Architecture

Problem Description

StackEngine’s work on that problem today

StackEngine’s planned work on that problem for the future

Problem #1 The Final Mile

The Final Mile

Developer uses containers

The Final Mile

Developer uses containers

System is built and images are produced

The Final Mile

Developer uses containers

System is built and images are produced

Images stored in a private registry

The Final Mile

Developer uses containers

System is built and images are produced

Images stored in a private registry

Now we need to deploy to a QA environment

The Final Mile

So we build a manual solution depending on human behavior

The Final Mile

So we build a manual solution depending on human behavior

Someone builds a tool

Crossing the Final Mile - Today

A Simple Solution to Deployment

Deploy My Stuff - TodayCreate a component to represent your registry

Deploy My Stuff - TodayCreate a component to represent the image you would like to run

Deploy My Stuff - TodayDistribute that image and run it on the hosts in your Mesh.

Deploy My Stuff - TodayMagic happens.

Deploy My Stuff - TodayLife is good

Crossing the Final Mile - Future

Heterogeneous, Policy Driven Deployment

Deploy My Stuff - Future

Support other container types

Deploy My Stuff - Future

Support other container types

Policy Based Deployment

Deploy My Stuff - FutureWe should put another bunny cloud over there …

Problem #2 Container Sprawl

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could completely circumvent ops

Container SprawlFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Container SprawlFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Rocket potentially segments the Market and means different performance, security, runtime and management characteristics

Container Sprawl

Containers, Dockerfiles, and Images on VMs

Container Sprawl

Containers, Dockerfiles, and Images on VMs

VMs on physical hardware located in availability zones

Container Sprawl

Containers, Dockerfiles, and Images on VMs

VMs on physical hardware hardware located in availability zones

AZs in data centers in geographic regions.

My Head HurtsContainer Sprawl

Artifact Inventory - Today

A drillable and searchable view of infrastructure

Where’s My Stuff - TodayHosts

Where’s My Stuff - TodayImages

Where’s My Stuff - TodayContainers

Where’s My Stuff - TodayFilters and Build Actions

Where’s My Stuff - TodaySearch

Artifact Inventory - Future

Geographically aware physical topology

Where’s My Stuff - Future

Geographically drillable physical topologies

Where’s My Stuff - Future

Geographically drillable physical topologies

Policy based inventory levels (pull not push)

Where’s My Stuff - FutureHey, why are there two purple stripes?

Problem #3 Telemetry

Telemetry

What does monitoring a containerized app look like?

Telemetry

What does monitoring a containerized app look like?

http://goo.gl/VkAQo4

Telemetry

What does monitoring a containerized app look like?

http://goo.gl/VkAQo4

What does log aggregation look like?

Telemetry

What does monitoring a containerized app look like?

http://goo.gl/VkAQo4

What does log aggregation look like?

http://goo.gl/uNPe9P

Decision Dimensions - TodayHost level metrics to help you manage

running containers.

What’s my stuff doing - Today

I have this much stuff.

What’s my stuff doing - Today

My hosts are not yet stuffed.

What’s my stuff doing - Today

The stuffing in a host is as expected.

What’s my stuff doing - Today

A bit of the stuffing is doing what it should.

What’s my stuff doing - Today

What is the stuff in this image?

What’s my stuff doing - Today

How do I visualize the stuff in an image?

Decision Dimensions - Future

There is never too much information about run time.

What’s My Stuff Doing - Future

Improve Host Metrics

What’s My Stuff Doing - Future

Improve Host Metrics

Improve Image Metrics

What’s My Stuff Doing - Future

Improve Host Metrics

Improve Image Metrics

Add Container Metrics

What’s My Stuff Doing - Future

Ahh … look … the bunny on the left is happy!

Problem #4 Centralized Control Plane

Centralized Control Plane

The ability to act upon a system from a single point of entry

Centralized Control Plane

The ability to act upon a system from a single point of entry

Actions should happen next to the information causing the action

Centralized Control Plane

The ability to act upon a system from a single point of entry

Actions should happen next to the information causing the action

Actions should happen within the context of inventory

Centralized Control Plane

The ability to act upon a system from a single point of entry

Actions should happen next to the information causing the action

Actions should happen within the context of inventory

UI, API, CLI

Actions - TodayPreventative, automatic, deterministic

Make Stuff do Stuff - Today

Manage Images

Make Stuff do Stuff - Today

Manage Containers

Actions - FutureInitialization, Taxonomy, Corrective

Make Stuff do Stuff - Future

Policy based actions by the Controller

Make Stuff do Stuff - Future

Policy based actions by the Controller

Powerful Control Plane Taxonomy (Tagging)

Make Stuff do Stuff - Future

Policy based actions by the Controller

Powerful Control Plane Taxonomy (Tagging)

Ants not Cattle

Make Stuff do Stuff - Future

Blow really hard and make the bunnies kiss.

Problem #5 Heterogeneous Service

Providers

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Images come from different places (Docker Hub, Quay.io)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)

Images come from different places (Docker Hub, Quay.io)

Within an IaaS service provider are other services (Databases, load balancers)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)

Images come from different places (Docker Hub, Quay.io)

Within an IaaS service provider are other services (Databases, load balancers)

Persistent Data stores present their own unique problem as a service

Components - Today

Abstractions for Control Plane Operation

Building Apps with My StuffComponents are infrastructure service abstractions

Building Apps with My StuffComponents can be arranged in logical ways to represent

physical infrastructure.

Building Apps with My Stuff

Start the db

Building Apps with My Stuff

Start the web server

Building Apps with My Stuff

Run the application server

Building Apps with My Stuff

Run the Cache

Components - Future

More and varied components

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Pluggable component architecture

Building Apps with My StuffWhat else could we make with bunnies and rainbows?

Roadmap

Problem #6 Service Discovery

Service Discovery

Allows multiple deployments of the same service on the same host.

Service Discovery

Allows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

Service Discovery

Allows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

We favor something like consul with a search interface like the chef server.

Service DiscoveryAllows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

We favor something like consul with a search interface like the chef server.

A step towards cattle containers instead of pet containers

Service DiscoveryAllows multiple deployments of the same service on the same host.

Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

We favor something like consul with a search interface like the chef server.

A step towards cattle containers instead of pet containers

Ultimate goal is “ants not cattle”

Roadmap

User Management

Roadmap

User Management

Addition of new Components

Roadmap

User Management

Addition of new Components

Better telemetry

Roadmap

User Management

Addition of new Components

Better telemetry

Scheduling

Roadmap

User Management

Addition of new Components

Better telemetry

Scheduling

A/B Deployment support

StackEngine is the StuffIt’s easier to dream with StackEngine.

Live Product DemoAl Gore and his gremlins willing

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

Thank You for Your Time and Comments.