stackengine demo - docker austin

Managing Docker Applications in Production

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged), Evangelist

History - Boyd

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

History - Boyd



Co-Organizer of DevOps Days Austin

History - Boyd




Became involved in Docker Austin in August.

History - BoydOracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)




Working on the first Container Days Austin for late Feb

History - Boyd





Working with Docker for the first Docker Days Austin

Keep an eye on @stackengine to see if we can make it happen!

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters



Funding announcement October




Currently in private beta




Currently in private beta

[email protected]

mailto:[email protected]

Common Ground

What is The Goal of Your

Company?

What is The Goal of Your

Company?Make Money!

So … What is DevOps?

Is DevOps a Process?

Is it an intersection of overlapping concerns?

Is DevOps a Culture?

DevOps:

DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

Presentation Prelude

Describe the problems StackEngine sees in using containers in production.

The Ideal Containerized Application

Netflix (unicorn)Service Oriented Architecture

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Benefits


Innovation velocity - Decreased build and test times make more time for trying that next great idea!

BenefitsFeature velocity - Increase developer productivity using containers to model application topology


Density - Reduce Costs

Benefits



Density - Reduce Costs

Separation of Concerns - “Code to an interface” moves to the infrastructure level

Benefits

Taken together a company that can get to the end state has a significant and lasting competitive advantage

Today’s Docker Reality

Concerns

This is still very early days for Docker

Concerns


It is even earlier for Rocket

Concerns



Tools to support containerized applications are immature or do not exist

Concerns




Service oriented architecture is hard

ConcernsThis is still very early days for Docker



Service oriented architecture is hard

Increasing developer throughput is only a local optima, thus a false economy

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

Laundry Ops: http://stackengine.com/laundryops-practical-devops-at-home/

http://stackengine.com/laundryops-practical-devops-at-home/

SDLC

Developers are moving forward

SDLC


QA sees production modeling, large test grids, more effective integration testing

SDLC


QA sees production modeling, large test grids, more effective integration testing

Build engineers see parallelized build grids for faster feedback and better build quality

xkcd … er … SDLCOperations panics - no tools or instrumentation

SysAdminDon’t want this guy panicking

Information Architecture

Problem Description


Problem Description

StackEngine’s work on that problem today


Problem Description

StackEngine’s work on that problem today

StackEngine’s planned work on that problem for the future

Problem #1 The Final Mile

The Final Mile

Developer uses containers

The Final Mile


System is built and images are produced

The Final Mile



Images stored in a private registry

The Final Mile



Images stored in a private registry

Now we need to deploy to a QA environment

The Final Mile

So we build a manual solution depending on human behavior

The Final Mile

So we build a manual solution depending on human behavior

Someone builds a tool

Crossing the Final Mile - Today

A Simple Solution to Deployment

Deploy My Stuff - TodayCreate a component to represent your registry

Deploy My Stuff - TodayCreate a component to represent the image you would like to run

Deploy My Stuff - TodayDistribute that image and run it on the hosts in your Mesh.

Deploy My Stuff - TodayMagic happens.

Deploy My Stuff - TodayLife is good

Crossing the Final Mile - Future

Heterogeneous, Policy Driven Deployment

Deploy My Stuff - Future

Support other container types

Deploy My Stuff - Future

Support other container types

Policy Based Deployment

Deploy My Stuff - FutureWe should put another bunny cloud over there …

Problem #2 Container Sprawl

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Container Sprawl

First there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could completely circumvent ops

Container SprawlFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Container SprawlFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Rocket potentially segments the Market and means different performance, security, runtime and management characteristics

Container Sprawl

Containers, Dockerfiles, and Images on VMs

Container Sprawl


VMs on physical hardware located in availability zones

Container Sprawl


VMs on physical hardware hardware located in availability zones

AZs in data centers in geographic regions.

My Head HurtsContainer Sprawl

Artifact Inventory - Today

A drillable and searchable view of infrastructure

Where’s My Stuff - TodayHosts

Where’s My Stuff - TodayImages

Where’s My Stuff - TodayContainers

Where’s My Stuff - TodayFilters and Build Actions

Where’s My Stuff - TodaySearch

Artifact Inventory - Future

Geographically aware physical topology

Where’s My Stuff - Future

Geographically drillable physical topologies

Where’s My Stuff - Future

Geographically drillable physical topologies

Policy based inventory levels (pull not push)

Where’s My Stuff - FutureHey, why are there two purple stripes?

Problem #3 Telemetry

Telemetry

What does monitoring a containerized app look like?

Telemetry


http://goo.gl/VkAQo4


Telemetry



What does log aggregation look like?

Telemetry



What does log aggregation look like?

http://goo.gl/uNPe9P

http://goo.gl/uNPe9P

Decision Dimensions - TodayHost level metrics to help you manage

running containers.

What’s my stuff doing - Today

I have this much stuff.


My hosts are not yet stuffed.


The stuffing in a host is as expected.


A bit of the stuffing is doing what it should.


What is the stuff in this image?


How do I visualize the stuff in an image?

Decision Dimensions - Future

There is never too much information about run time.

What’s My Stuff Doing - Future

Improve Host Metrics



Improve Image Metrics



Improve Image Metrics

Add Container Metrics


Ahh … look … the bunny on the left is happy!

Problem #4 Centralized Control Plane

Centralized Control Plane

The ability to act upon a system from a single point of entry



Actions should happen next to the information causing the action




Actions should happen within the context of inventory




Actions should happen within the context of inventory

UI, API, CLI

Actions - TodayPreventative, automatic, deterministic

Make Stuff do Stuff - Today

Manage Images

Make Stuff do Stuff - Today

Manage Containers

Actions - FutureInitialization, Taxonomy, Corrective

Make Stuff do Stuff - Future

Policy based actions by the Controller



Powerful Control Plane Taxonomy (Tagging)



Powerful Control Plane Taxonomy (Tagging)

Ants not Cattle


Blow really hard and make the bunnies kiss.

Problem #5 Heterogeneous Service

Providers

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Images come from different places (Docker Hub, Quay.io)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)


Within an IaaS service provider are other services (Databases, load balancers)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)


Within an IaaS service provider are other services (Databases, load balancers)

Persistent Data stores present their own unique problem as a service

Components - Today

Abstractions for Control Plane Operation

Building Apps with My StuffComponents are infrastructure service abstractions

Building Apps with My StuffComponents can be arranged in logical ways to represent

physical infrastructure.

Building Apps with My Stuff

Start the db


Start the web server


Run the application server


Run the Cache

Components - Future

More and varied components

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Pluggable component architecture

Building Apps with My StuffWhat else could we make with bunnies and rainbows?

Roadmap

Problem #6 Service Discovery

Service Discovery

Allows multiple deployments of the same service on the same host.

Service Discovery


Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

Service Discovery



We favor something like consul with a search interface like the chef server.

Service DiscoveryAllows multiple deployments of the same service on the same host.



A step towards cattle containers instead of pet containers

Service DiscoveryAllows multiple deployments of the same service on the same host.



A step towards cattle containers instead of pet containers

Ultimate goal is “ants not cattle”

Roadmap

User Management

Roadmap

User Management

Addition of new Components

Roadmap

User Management


Better telemetry

Roadmap

User Management


Better telemetry

Scheduling

Roadmap

User Management


Better telemetry

Scheduling

A/B Deployment support

StackEngine is the StuffIt’s easier to dream with StackEngine.

Live Product DemoAl Gore and his gremlins willing

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

Thank You for Your Time and Comments.

stackengine demo - docker austin

Technology