© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

November 12th, 2014 | Las Vegas

Monitoring and Running Docker Containers at Scale Alexis Lê-Quôc, Datadog

@alq — CTO at Datadog

Datadog

•  Monitoring service •  Made for the cloud •  Aggregates everything •  Support for Docker

(since 1.0)

Goals 1. Present key Docker metrics 2. Explain operational complexity 3. Rethink monitoring of Docker containers

Agenda •  A (very) brief history of containers •  Docker containers on AWS •  Key Docker metrics •  Operational complexity •  Monitoring Docker effectively

•  Demo

A brief history of containers

Containers in a nutshell •  Been around for a long time

–  jails, zones, cgroups •  No full-virtualization overhead •  Used for runtime isolation (e.g. jails) •  Docker: escape from dependency hell

Escape from dependency hell a.out

shared libs

packages

omnibus

Docker ~

Container ~ single static binary Process Container Host

Source Dockerfile Chef/Puppet Kickstart

.TEXT /var/lib/docker Full distro

PID Name/ID Hostname

Docker on AWS: some numbers

(Some) Docker use cases •  Continous integration

–  eliminate dependency variance –  same code from dev laptop to production –  git-like workflow

•  Continuous delivery –  (quasi) stateless components –  web workers, video encoders, etc. –  not for data stores (Amazon RDS a better fit)

Instance types

20% 20% 19%

13%

8%

21%

c3.2xl m3.medium m3.large m3.xlarge m1.large the rest

Source: Datadog, October 2014

Containers per instance •  Average: 5 (October 2014) •  Highly dependent on the workload •  This is just the beginning… •  Expect higher container density going forward

Source: Datadog, October 2014

Key Docker metrics

Monitoring fundamentals Work

Resource consumption

Measures the amount of value created

Measures the amount of resources consumed to create value

What your customers care about What your customers don’t care about

Database: queries answered Web server: requests served Queue: wait time distribution

Database: I/O throughput Web server: active connections OS: CPU utilization Container: memory footprint

Docker containers consume… •  Memory •  CPU •  I/O •  Network

Memory Name Why it matters

pgmajfault Paging to/from disk is slow

pgfault Context switches hurt application performance

resident set size (rss) Too much RSS causes paging and swapping

swap Swapping in/out is slow

CPU Name Why it matters

user Measures work being done

system System calls, a necessary evil

Block I/O Name Why it matters

blkio.io_service_bytes I/O is (often) bottleneck

blkio.io_queued Measures saturation

Network Name Why it matters

tx/rx_errors Because… errors are bad.

tx/rx_dropped Measures contention

tx/rx_bytes Measures traffic

How to collect metrics •  https://github.com/google/cadvisor

Operational complexity

Combinatorial multiplication

Hardware

OS

Off-the-shelf

Your Application

Hardware

Hypervisor

Off-the-shelf

App

OS OS

Off-the-shelf

App

Hardware

Hypervisor

OS OS

A A A A

Containers

O O O O

Operational complexity •  Average containers per instance: N (N=5, 10/2014) •  N-times as many “hosts” to manage •  Affects

–  provisioning: prep’ing & building containers –  configuration: passing config to containers –  orchestration: deciding where/when containers run –  monitoring: making sure containers run properly

Monitoring: metric counts on Amazon EC2

•  1 Amazon EC2 instance –  10 CloudWatch metrics

•  1 operating system (e.g. linux) –  100 metrics

•  1 Container –  50 metrics

•  1 off-the-shelf application –  ~50 metrics

Combinatorial multiplication

100 500 instances containers

Assuming only 5 containers per instance

Combinatorial multiplication

160 410 metrics per instance

metrics per instance

Assuming only 5 containers per instance

Velocity

hours, days, months

minutes, hours, days

EC2 instance half-life Container half-life

Aggravating factors •  Hub-based provisioning

–  new images every day •  Autonomic orchestration

–  from imperative to declarative –  automated –  individual containers don’t matter –  e.g. kubernetes, mesos

A lot more, A lot faster.

If your monitoring is still centered on individual hosts or instances…

Host-centric monitoring

Monitor

Monitor

GAP

Hypervisor

OS OS

A A A A

Containers

O O O O

A lot more pain, A lot faster.

Monitoring containers effectively

A new approach to container monitoring

Layers + Tags

Layers of monitoring

Monitor

Hypervisor

OS OS

A A A A

Containers

O O O O

Layers of monitoring

CloudWatch

Infrastructure Monitoring

APM

Hypervisor

OS OS

A A A A

Containers

O O O O

Layers of monitoring

cpu/net/io

filesystem docker mem docker cpu db queries

web requests

app throughput

CloudWatch

Infrastructure Monitoring

APM

e.g.

Hypervisor

OS OS

A A A A

Containers

O O O O

Layers of monitoring •  Access to metrics from all the layers •  Amazon CloudWatch, OS metrics, Docker metrics,

app metrics in 1 place •  Shared timeline

If your monitoring does not cover all layers, pain.

Tags

You use them already

Tags •  Monitoring is like Auto-Scaling Groups •  Monitoring is like Docker orchestration •  From imperative to declarative •  Query-based •  Queries operate on tags

Monitoring with tags and queries

“Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… and make sure resident set size < 1GB on c3.xl”

Monitoring with tags and queries

“Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… and make sure resident set size < 1GB on c3.xl”

Monitoring with tags and queries

“Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… that use more than 1.5x the average on c3.xl”

“Dude, where’s my server?”

“Dude, where’s my container?”

If your monitoring is not tag-based, pain.

Demo

Take-aways 1. Docker increases operational complexity by an order

of magnitude unless… 2. You have layered monitoring, from the instance to

the container and to the application, and… 3. You monitor using tags and queries

Please give us your feedback on this presentation

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Join the conversation on Twitter with #reinvent