monitoring docker environments with outlyer
TRANSCRIPT
Monitoring, done differently.
Monitoring Docker Environments with Outlyer
Todd RadelDirector, Customer Success
About Me
2
● 20 years’ software development experience
● Java, .NET, Python, Perl, C++
● APM industry veteran4 years at AppDynamics
tradel
http://lnked.in/tradel
@tradel
Agenda
• How Outlyer works with Docker
• Installing Outlyer agents on Docker
• How to use Host View to visualize your environment
• How plugins and packs work with containers
• Open house for Q&A
3
How Outlyer works with Docker
4
“Docker is hard.”
5
Colin HemmingsCTO, Outlyer
Challenges in the Docker world
• Many versions of Docker, Kubernetes, Swarm, Rancher, etc.
• Breaking API changes between versions
• Evolving and releasing rapidly
• Networking model causes gaps in visibility
6
Comparing Outlyer to other solutions
Basic metrics only (CPU/memory)
Over 500 metrics from base plugin (CPU, memory, network, I/O, load average, etc)
Difficult installation Installation with single command
No service monitoring Most plugins work in Docker
7
A note about labels and tags
• Docker uses “labels” to attach metadata to containers
• Kubernetes/Swarm/Rancher add many of their own labels
• Outlyer agent automatically imports Docker labels as Outlyer tags
(but only label names, not values (yet))
8
Outlyer
TAGS
Docker
LABELS
Installing Outlyer Agent on Docker
9
The “hard” way
DATALOOP_AGENT_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
DATALOOP_NAME=docker_container_name
docker run -d --net host \
-e "DATALOOP_AGENT_KEY=${DATALOOP_AGENT_KEY}" \
-e "DATALOOP_NAME=${DATALOOP_NAME}" \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc:/rootfs/proc:ro \
-v /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \
outlyer/agent:latest
10
The hard way
DATALOOP_AGENT_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
DATALOOP_NAME=docker_container_name
docker run -d --net host \
-e "DATALOOP_AGENT_KEY=${DATALOOP_AGENT_KEY}" \
-e "DATALOOP_NAME=${DATALOOP_NAME}" \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc:/rootfs/proc:ro \
-v /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro \
outlyer/agent:latest
11
Needed to allow the agent to
gather host metrics
Automatically uses the latest
version - or check tags at
https://hub.docker.com/r/outlyer
/agent/ for specific versions
Using Docker Compose
wget https://rawgit.com/outlyerapp/docker-alpine/master/agent/docker-compose.yaml
vim docker-compose.yaml
docker-compose -f docker-compose.yaml up
12
Using Docker Compose
version: '2'
services:
dataloop-docker:
image: outlyer/agent:latest
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock:rw
- /sys/fs/cgroup:/rootfs/sys/fs/cgroup:ro
- /var/lib/docker/:/var/lib/docker:ro
- /proc:/rootfs/proc:ro
environment:
- DATALOOP_NAME=outlyer-agent
- DATALOOP_AGENT_KEY=<your dataloop agent key>
13
Kubernetes Deployment
wget https://rawgit.com/outlyerapp/docker-alpine/master/agent/kubernetes_daemonset.yaml
kubectl apply -f kubernetes_daemonset.yaml
• Deploys one container on every host in cluster (masters and workers)
• Lights up your entire cluster with a single command
14
Swarm and Rancher Deployment
• Swarm does not currently support host networking
Use manual method for now
• Deploy on Rancher through Rancher UI
Config files will be available soon
15
Using Host View to visualize your cluster
16
What is Host View?
• Displays health of every container
and host
• Sort, group, and filter by tags
or Docker labels
• Visualize health by
CPU/memory/presence
• Show links between containers
and hosts
• Container status at a glance
17
How plugins and packs work in Docker
18
The problem
• Most plugins (and Nagios scripts) want to access “localhost” for ports,
files, etc.
HOST = 'localhost'
PORT = 27017
• But the agent runs in a different container from the app you want to
monitor
19
The magic solution
• Docker compatibility layer
• Transparently intercepts common API calls to localhost
• Replaced with calls to target container
20
The magic solution
Most plugins and packs work the same inside and outside of Docker!
21
Adapting your own plugins
• See if the client library is already covered by plugin-helper.(Requests, MongoClient, Psycopg2, Socket, Subprocess, others)
• If so: no code changes necessary!
• If not: use the plugin-helper to find the target container.
22
Using the plugin helper library
try:
from outlyer.plugin_helper.container import is_container, get_container_ip
if is_container():
container_ip = get_container_ip()
except ImportError:
print(“Plugin helper library is not available!”)
print(“Please upgrade your Outlyer agent to the latest version.”)
23
Coming Soon!
24
New features in Fall 2017 release
• Full support for key/value labels
• Host View becomes the default view
• “Scoped” dashboards with label selectors
25
Questions & Answers
26
Resources
27
Resources
• Monitoring Docker environments with Outlyer
(product documentation)
• Plugin Helper library
(for adapting your plugins to Docker)
• Support Slack
(sign up and ask questions)
28
Thank You!
29