monitoring docker at scale - docker san francisco meetup - august 11, 2015
TRANSCRIPT
![Page 1: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/1.jpg)
Monitoring Docker at Scale
Matt WilliamsEvangelist @ [email protected]@technovangelist
…and having a way to answer every question you have about it
![Page 2: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/2.jpg)
• Docker at a high level• Implementing a Docker-based app • How to monitor at scale• Implementing a monitoring platform
![Page 3: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/3.jpg)
Implementing an app
![Page 4: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/4.jpg)
App architecture
loadbalancer
web
web
![Page 5: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/5.jpg)
Steps to working with Docker
• Create the Docker host• Create a container from an image
• The image could have come from an online repository• Orchestrate a set of containers to replicate your app
![Page 6: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/6.jpg)
docker-machine
• Creates docker hosts to run containers on (must be linux)• Can create hosts on:
• Mac (boot2docker)• VMWare Fusion or VirtualBox
• Windows (boot2docker)• VirtualBox
• AWS• Azure• Digital Ocean
• Google• Openstack (including HP Cloud)• Rackspace• Softlayer• VMWare VCloud and VSphere
![Page 7: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/7.jpg)
docker-machine create -d vmwarefusion fusiondkr
eval "$(docker-machine env fusiondkr)”
docker-machine
![Page 8: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/8.jpg)
docker-machine create -d "openstack" \--openstack-flavor-name "standard.large" \--openstack-image-id "bec3cab5-4722-40b...218e22fe" \--openstack-floatingip-pool "Ext-Net" \--openstack-ssh-user "ubuntu" \hpdocker
eval "$(docker-machine env hpdocker)"
docker-machine
![Page 9: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/9.jpg)
docker-machine
docker-machine create -d amazonec2 \--amazonec2-access-key $AWS_ACCESS_KEY_ID \--amazonec2-secret-key $AWS_SECRET_ACCESS_KEY \--amazonec2-ami $ami \--amazonec2-instance-type $instance_size \--amazonec2-vpc-id $vpc_id \--amazonec2-security-group $security_group \--amazonec2-region $aws_region \<machine name>
![Page 10: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/10.jpg)
Docker Hub
• Collection of public / private repositories of Docker images
![Page 11: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/11.jpg)
docker
• Use command line params or provide Dockerfile• Create containers and images
![Page 12: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/12.jpg)
![Page 13: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/13.jpg)
![Page 14: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/14.jpg)
![Page 15: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/15.jpg)
![Page 16: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/16.jpg)
![Page 17: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/17.jpg)
![Page 18: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/18.jpg)
![Page 19: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/19.jpg)
![Page 20: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/20.jpg)
Web
![Page 21: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/21.jpg)
Web
![Page 22: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/22.jpg)
Load Balancer
![Page 23: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/23.jpg)
docker-compose
• Take a dockercompose yaml file with docker container info• Build docker-based application • Containers linked as needed• Can work locally or against any docker-machine host
![Page 24: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/24.jpg)
![Page 25: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/25.jpg)
![Page 26: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/26.jpg)
![Page 27: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/27.jpg)
scale
docker-compose scale web=20
![Page 28: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/28.jpg)
one more tip…
• If sharing a volume in Docker on top of VirtualBox and using nginx/apache:• sendfile off;
![Page 29: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/29.jpg)
Download the demo
http://dtdg.co/dkrcon
![Page 30: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/30.jpg)
Docker Stats API
![Page 31: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/31.jpg)
docker stats docker stats \
nginxredisdocker_datadog_1 \nginxredisdocker_loadbalancer_1 \nginxredisdocker_registrator_1 \nginxredisdocker_consul_1 \nginxredisdocker_web_1
![Page 32: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/32.jpg)
Remote API{"read" : "2015-‐01-‐08T22:57:31.547920715Z","network" : {"rx_dropped" : 0,"rx_bytes" : 648,"rx_errors" : 0,"tx_packets" : 8,"tx_dropped" : 0,"rx_packets" : 8,"tx_errors" : 0,"tx_bytes" : 648
},"memory_stats" : {"stats" : {"total_pgmajfault" : 0,"cache" : 0,"mapped_file" : 0,"total_inactive_file" : 0,"pgpgout" : 414,"rss" : 6537216,"total_mapped_file" : 0,"writeback" : 0,"unevictable" : 0,"pgpgin" : 477,
"total_unevictable" : 0,"pgmajfault" : 0,"total_rss" : 6537216,"total_rss_huge" : 6291456,"total_writeback" : 0,"total_inactive_anon" : 0,"rss_huge" : 6291456,"hierarchical_memory_limit" : 67108864,"total_pgfault" : 964,"total_active_file" : 0,"active_anon" : 6537216,"total_active_anon" : 6537216,"total_pgpgout" : 414,"total_cache" : 0,"inactive_anon" : 0,"active_file" : 0,"pgfault" : 964,"inactive_file" : 0,"total_pgpgin" : 477
},"max_usage" : 6651904,"usage" : 6537216,"failcnt" : 0,"limit" : 67108864
},"blkio_stats" : {},"cpu_stats" : {"cpu_usage" : {"percpu_usage" : [16970827,1839451,7107380,10571290
],"usage_in_usermode" : 10000000,"total_usage" : 36488948,"usage_in_kernelmode" : 20000000
},"system_cpu_usage" : 20091722000000000,"throttling_data" : {}
}}
![Page 33: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/33.jpg)
Remote API
http --stream –f --verify=no \--cert=$DOCKER_CERT_PATH/cert.pem \--cert-key=$DOCKER_CERT_PATH/key.pem \https://172.16.88.129:2376/containers/c4a16378a11c/stats
docker-machine lsdocker-machine ip
docker ps
http://httpie.org/
![Page 34: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/34.jpg)
Remote API
wget --no-check-certificate \--certificate=$DOCKER_CERT_PATH/cert.pem \--private-key=$DOCKER_CERT_PATH/key.pem \https://172.16.88.129:2376/containe…ats
![Page 35: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/35.jpg)
Monitoring at scale
![Page 36: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/36.jpg)
Operational Complexity
• Average containers per host: N (N=5, 10/2014)• N-times as many “hosts” to manage• Affects
• provisioning: prep’ing & building containers• configuration: passing config to containers• orchestration: deciding where/when containers run• monitoring: making sure containers run properly
![Page 37: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/37.jpg)
Complexity increases with…
• Number of things to measure• Velocity of change
![Page 38: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/38.jpg)
…Number of things to measure
• 1 Hosted Virtual Machine• ~10 metrics depending on vendor
• 1 operating system (e.g. linux)• 100 metrics
• N containers• 100*N metrics
• 110 + 100*N metrics per vm
![Page 39: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/39.jpg)
Combinatorial multiplication
Assuming 5 containers per host
virtual machines
![Page 40: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/40.jpg)
Combinatorial multiplication
Assuming 5 containers per host
![Page 41: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/41.jpg)
Combinatorial multiplication
Assuming only 5 containers per host
virtual machines
![Page 42: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/42.jpg)
Velocity
![Page 43: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/43.jpg)
Tags
![Page 44: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/44.jpg)
![Page 45: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/45.jpg)
Tags
• From imperative to declarative• Query-based• Queries operate on tags
![Page 46: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/46.jpg)
“Monitor all Docker containers running image web”“… in region us-‐west-‐2 across all availability zones”“… and make sure resident set size < 1GB on c3.xl”
![Page 47: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/47.jpg)
“Monitor all Docker containers running image web”“… in region us-‐west-‐2 across all availability zones”“… and make sure resident set size < 1GB on c3.xl”
![Page 48: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/48.jpg)
“Monitor all Docker containers running image web”“… in region us-‐west-‐2 across all availability zones”“… that use more than 1.5x the average on c3.xl”
![Page 49: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/49.jpg)
Tags
• demo:nginx• demo:docker• demo:redis• demo:php• role:demo• platform:aws
• (platform:hpcloud, platform:fusion, platform:azure)
![Page 50: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/50.jpg)
How We Collect Stats for Datadog
![Page 51: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/51.jpg)
Installing the containerdocker run -d --privileged --name dd-agent \
-h `hostname` \-v /var/run/docker.sock:/var/run/docker.sock \-v /proc/mounts:/host/proc/mounts:ro \-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \-e API_KEY=80d4600a…8830 datadog/docker-dd-agent
![Page 52: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/52.jpg)
Installing the containerdatadog:image: ddagentenvironment:- API_KEY
privileged: truevolumes:- /var/run/docker.sock:/var/run/docker.sock- /proc/mounts:/host/proc/mounts:ro- /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
ports:- "8125:8125"
command: dd-agent foreground
![Page 53: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/53.jpg)
![Page 54: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/54.jpg)
![Page 55: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/55.jpg)
![Page 56: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/56.jpg)
![Page 57: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/57.jpg)
![Page 58: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/58.jpg)
![Page 59: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/59.jpg)
Summary
• Monitoring docker is hard because• there are oodles of containers• containers are created and killed often• # of metrics is enormous
• Declarative monitoring is the only way (tagging)
![Page 60: Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015](https://reader034.vdocuments.us/reader034/viewer/2022052603/55d708bbbb61ebe15d8b4571/html5/thumbnails/60.jpg)
Monitoring Docker at Scale
Matt WilliamsEvangelist @ [email protected]@technovangelist
…and having a way to answer every question you have about it