Download - Sysdig Monitorama Slides
![Page 1: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/1.jpg)
The Dark Art of Container Monitoring
Loris Degioanni
![Page 2: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/2.jpg)
Me
Loris Degioannisysdig’s creator and CEOPast: WinPcap, Wireshark
@ldegio
![Page 3: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/3.jpg)
Containers Are Great…
• less overhead• faster deployments• reproducibility of environments• cost optimizations• Isolation• flexibility
![Page 4: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/4.jpg)
…But Monitoring Them Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable
![Page 5: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/5.jpg)
…But Monitoring Them Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight• Disposable
Is it possible to respect these properties and also have great monitoring?
![Page 6: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/6.jpg)
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
![Page 7: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/7.jpg)
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
![Page 8: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/8.jpg)
In This Talk
•Overview of container monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
![Page 9: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/9.jpg)
In This Talk
![Page 10: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/10.jpg)
Some Things We Want to Monitor
• Resource usage (CPU/Memory/Disk)• Network activity• Application activity• File I/O activity• Errors/faults• Topology
![Page 11: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/11.jpg)
Resource Usage (CPU/Memory/Disk)
• ps/top/htop from the host• Pro: always there• Cons: Very limited container context (full cgroup string only), only work from
the host
• docker ps / docker top• cAdvisor• docker stats• sysdig
![Page 12: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/12.jpg)
Installing cAdvisor
sudo docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:rw \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--publish=8080:8080 \--detach=true \--name=cadvisor \google/cadvisor:latest
![Page 13: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/13.jpg)
cAdvisor Pros and Cons
Pros• easy to install• works from a container• Nice local UI• API• Integrations (e.g. InfluxDB, Prometheus, Heapster)
Cons: • (very) Limited set of metrics• Containers are black boxes
![Page 14: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/14.jpg)
Docker Stats + API
• docker stats mysql• echo -e "GET /containers/mysql/stats HTTP/1.1\r\n" |
nc -U /var/run/docker.sock
![Page 15: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/15.jpg)
Docker Stats + API
• cpu_stats• network• memory_stats• blkio_stats
http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats-based-on-resource-usage
![Page 16: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/16.jpg)
Docker Stats Pros and Cons
Pros• Part of Docker• Richer than cAdvisor• REST API• Integrations
• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin• commercial vendors
Cons: • Docker only• Containers are black boxes
![Page 17: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/17.jpg)
sysdig
•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + Lua•Open Source
![Page 18: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/18.jpg)
csysdig
•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + htop + iftop + Lua•Open Source•Nice curses UI
![Page 19: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/19.jpg)
Design Goals
•Production-ready• Simple • lightweight
•Rich data•Native support for containers• Runs in a container
•Natural workflow
![Page 20: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/20.jpg)
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
![Page 21: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/21.jpg)
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
Instrumentation through kernel module
![Page 22: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/22.jpg)
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
Capture andanalysis
![Page 23: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/23.jpg)
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
(optionally) Save to a trace file
foo.scap
![Page 24: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/24.jpg)
Viewing Network/App Activity
•cAdvisor•docker stats• iftop/tcpdump/tshark in the container•sysdig
![Page 25: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/25.jpg)
Viewing File I/O Activity and Errors
•iotop/lsof in the container•sysdig
![Page 26: Sysdig Monitorama Slides](https://reader035.vdocuments.us/reader035/viewer/2022062710/55b6cf6cbb61eb286c8b48ff/html5/thumbnails/26.jpg)
Topology
•Sysdig cloud