logging & docker - season 2
TRANSCRIPT
Sumo Logic Confidential
Logging & Docker
Christian Beedgen, CTO & Co-Founder, Sumo Logic
Seattle Docker Meetup, October 13, 2015
Sumo Logic Confidential
$ whoami• Co-Founder & CTO, Sumo Logic
Cloud-based Machine Data Analytics ServiceApplications, Operations, Security
• Chief Architect, ArcSightMajor SIEM player in the enterprise spaceLog Management for security and compliance
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York Cityhttp://www.slideshare.net/
raychaser/6-million-ways-to-log-in-docker-nyc-docker-meetup-12172014
Sumo Logic Confidential
Season 2Where Are We In Late 2015?
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
• Also, one process per container, plz!
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Early hardcore crowd would just collect /var/lib/docker/containers/**
• And then of course there’s the UX: docker logs• docker logs is using a daemon API for getting the logs• This leads to logspout – attach to API, forward to Syslog• https://github.com/gliderlabs/logspout
Sumo Logic Confidential
Docker 1.6 Introduced Log Drivers
• Hallelujah• Initially supports json-file, syslog, null• json-file – default, this is the old mechanism
– Continues to this day to be required for API access and docker logs• docker run -–log-driver syslog …
– Sends to local Syslog, no more writing to disk• docker run –-log-driver null
– STFU, basically
Sumo Logic Confidential
Docker 1.7 Introduces --log-opt
• Now we can pass parameters to the log drivers!• docker run \
--log-driver syslog \
--log-opt syslog-address=(udp|tcp)://… \
--log-opt syslog-facility=(kern|daemon|user|local0|…) \
--log-opt syslog-tag=“myapp”
• Forward directly to local Syslog aggregator, or to a cloud-based logging service
• Docker 1.7 also added support to log to journald
Sumo Logic Confidential
Docker 1.8, 1.9 - Even More Log Drivers
• Fluentd
• GELF
• AWS
Sumo Logic Confidential
Also in Docker 1.8 – Options For json-file
• json-file still the default, still required for docker logs and /logs API• Long standing problem – will eventually fill up your disk• Folks have been using logrotate hacks…• Now, json-file log driver can be configured:
• Basically, keep up to max-file files, roll current at max-size
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
What Is Sumo Working On?
• We have containerized our collectors– https://github.com/SumoLogic/sumologic-collector-docker– docker run -d -p 514:514 -p 514:514/udp \
--name="sumo-logic-collector" \
sumologic/collector:latest-syslog \
[Access ID] [Access key] – https://www.sumologic.com/2015/09/09/update-on-logging-with-docker/
Sumo Logic Confidential
What Is Sumo Working On?
• We are working towards our vision of Comprehensive Monitoring– https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than
-just-logs/
• We have released an initial App for Docker at DockerCon 2015