![Page 1: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/1.jpg)
Three Pillars of Observability in Kubernetes with Elastic Stack
Eric WestbergSolutions Architect, Elastic
![Page 2: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/2.jpg)
Store, Search, & Analyze
Visualize & Manage
Ingest
Elastic Stack
Kibana
Elasticsearch
Beats Logstash
Elastic Stack
![Page 3: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/3.jpg)
4
• Custom on-prem & cloud deployments
• Public cloud fully-managed deployments– Google Kubernetes Engine (GKE)
– Amazon Elastic Container Service for Kubernetes (EKS)
– Azure Kubernetes Service (AKE)
• Pivotal Container Service (PKS)
• Red Hat OpenShift
Kubernetes is Taking Over the Enterprise
![Page 4: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/4.jpg)
8
It Comes Down to The Three Pillars of Observability
Twitter:https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.htmlPeter Bourgonhttps://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html
![Page 5: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/5.jpg)
9
Elastic at the Center Stage
![Page 6: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/6.jpg)
1010
Logging
● Cluster level logging
● Services logging (eg. nginx, mysql)
● Custom application logging
![Page 7: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/7.jpg)
11
Kubernetes Logging
• Need for a logging solution– Kubernetes does not have a native solution
– kubectl logs is too hard for large clusters
• Cluster-level logging– Logs have separate storage and lifecycle independent of nodes, pods and containers
– Kubernetes provides no native storage solution for log data
• Application-level logging– Complicated
– Packaged applications (eg. nginx)
– Custom applications
![Page 8: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/8.jpg)
12
Two Packaged Solutions
• Fluentd DaemonSet– Log collection, parsing and distribution
• Fluentd + Stackdriver for GCP
• Fluentd + Elasticsearch
![Page 9: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/9.jpg)
13
Better Log Collection with Filebeat
kubectl create -f filebeat-kubernetes.yaml
![Page 10: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/10.jpg)
14
Kubernetes deployment
Node 1
Metricbeat
Filebeat
Node 2
Metricbeat
Filebeat
Node n
Metricbeat
Filebeat
Filebeat DaemonSet
Metricbeat DaemonSet
![Page 11: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/11.jpg)
15
Metadata processorsEnrich events with useful metadata to correlate logs, metrics & traces
• cloud.availability_zone• cloud.region• cloud.instance_id • cloud.machine_type• cloud.project_id• cloud.provider
• docker.container.id• docker.container.image• docker.container.name• docker.container.labels
• kubernetes.pod.name• kubernetes.namespace• kubernetes.labels• kubernetes.annotations• kubernetes.container.name• kubernetes.container.image
add_cloud_metadata add_docker_metadata add_kubernetes_metadata
![Page 12: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/12.jpg)
16
Filebeat Auto-Discovery
filebeat.autodiscover:providers:- type: kubernetestemplates:- condition:
contains:kubernetes.container.image: "nginx"
config:- module: nginxaccess: # For nginx access logprospector:type: dockercontainers.ids:- "${data.kubernetes.container.id}"
• A module contains– Log file path
– Ingest pipeline
– Fields definitions
– Sample dashboards
![Page 13: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/13.jpg)
17
• Apache2 module
• Auditd module
• Icinga module
• IIS module
• Kafka module
• Logstash module
• MongoDB module
Filebeat ModulesSimplify collection, parsing and visualization of common log formats
• MySQL module
• Nginx module
• Osquery module
• PostgreSQL module
• Redis module
• System module
• Traefik module
![Page 14: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/14.jpg)
1818
Metrics
● Metrics data sources
● Popular solutions
● Metricbeat
![Page 15: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/15.jpg)
19
Kubernetes Monitoring
• What to monitor– Cluster monitoring– Pod monitoring– Application monitoring
• Metrics sources– cAdvisor & Heapster– Kube-state-metrics– Prometheus– APM
• Solutions– Heapster/InfluxDB/Grafana– Heapster/Elasticsearch– Prometheus/Grafana– APM - Datadog, Dynatrace– Metricbeat with Autodiscovery
Collect Store Analyze
ElasticsearchInfluxDB...
KibanaGrafana...
MetricbeatHeapsterPrometheus...
SearchDashboardAlerts...
Data ModelMetrics Sources
![Page 16: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/16.jpg)
20
Comprehensive Metrics Collection Metricbeat
• Kubernetes module• Monitors pods and services
– Cluster, pod & container metrics– Application metrics through auto-discovery
(eg. Nginx)• Metrics sources - Cover them ALL
– Kubelet (heapster, cAdvisor)– kube-state-metric– Kubernetes events– Prometheus module (beta)
• Curated Infra UI – Dedicated Kibana app
![Page 17: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/17.jpg)
21
Out -of -the -box Dashboards
![Page 18: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/18.jpg)
22
Curated UI for KubernetesVisualize the cluster and group by nodes or namespaces or pods
![Page 19: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/19.jpg)
23
Monitor Services inside Containers with Auto-Discovery
Metricbeat Filebeat
Node n
Logs
MetricsNginx
metricbeat.autodiscover:
providers:
- type: kubernetes
host: ${HOSTNAME}
templates:
- condition.contains:
kubernetes.container.name: nginx
config:
- module: nginx
period: 10s
metricsets: ["stubstatus"]
hosts: ["${data.host}:8080"]
![Page 20: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/20.jpg)
24
Metricbeat ModulesSimplify collection and visualization of common metrics
● Aerospike module● Apache module● Ceph module● Couchbase module● Docker module● Dropwizard module● Elasticsearch module● Etcd module● Golang module● Graphite module● HAProxy module● HTTP module
● Jolokia module● Kafka module● Kibana module● Kubernetes module● kvm module● Logstash module● Memcached module● MongoDB module● Munin module● MySQL module● Nginx module
● PHP_FPM module● PostgreSQL module● Prometheus module● RabbitMQ module● Redis module● System module● uwsgi module● vSphere module● Windows module● ZooKeeper module
![Page 21: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/21.jpg)
2525
Tracing
● Elastic APM
![Page 22: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/22.jpg)
27
First Major Open Source APM SolutionAgents, Server, Dashboards
![Page 23: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/23.jpg)
28
APM Tracing -Transaction
● Distributed tracing● Trace the entire
transaction, across all instrumented services
● Includes calls to external services and databases
![Page 24: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/24.jpg)
29
You can do MORE ...
• Enforce access policies with Security
• Be notified about changes & problems with Alerting
• Be smarter with Machine Learning
• ...
![Page 25: Three Pillars of Observability in Kubernetes with Elastic ......Three Pillars of Observability in Kubernetes with Elastic Stack . Eric Westberg. Solutions Architect, Elastic. Store,](https://reader030.vdocuments.us/reader030/viewer/2022040204/5ec55f7edb364e203c69a74c/html5/thumbnails/25.jpg)
THANK YOUWeb : www.elastic.co Products : https://www.elastic.co/products Forums : https://discuss.elastic.co/ Community : https://www.elastic.co/community/meetupsTwitter : @elastic