interactive monitoring for kubernetes
TRANSCRIPT
![Page 2: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/2.jpg)
![Page 3: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/3.jpg)
VisualisationMonitoring Tracing
0255075
100
![Page 4: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/4.jpg)
Monitoring
0255075
100
![Page 5: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/5.jpg)
Traditional 3-tier architectureIncoming traffic
Load balancers
Application servers
Database & replica
![Page 6: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/6.jpg)
Microservice architecture
Public APIWeb UI
NoSQL serversDatabase
Message Broker
Services
![Page 7: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/7.jpg)
![Page 8: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/8.jpg)
Microservices should be treated like cattle not pets
![Page 9: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/9.jpg)
![Page 10: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/10.jpg)
USE Method* - for every resource, check: • utilization, • saturation, and • errors
RED Method - for every service, check request: • rate, • error (rate), and • duration (distributions)
* http://www.brendangregg.com/usemethod.html
An alternative view
![Page 11: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/11.jpg)
Okay, but how?var rpcDurations = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "rpc_durations_histogram_microseconds",
Help: "RPC latency distributions.",
Buckets: prometheus.LinearBuckets(0, 100, 20),
})
func init() {
prometheus.MustRegister(rpcDurations)
}
func handleRequest(w http.ResponseWriter, r *http.Request) {
begin := time.Now()
...
rpcDurations.WithLabelValues(r.Method).Observe(
float64(time.Since(begin).Nanoseconds()))
}
![Page 12: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/12.jpg)
![Page 13: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/13.jpg)
There must be a better way…
Kubeproxy
Replicas
incoming traffic from other services
![Page 14: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/14.jpg)
Demo Time
![Page 16: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/16.jpg)
Monitoring
0255075
100
![Page 17: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/17.jpg)
Visualisation
![Page 18: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/18.jpg)
![Page 19: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/19.jpg)
Weave Scope
![Page 20: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/20.jpg)
Connection Tracking/home/weave # conntrack -E [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41066 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41066 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=36236 dport=32778 src=172.17.0.8 dst=192.168.99.100 sport=80 dport=36236 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41068 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41068 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52996 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80 dport=52996 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41070 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41070 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52998 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80 dport=52998 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41072 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41072 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=57975 dport=32777 src=172.17.0.7 dst=192.168.99.100 sport=80 dport=57975 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41074 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41074 [ASSURED]
/home/weave # cat /proc/net/tcp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 16810 1 ffff8800d79c1800 100 0 0 10 0 1: 0100007F:EB74 0100007F:0FC8 06 00000000:00000000 03:0000016D 00000000 0 0 0 3 ffff8800ae3f6e80 2: 0100007F:EB69 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 307011 1 ffff8800cf467040 21 4 30 10 -1 3: 0100007F:EB7B 0100007F:0FC8 06 00000000:00000000 03:00000D27 00000000 0 0 0 3 ffff8800d7a47538 4: 0100007F:EB7C 0100007F:0FC8 06 00000000:00000000 03:0000110E 00000000 0 0 0 3 ffff8800cf656c70 5: 0100007F:EB67 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 306868 1 ffff8800d79c1040 21 4 27 10 -1 6: 0100007F:EB76 0100007F:0FC8 06 00000000:00000000 03:00000556 00000000 0 0 0 3 ffff8800d37ac748 7: 0100007F:EB7F 0100007F:0FC8 06 00000000:00000000 03:000014F7 00000000 0 0 0 3 ffff8800d87f0c70
![Page 21: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/21.jpg)
all connections
from /proc
conntrack!
Connection Tracking
load balanced
![Page 22: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/22.jpg)
Demo Time
![Page 23: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/23.jpg)
https://github.com/weaveworks/scope
Try it out!
https://weave-scope-slack.herokuapp.com
v0.13.1
https://www.weave.works/products/weave-scope
https://scope.weave.works
![Page 24: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/24.jpg)
What does this look like with Kubernetes?
![Page 25: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/25.jpg)
Visualisation
![Page 26: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/26.jpg)
Tracing
![Page 27: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/27.jpg)
Distributed Tracing
![Page 28: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/28.jpg)
Not a new topic
• Lots of literature• Existing open source
projects• e.g. Zipkin, originally from
![Page 29: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/29.jpg)
• Challenge: detecting causality between incoming and outgoing requests
• Existing solutions require propagation of some unique ID (dapper, zipkin)
• This requires application-specific modifications
some service
incomingrequest
outgoingrequests
?
![Page 30: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/30.jpg)
Can this be done without application modifications?
![Page 31: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/31.jpg)
By intercepting systems calls, build up a data structure of:• which threads are reading
to / writing from which FDs• which FDs are talking to
which IPs & ports
Use this to infer causality between incoming and outgoing connections.
some service
kernel
?
System calls
![Page 32: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/32.jpg)
Demo Time
![Page 33: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/33.jpg)
Try it out!
https://github.com/weaveworks/scope/tree/master/experimental/tracer
![Page 34: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/34.jpg)
Tracing
![Page 35: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/35.jpg)
VisualisationMonitoring Tracing
0255075
100
![Page 37: Interactive Monitoring for Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022052514/5876fcd51a28abf3398b6897/html5/thumbnails/37.jpg)
https://weave-scope-slack.herokuapp.com
https://github.com/weaveworks/scope
https://github.com/weaveworks/flux
https://scope.weave.works
Links
https://github.com/weaveworks/scope/tree/master/experimental/tracer