Kafka meetup JP #3 - Engineering Apache Kafka at LINE

Download Kafka meetup JP #3 - Engineering Apache Kafka at LINE

Post on 28-Jan-2018

898 views

Category:

Engineering

0 download

TRANSCRIPT

1. Engineering Apache Kafka at LINE Yuto Kawamura 2. Speaker introduction - Name: Yuto Kawamura(kawamuray) - Software engineer @ LINE - Designing and implementing inter-service data passing infrastructure - Apache Kafka - Apache Kafka contributor - KAFKA-4614 Improved brokers response time - KAFKA-4024 Removed unnecessary blocking behavior of producer - Past publications - Applying Kafka Streams for internal message delivery pipeline - https://engineering.linecorp.com/en/blog/detail/80 - Monitoring Apache Kafka w/ Prometheus - https://www.slideshare.net/kawamuray/monitoring-kafka-w-prometheus 3. Apache Kafka - a new layer for service isolation Application server Another app server Stats system Abuser detection system Produce once, unified data format, general info Failure destination can resume consuming data after recovered. Apache Kafka - Publisher can stay agnostic for: - who uses data - remote failure - Subscriber can: - stay isolated from unexpectedly high workload on publisher - stop consuming temporary and continue after recovered from failure - can access all data from different services in the same way All data in 1-path 4. Data and usage - What kind of data are stored? - Structured events - Application servers request log(!= access log) - Mutation logs of HBase datastore - Task processing request - How are those data used? - Data replication - Protect service storage from unexpected access - Abuser detection - User action based processing - Metrics generation - Statistics - Asynchronous task processing 5. Facts - Kafka version: 0.10.0.1 + inhouse patches and backports - 140+ billion messages /day - 38+ TB incoming data /day - 3.5+ million messages /sec on peak - tens of broker servers - Single, multi-tenanted cluster(+ secondary cluster for backup) - Target latency(Produce response time): - < 1 ms for 50th %ile - < 10ms for 99th %ile 6. Monitoring - Prometheus + Grafana - ref: https://www.slideshare.net/kawamuray/monitoring-kafka-w-prometheus - jmx_exprter and node_exporter - for Kafka broker metrics - simpleclient_java - for Producer/Consumer metrics - kafka-consumer-group-exporter - for general consumer monitoring - https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter - Templated Grafana dashboard for generalized consumer monitoring - Consumers can start monitoring their consumers health immediately after the deployment 7. Hello, experts :D 8. Advanced observations - Apache Kafka performs really well. However... - Its heavily relying on various system primitives - page cache - https://kafka.apache.org/documentation/#design_filesystem - sendfile(2) (linux) - https://kafka.apache.org/documentation/#maximizingefficiency - FileRecords#writeTo: bytesTransferred = channel .transferTo(position, count, destChannel); 9. Observing Kafkas performance detail - Sometimes built-in metrics and jtools(jstack, jmap, jvisualvm) arent enough to learn everything - e.g, Reading disk is unusual thing in our cluster - happens only when some consumer being delayed and start requesting old offsets - Dive into OS metrics - node_disk_bytes_read(from node_exporter) 10. The problem is... - Slow response for FetchConsumer caused by reading disk, potentially involves other responses being sent within the same network processor - Disk read potentially happens transparently inside sendfile(2) syscall - => not observable from application(Kafka) level - Some consumer impl like old storm-kafka doesnt checkpoints consumption offset to broker - => cant observe currently consumed offset from broker side 11. Dive into kernel - SystemTap - SystemTap - A tool to perform dynamic tracing on Linux operation system - Can hook arbitrary execution point of Linux kernel internal - Very less overhead with compare to other traditional tools - Similar tools: - DTrace - eBPF 12. Observing syscall duration global s global records global sampled probe begin { printf("Start observing syscall %s duration... Press C-c to exitn", @1) } probe syscall.$1 { s[tid()] = gettimeofday_us() } probe syscall.$1.return { elapsed = gettimeofday_us() - s[tid()] delete s[tid()] records /path/to/kafka-data/topic-foo-bar-9/00000000077927876918.log lrwx------ 1 user user 64 May 19 14:01 /proc/PID/fd/11388 -> socket:[311161013] $ netstat -npe | grep 311161013 tcp 0 0 LOCAL_IP:12345 REMOTE_IP:60258 ESTABLISHED 123 311161013 PID/java - Gotcha! - Technique was leveraged when I worked for https://issues.apache.org/jira/browse/KAFKA-4614