redlining kafka pipelines

 Joel Koshy  Sr. Staff Software Engineer

 LinkedIn

Outline

Background

A look back at Kafka 0.7

Kafka 0.8, 0.9 and shifting bottlenecks

Kafka 0.10 message format changes

Mirroring – the 0.7 way

Q&A

Kafka at LinkedIn

Kafka 0.7

Kafka 0.7 message format (uncompressed)

Kafka 0.7 message format (compressed)

Kafka 0.7 message format

Message offset is the physical offset


Internal messages of compressed message sets are not addressable via a scalar offset


Consumer checkpoints offset M for this message

•  Tricky to checkpoint within compressed message set

•  Hard to rewind by N messages

•  Unsuitable for features such as log compaction

Drawbacks of 0.7 message format

B U T ! ! !

Very efficient (broker did not need to modify messages)

Predominantly network in 0.7

S H I F T I N G B O T T L E N E C K S O V E R T I M E

Kafka 0.8 Replication

Replication in Kafka 0.8


Predominantly network in 0.7 Still network in 0.8, gradually tilting toward storage


NOT a physical offset!

Kafka 0.8 message format (compressed)

Envelope offset is the largest offset in the set

Broker now needs to assign logical offsets even to internal messages

Handling produce requests

Kafka 0.9 Security (and many other features)

SSL

•  Forego zero-copy optimization

•  CPU overhead to decrypt/encrypt

•  Minor impact

•  (Used only on our mirroring pipelines at the time) Kafka 0.9

Mirroring topology and storage policy changes

Mirroring topology and storage policy changes

•  Reduced retention period across the board

•  File system tuning

2 0 1 5 - 2 0 1 6

New hardware 10Gbps NICs, bigger disks, XFS


Predominantly network in 0.7 Storage and network in 0.8, 0.9 (1Gbps NICs)

Increasingly CPU in 0.9 (10Gbps NICs)

Kafka 0.9 broker profile


Cluster expansion

Kafka 0.10 New message format

Kafka 0.10 message format (compressed + append time)

Kafka 0.10 message format (compressed + create time)

Handling produce requests

Handling fetch requests

Migrate clients before switching to 0.10 message format

Ideal Less ideal Worse Worst

Majority producer version 0.10 0.9 0.10 0.9

Majority consumer version 0.10 0.10 0.9 0.9

Migrating to the new message format

U S E C A U T I O N ! !

Severe performance degradation with older clients and there is no roll-back after switching

So know your clients!

•  Useful to have a shepherding system in your service infra

•  EOL older libraries

•  Check API versions in public access logs

•  Add API version metrics to the Kafka broker

PRODUCERS CONSUMERS

Impact of new message format

•  Fine-grained time-based offset lookup

•  Facilitates correctness of time-based retention

Other benefits of 0.10 format

“Time” under 0.9 message format

Message time is mtime of containing segment so time granularity is at segment-level

“Time” under 0.9 message format

Time resets on partition reassignment

Time in 0.10 message format

Time indexes built from timestamps in messages

Time in 0.10 message format

Mirroring pipelines (embracing the 0.7 way)

0.7 mirror maker was a dumb fast pipe

… limited only by network

0 . 7 M I R R O R M A K E R

0.8+ mirror maker

•  Needs to preserve order of keyed messages

0.8+ mirror maker

•  Needs to preserve order of keyed messages

•  0.8+ consumers do not support shallow iteration (KAFKA-732)

•  0.8+ producers do not support pass-through mode

Kafka 0.8+ mirror maker profile

Handling keyed messages in pass-through mode

•  Need to preserve order of keyed messages… but pass-through mirror maker cannot repartition

Handling keyed messages in pass-through mode

•  Need to preserve order of keyed messages… but pass-through mirror maker cannot repartition

•  Work around is to require identical partition counts across all clusters and do identity partitioning

•  i.e., Pinput= Poutput

•  Restore shallow iteration in consumer (KAFKA-1895)

•  “Todd’s trick” – introduce an identity compression codec in producer

•  Uniform partition counts across clusters

0.10 pass-through mirroring how-to

•  Restore shallow iteration in consumer (KAFKA-1895)

•  “Todd’s trick” – introduce an identity compression codec in producer

•  Uniform partition counts across clusters

•  … and a few more subtleties (future talk)

0.10 pass-through mirroring how-to

•  Jiangjie Qin (KIP-3[1,2,3])

•  Todd Palino (pass-through mirroring)

•  Kafka open source community

Acknowledgments

redlining kafka pipelines

Data & Analytics