kafka y python

Define y gobierna tus APIs

Kafka y python

17/05/2016

Python Madrid · Python y Kafka

Kafka y python

Kafka y Python

Python Madrid · Python y Kafka

¿Quién soy?

Ingeniero de Software @Paradigma Digital

@lvaroleonaleonsan

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Origen

2011

- Op

en so

urce

2012

- Ap

ache

Incu

bato

r

gra

duat

ion

2014

- Co

nflu

ent

($6,

9 M

)

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ ¿Qué es?

“

”

If you think of Hadoop as long-term memory, the question then is how you get the memories in there to begin with

Apache Kafka is like the central nervous system, which collects all of these messages from the underlying systems and transmits them into the memory vault, or storage.

- Eric Vishria

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Motivation

To be able to act as a unified platform for handling all the real-time data feeds a large company might have.

…

…

…

…

…

…

Event Tracking

ApplicationLogs

ApplicationMessages

ApplicationMonitoringdata

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ How to ?

● Distributed, the essence● Scalable● Efficient● Durable, fault tolerance

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Básicos

P PP

C C C C

…

…

…

Kafka Cluster

● Producers● Brokers● Consumers

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Cluster: Topics & Partitions

Kafka Cluster ● Topics● Partitions● Message

1 2 3 4 5

1 2 3 4

1 2 3

1 2 3

T1P0

T1P1

T2P0

T2P1

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Partitions & Replication

Kafka Cluster ● Replication factor○ Leader○ Followers

● ISR○ In-sync policies

1 2 3 4 5

Bro

ker1

Bro

ker2

1 2 3 4 5

PM

PM

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Producers

● Publish Messages● Choose partitions

○ policies

● Producer configuration○ ACKs○ Retries○ Batch size○ ...

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Consumers

● “Subscribe” to a feed● Consumer groups Kafka Cluster

Partition 0

Bro

ker1

Bro

ker2

Partition 1

○ Queue○ Publish-subscribe

CC

C

● Order guarantees

C

C

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Efficiency

● Small I/O problem○ Message sets

● Message set compression○ policies

● Standard binary message format○ Transfer without modifications

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients

● Kafka-python○ 0.8+, recomendada 0.9○ Python 3.6+○ Python 3.3+

https://github.com/dpkp/kafka-python

● Pykafka○ 0.8.2+○ Python 2.7+○ Python 3.4+

https://github.com/Parsely/pykafka

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ Kafka-python

● Producer class kafka.KafkaProducer: def __init__(self, **configs)

def send(self, topic, value=None, key=None, partition=None)● class RecordAccumulator:● class Partitioner:

def flush(self, timeout=None)

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ Kafka-python

● Consumer ○ message iterator

class kafka.KafkaConsumer(six.Iterator): def __init__(self, *topics, **configs) def __next__(self)

def subscribe(self, topics=(), pattern=None, listener=None) def unsubscribe(self) def assign(self, partitions) def seek(self, partition, offset) def commit(self, offsets=None)

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ Kafka-python

● Cluster○ client manages some cluster metadata

class kafka.ClusterMetadata: def __init__(self, **configs)

def available_partitions_for_topic(self, topic) def leader_for_partition(self, partition) def partitions_for_broker(self, broker_id)

…

def update_metadata(self, metadata)

● ConsumerCoordinator

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ pyKafka

● Producerclass pykafka.Producer: def __init__(self, . . . ) def produce(self, message, partition_key=None)

● Consumer class pykafka.SimpleConsumer: def __init__(self, . . .)

def consume(self, block=True)

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ Kafka-python

● Ejemplo

client = pykafka.KafkaClient(. . .) topic = client.topics[0] producer = topic.get_sync_producer()

. . .

consumer = topic.get_simple_consumer()for message in consumer:

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ Python Clients \ Demo

Demo Time

Kafka y Python

Python Madrid · Python y Kafka

Kafka y Python \ Thanks

for your attentionThank you

Kafka y Python

Python Madrid · Python y Kafka

Kafka y Python \ Questions

¿ ?

Kafka y Python

Python Madrid · Python y Kafka

Kafka \ El Clúster