stream processing at scale with spring xd and kafka

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

SPRINGONE2GXWASHINGTON, DC

Stream Processing at Scale with Spring XD and Kafka

By Marius Bogoevici @mariusbogoevici

(and more)

About me

• Staff Engineer at Pivotal

• Works on Spring XD, Spring Cloud Stream, Spring Cloud Data Flow, Spring Integration, Spring Integration

• Formerly: InfinityQuick, Red Hat (JBoss), SpringSource

Safe Harbor Statement

The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 4

Stream Processing in the Big Data

landscape

The domain of stream processing

• Now have the ability to cheaply store and analyze huge quantities of data

• Traditionally the realm of batch processing but also demand for real-time

• aka ‘Stream Processing’

• Some characteristics of Stream Processing

• High volume

• Low latency

• Often data is grouped and ordered

Streaming and real-time analysis examples

• “Internet of Things”

• Fraud Detection

• Measuring Quality of Service

• Predictive Maintenance

• Log aggregation

Our Toolkit

Apache Kafka message

• Initially developed at LinkedIn, open-sourced in 2011

• High-throughput distributed messaging system

• Publish-subscribe messaging rethought as a message log

• Decoupled data pipelines between producers and consumers

Apache Kafka: Topics and logs

• Each topic, divided in many partitions read/written independently

• Producers can target partitions and topics directly

• Distributed, replicated

• Linear reads, writes

Producers and consumers

• Producers, consumers completely decoupled from each other

• Partitions are distributed and replicated

• Reads/writes from the partition leader

Why Kafka for Stream Processing

• High throughput, low latency due to linear reads/writes

• High performance on commodity hardware

• Replayability (consuming from existing offsets)

• Can be used as a master dataset

• Consumer groups are completely independent of each other

• Guaranteed ordered delivery within a partition

• Competing consumer scenarios are difficult to implement

• Generally, not an issue with stream processing

Spring Integration Kafka

• Started in 2011

• Goal: adapting to the abstractions Spring Messaging and Spring Integration

• Easy access to the unique features of Kafka;

• Namespace, Java DSL support

• To migrate to 0.8.3 once available

• Defaults focused towards performance (disable ID generation, timestamp)

Spring Integration Kafka: Channel Adapters

Kafka Inbound Channel Adapter Kafka Outbound Channel AdapterSpring Messaging Message Channel

Spring Messaging Message

DI Configuration DI Configuration

Spring Integration Kafka Producer Configuration

• Default producer configuration

• Distinct per-topic producer configurations

• Destination target or partition controlled via SPEL expressions or headers

<int-‐kafka:producer-‐context id="kafkaProducerContext"> <int-‐kafka:producer-‐configurations> <int-‐kafka:producer-‐configuration broker-‐list="localhost:9092" key-‐class-‐type="java.lang.String" value-‐class-‐type="java.lang.String" topic="test1" value-‐serializer="kafkaSerializer" key-‐serializer="kafkaSerializer" compression-‐type="none"/> <int-‐kafka:producer-‐configuration broker-‐list="localhost:9092" topic="regextopic.*" compression-‐type="gzip"/> </int-‐kafka:producer-‐configurations> </int-‐kafka:producer-‐context>

Spring Integration Kafka Consumer• Own client based on Simple Consumer API

• Listen to specific partitions!

• Offset control - when to be written and where (no Zookeeper);

• Programmer-controlled acknowledgment;

• SI defaults focused towards performance

• No ID, timestamp generation (can be turned on, optionally)

• Concurrent message processing (preserving per-partition ordering) via Reactor

• Basic operations via KafkaTemplate

• Kafka specific headers

Spring Integration Kafka Consumer Example

<int-‐kafka:zookeeper-‐connect id="zkConnect" zk-‐connect=“localhost:2181"/>

<int-‐kafka:message-‐driven-‐channel-‐adapter id="adapter" channel="output" connection-‐factory="connectionFactory" key-‐decoder="decoder" payload-‐decoder="decoder" max-‐fetch="100" topics="${kafka.test.topic}"/>

Spring Integration Kafka Message Listener

• Auto-acknowledging

• With manual acknowledgment

public interface MessageListener { void onMessage(KafkaMessage message); }

public interface AcknowledgingMessageListener { void onMessage(KafkaMessage message, Acknowledgment acknowledgment); }

Spring Integration Kafka: Message Headers

public abstract class KafkaHeaders {private static final String PREFIX = "kafka_";public static final String TOPIC = PREFIX + "topic";public static final String MESSAGE_KEY = PREFIX + "messageKey";public static final String PARTITION_ID = PREFIX + "partitionId";public static final String OFFSET = PREFIX + "offset";public static final String NEXT_OFFSET = PREFIX + "nextOffset";public static final String ACKNOWLEDGMENT = PREFIX +

"acknowledgment";}

Spring Integration Kafka: Offset Management

• Injectable strategy

• Allows customizing the starting offsets

• Implementations: SI MetadataStore-backed (e.g. Redis, Gemfire), Kafka compacted topic-backed (pre-0.8.2), Kafka 0.8.2 native

• Messages can be auto acknowledged (by the adapter) or manually acknowledged (by the user)

• Manual acknowledgment useful when messages are processed asynchronously

• Acknowledgment passed as message header or as argument

DEMO: Spring Integration Kafka at Work

Spring XD and Stream processing

• Higher abstractions are required

• Integrating seamlessly and transparently with the middleware

• Building on top of Spring Integration and Spring Batch

• Pre-built modules using the entire power of the Spring ecosystem

Streams in Spring XD

HTTP$JMS$Ka*a$

RabbitMQ$JMS$

Gemfire$File$SFTP$Mail$JDBC$Twi;er$Syslog$TCP$UDP$MQTT$Trigger$

Filter$Transformer$

Spli;er$Aggregator$HTTP$Client$

JPMML$Evaluator$Shell$Python$Groovy$Java$RxJava$

Spark$Streaming$

File$HDFS$HAWQ$Ka*a$

RabbitMQ$Redis$Splunk$Mongo$Redis$JDBC$TCP$Log$Mail$

Gemfire$MQTT$

Dynamic$Router$Counters$

Note: Named channels allow for a directed graph of data flow

channel

Spring XD: Stream DSL

Spring XD - Message Bus abstraction

• Binds module inputs and outputs to a transport

Binds module inputs and outputs to a transport Performs Serialization (Kryo) Local, Rabbit, Redis, and Kafka

Spring XD Distributed Runtime

Spring XD and Kafka - the message bus

• Each pipe between modules is a topic;

• Spring XD creates topics automatically;

• Topics are pre-partitioned based on module count and concurrency;

• Overpartitioning is available as an option;

• Multiple consumer modules ‘divide’ the partition set of a topic using a deterministic algorithm;

Spring XD and Kafka: Partitioning

• Required in distributed stateful processing: related data must be processed on the same node;

• Partitioning logic configured in Spring XD via deployment manifest

• partitionKeyExpression=payload.sensorId

• When using Kafka as a bus, partition key logic maps directly to Kafka transport partitioning natively

Partition 0

Partition 1

Average Processor

http | avg-temperatures

Spring XD and Kafka:

Combine the high performance of Kafka with the powerful abstractions of Spring XD

Performance metrics of Spring XD 1.2

Demo: Stream processing at scale with Spring XD

High%volume%

Low%Latency%

Programming%Model%

Distributed%Run;me%

Ka=a%%(moving%data)%

RxJava%(processing)%

Spring%XD%(distribu;ng)%

Cloud Native

Stream Processing at Scale: Message-driven Microservices

Scaling with streams

• Parallelism is key for scaling

• A single machine’s resources can be exhausted really fast

• CPU: serialization

• Memory: keeping state, caching lookup data

• Disk: especially for brokers

• Network: saturating the interfaces

• Vertical scaling == exponential cost

• And failover redundancy is still a requirement

• Cluster sizes can get really big

• tens, hundreds of module instances

• which leads to ….

Spring Cloud Stream/Spring Cloud Data Flow

• Successors to Spring XD 1.x

• New goals:

• Cloud native;

• Micro service architecture for modules;

• Most valuable features should be preserved;

• Abstracting out the connection to the transport

• Same abstractions, backwards compatible DSL;

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 40Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring XD Modules

•  Execute inside of a Spring XD Container

•  Consist of beans in an application context

•  Multiple modules may run in a container, each with their own class loader

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 41Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 7

•  OOTB modules •  Executable Boot Apps

•  Boot for Spring Integration and Batch •  Auto-Configuration, Bindings

Container

Modules

Spring Cloud Data Flow

Spring Cloud Stream

Modules

Spring Cloud Task Modules

Spring Cloud Stream

Spring Cloud Task

•  REST API, Shell, UI •  Module Deployer SPI:

•  Singlenode •  Lattice •  YARN •  Cloud Foundry

From Modules to Microservices

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 42Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 10

Container Container

Spring XD 1.x

Spring Cloud Stream

•  Modules run inside containers •  Container is a Boot app •  Modules are ApplicationContexts

•  Modules are executable Boot apps •  Easier to use Spring Cloud features •  Easier to test, consistent dev experience •  Portable and “cloud-native”

ZooKeeper Spring Cloud Data Flow (optional) SPI Implementation

Transparent development model

public interface Source { String OUTPUT = "output"; @Output(Source.OUTPUT) MessageChannel output(); }

@EnableBinding(Source.class) @EnableConfigurationProperties(TimeSourceProperties.class) @Import(PeriodicTriggerConfiguration.class) public class TimeSource { @Autowired private TimeSourceProperties properties; @InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller( trigger = PeriodicTriggerConfiguration.TRIGGER_BEAN_NAME, maxMessagesPerPoll = "1")) public String publishTime() { return new SimpleDateFormat(this.properties.getFormat()).format(new Date()); } }

Where is Kafka ?

• Out of sight, but not out of mind

• On the class path ( just like all the other binders)

• Bindings/Channel Adapters are created transparently

• Channel names are topic names (overridable by Boot properties)

• Reusing code of the Spring XD 1.x Kafka message bus

• Partitioning/multiple instance support, with a microservice twist

• Spring Boot properties to compute the target partition set (instanceIndex, instanceCount)

INSTANCE_INDEX=0

INSTANCE_INDEX=1

INSTANCE_INDEX=6

Kafka Service

http.0 (0) http.0 (1)

http.0 (2)

http.0 (6)

Broker 0 Broker 1 Broker 4

stream create logger --definition "http | log"

stream deploy logger --properties module.log.partitioned=true, module.log.count=7

= massive scale!

Demo: Scaled, Partitioned Streams on PCF and Lattice

Tell your friends!

Developing Real-Time Data Pipelines with Apache Kafka - Joe Stein, Thursday, 10:30

Learn More. Stay Connected.

@springcentral Spring.io/video

stream processing at scale with spring xd and kafka

Technology

stream processing at linkedin: apache kafka & apache …...

apache samza: reliable stream processing atop apache kafka...

gnw03: stream processing with apache kafka by gwen shapira

data stream processing - amazon s3 · • high performance...

introducing kafka streams: large-scale stream processing...

scalable stream processing with apache kafka and apache...

event stream processing with kafka and samza

streaming data and stream processing with apache kafka

event-stream processing with kafka

distributed stream processing with apache kafka

kafka in production - wordpress.com · apache kafka apache...

building kafka-powered activity stream

spark streaming & kafka-the future of stream processing

event stream processing using kafka...

apache kafka, and the rise of stream processing

vip xd single/quad-stream decoder - network webcamscctv |...

kafka and stream processing, taking analytics real-time,...

vip xd single/quad-stream decoder - bosch security...

kafka streams: the stream processing engine of apache kafka

building stream infrastructure across multiple data centers...