akka streams - from zero to kafka

31
AKKA STREAMS FROM ZERO TO KAFKA Created by / Mark Harrison @markglh

Upload: mark-harrison

Post on 16-Apr-2017

102 views

Category:

Software


3 download

TRANSCRIPT

Page 1: Akka Streams - From Zero to Kafka

AKKA STREAMSFROM ZERO TO KAFKACreated by / Mark Harrison @markglh

Page 2: Akka Streams - From Zero to Kafka

HOW IT ALL BEGAN“Reactive Streams is an initiative to provide astandard for asynchronous stream processing

with non-blocking back pressure. Thisencompasses efforts aimed at runtime

environments (JVM and JavaScript) as well asnetwork protocols.”

Page 3: Akka Streams - From Zero to Kafka

WHYEf�ciently processing large indeterminate streams is hardAvoiding blocking is essential to maximise performanceEvery stage in the stream needs to be able to push and pullWe don't want to overload (or starve!) downstreamconsumers...

Page 4: Akka Streams - From Zero to Kafka
Page 5: Akka Streams - From Zero to Kafka

HOWTreat data as a stream of elementsAsynchronous non-blocking data and demand �owsDemand �ows upstream, causing data to �ow downstreamData �ow is therefore restricted by demand

Back Pressure!!Demand happens on a separate �ow!

Page 6: Akka Streams - From Zero to Kafka

WHATThe Reactive Streams speci�cation is just that

A collection of interfaces methods and protocolsProvides example implementations and a TCK forveri�cationAimed at providing a way to build commonimplementations

Page 7: Akka Streams - From Zero to Kafka

INTRODUCING AKKA STREAMS!!AKKA'S IMPLEMENTATION OF REACTIVE STREAMS

Page 8: Akka Streams - From Zero to Kafka

DESIGN PRINCIPLESExplicitness over magic (I'm looking at you Shapeless!)Fully composable

Each component, or set of componenents can be combinedEach building block is immutableFully compatible with other Reactive Stream implementations

Page 9: Akka Streams - From Zero to Kafka

BUILDING BLOCKS

Page 10: Akka Streams - From Zero to Kafka

BUILDING BLOCKS CONT...Source

Traditionally known as a producerSupplies messages that will �ow downstreamExactly one output stream

SinkTraditionally known as a consumerEnd point of the stream, this is where messages end up

Page 11: Akka Streams - From Zero to Kafka

BUILDING BLOCKS CONT...Flow

A processing stage in the StreamUsed to compose StreamsExactly one input and one output streamSee also BidirectionalFlow (two in -> two out)

Page 12: Akka Streams - From Zero to Kafka

BUILDING BLOCKS CONT...RunnableGraphs

A pre-assembled set of Stream components, packaged intoa Graph.All exposed ports are connected (between a Source andSink)This can then be Materialized

Page 13: Akka Streams - From Zero to Kafka

BUILDING BLOCKS CONT...Composite Flows

It is possible to wrap several components into morecomplex onesThis composition can then be treated as one block

Partial Flow GraphsAn incomplete Flow (Graph)Can be used to construct more complex Graphs easily

Page 14: Akka Streams - From Zero to Kafka

BUILDING BLOCKS CONT...Materializer

Once complete, the �ow is Materialized in order to startstream processingSupports fully distributed stream processing

Each step must be either serializable immutable valuesor ActorRefs

Fails immediately at runtime if the Graph isn't complete

Page 15: Akka Streams - From Zero to Kafka

ERRORS VS FAILURESErrors handlied within the stream as normal data elements

Passed using the onNext functionFailure means that the stream itself has failed and is collapsing

Raises the onError signal... (???)Each block in the �ow can choose to absorb or propagate theerrors

Possibly resulting the the complete collapse of the �ow

Page 16: Akka Streams - From Zero to Kafka
Page 17: Akka Streams - From Zero to Kafka

FIRST THINGS FIRSTWe need to create an ActorSystem and Materializer

implicit val system = ActorSystem("actors") implicit val materializer = ActorMaterializer()

Page 18: Akka Streams - From Zero to Kafka

SIMPLE STREAMWe need to create an ActorSystem and Materializer

Source(1 to 5) .filter(_ < 3) // 1, 2 .map(_ * 2) // 2, 4 .to(Sink.foreach(println)) .run() //prints 2 4

Page 19: Akka Streams - From Zero to Kafka

COMPOSING ELEMENTS TOGETHERWe can combine multiple components together

Composing elements together val nestedSource = Source(1 to 5) .map(_ * 2) val nestedFlow = Flow[Int] .filter(_ <= .map(_ + 2) val sink = Sink.foreach(println) //link up the Flow to a Sink val nestedSink = nestedFlow.to(Sink.foreach(println)) // Create a RunnableGraph - and run it! Prints 4 6 nestedSource.to(nestedSink).run()

Page 20: Akka Streams - From Zero to Kafka

COMPOSING ELEMENTS TOGETHER CONT...Alternatively we could do this, linking them in one step

nestedSource .via(nestedFlow) .to(Sink.foreach(println(_)))

Page 21: Akka Streams - From Zero to Kafka

COMPOSING ELEMENTS TOGETHER CONT...

Page 22: Akka Streams - From Zero to Kafka

GRAPH PROCESSING STAGESFan OutBroadcast[T] – (1 input, N outputs)Balance[T] – (1 input, N outputs)...

Fan InMerge[In] – (N inputs , 1 output)...

Timer DrivengroupedWithin(Int, Duration)

Groups elements when either the number or duration isreached (whichever is �rst). Very useful for batchingmessages.

See the Akka Stream docs for more!

Page 23: Akka Streams - From Zero to Kafka

GRAPH PROCESSING STAGES CONT...

Page 24: Akka Streams - From Zero to Kafka

THE GRAPH DSLWhenever you want to perform multiple operations tocontrol the Flow of a Graph, manually constructing them asabove can become very clumbersome and tedius, not tomentioned hard to maintain.For this reason the Akka team have written a DSL to helpwrite complex Graphs.

Page 25: Akka Streams - From Zero to Kafka

THE GRAPH DSLval g = FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => //This provides the DSL import FlowGraph.Implicits._ val in = Source(1 to 3) val out = Sink.foreach(println) //2 outputs, 2 inputs val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge } g.run() //Prints 31 31 32 32 33 33

Page 26: Akka Streams - From Zero to Kafka

THE GRAPH DSL CONT...

Page 27: Akka Streams - From Zero to Kafka

EXAMPLE - REACTIVE KAFKAThe guys at SoftwareMill have implemented a wrapper forApache Kafka

Tried and tested by yours trulyhttps://github.com/softwaremill/reactive-kafka

Page 28: Akka Streams - From Zero to Kafka

EXAMPLE - REACTIVE KAFKA CONT...Source is a Kafka ConsumerSink is a Kafka Publisher

val kafka = new ReactiveKafka() val publisher: Publisher[StringKafkaMessage] = kafka.consume( ConsumerProperties(...) ) val subscriber: Subscriber[String] = kafka.publish( ProducerProperties(...) ) Source(publisher).map(_.message().toUpperCase) .to(Sink(subscriber)).run()

Page 29: Akka Streams - From Zero to Kafka

A REAL WORLD EXAMPLE

Page 30: Akka Streams - From Zero to Kafka

A REAL WORLD EXAMPLE CONT...FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => import FlowGraph.Implicits._ val in = Source(kafkaConsumer) val out = Sink.foreach(println) val bcast = builder .add(Broadcast[StringKafkaMessage](2)) val merge = builder .add(Merge[StringKafkaMessage](2)) val parser1, parser2 = Flow[StringKafkaMessage] .map(...) val group = Flow[StringKafkaMessage].grouped(4) in ~> bcast ~> parser1 ~> merge ~> group ~> out bcast ~> parser2 ~> merge }.run()

Page 31: Akka Streams - From Zero to Kafka

IT'S BEEN EMOTIONAL...Slides at

Follow me

http://markglh.github.io/AkkaStreams-Madlab-Slides

@markglh