exploring reactive integrations with akka streams, alpakka and apache kafka

66
akka streams, Exploring Reactive Integrations with Alpakka and Kafka Konrad `ktoso` Malawski

Upload: lightbend

Post on 22-Jan-2018

8.750 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streams,

Exploring Reactive Integrations with

Alpakka and Kafka

Konrad `ktoso` Malawski

Page 2: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Konrad `ktoso` MalawskiAkka Team, Reactive Streams TCK, Persistence, HTTP, Remoting / Cluster

Page 3: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Make building powerful concurrent & distributed applications simple.Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM

Page 4: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community

Complete Java & Scala APIs for all features

What’s in the toolkit?

Page 5: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 6: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

“Stream”has many meanings

Page 7: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streamsAsynchronous back pressured stream processing

Source Sink

Flow

Page 8: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streamsAsynchronous back pressured stream processing

Source Sink

(possible) asynchronous

boundaries

Flow

Page 9: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streamsAsynchronous back pressured stream processing

Source Sink

10 msg/s 1 msg/s

OutOfMemoryError!!

Flow

Page 10: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streamsAsynchronous back pressured stream processing

Source Sink

10 msg/s 1 msg/s

hand me 3 morehand me 3 more

1 msg/s Flow

Page 11: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

akka streamsNot only linear streams

Source

SinkFlow

SourceSink

FlowFlow

Page 12: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

And the many meanings it carries.

Reactive

Page 13: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

The many meanings of Reactive

reactivemanifesto.org

Page 14: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

The many meanings of Reactive

Page 15: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 16: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 17: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

“Not-quite-Reactive-System” The reason we started researching into transparent to users flow control.

Page 18: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 19: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 20: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 21: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 22: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 23: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka
Page 24: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols

http://www.reactive-streams.org

Page 25: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive StreamsA buiding-block of Reactive Systems, not the “entire story”.

Page 26: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols

http://www.reactive-streams.org

Page 27: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Part of JDK 9java.util.concurrent.Flow

http://openjdk.java.net/projects/jdk9/

Page 28: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

JEP-266 – soon…!public final class Flow { private Flow() {} // uninstantiable

@FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); }

public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); }

public static interface Subscription { public void request(long n); public void cancel(); }

public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { }}

Page 29: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive Streams

RS Library A RS library B

async boundary

Page 30: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive Streams

RS Library A RS library B

async boundary

Make building powerful concurrent & distributed applications simple.

Page 31: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

The APIAkka Streams

Complete and awesome Java and Scala APIs (Just like everything in Akka)

Page 32: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Akka Streams in 20 seconds:

Source<Integer, NotUsed> source = null; Flow<Integer, String, NotUsed> flow = Flow.<Integer>create().map((Integer n) -> n.toString()); Sink<String, CompletionStage<Done>> sink = Sink.foreach(str -> System.out.println(str)); RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);runnable.run(materializer);

Page 33: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Akka Streams in 20 seconds:

CompletionStage<String> firstString = Source.single(1) .map(n -> n.toString()) .runWith(Sink.head(), materializer);

Page 34: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Akka Streams in 20 seconds:

Page 35: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Akka Streams in 20 seconds:

Page 36: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Materialization

Gears from GeeCON.org,(it’s an awesome conf)

Page 37: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

What is “materialization” really?

Page 38: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

What is “materialization” really?

Page 39: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

What is “materialization” really?

Page 40: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Introspection, and more coming…

Highly sophisticated stream introspection capabilities.

Page 41: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

AlpakkaA community for Streams connectors

http://blog.akka.io/integrations/2016/08/23/intro-alpakka

Page 42: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Alpakka – a community for Stream connectors

Threading & Concurrency in Akka Streams Explained (part I)

Mastering GraphStages (part I, Introduction)

Akka Streams Integration, codename Alpakka

A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II)

Writing Akka Streams Connectors for existing APIs

Flow control at the boundary of Akka Streams and a data provider

Akka Streams Kafka 0.11

Page 43: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Alpakka – a community for Stream connectors

http://developer.lightbend.com/docs/alpakka/current/

Page 44: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Alpakka – a community for Stream connectors

Demo

Page 45: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Akka Streams & HTTP

streams& HTTP

Page 46: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a ______ protocol?

Page 47: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a STREAMING protocol!

Page 48: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming in Akka HTTP

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

HttpServer as a: Flow[HttpRequest, HttpResponse]

Page 49: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming in Akka HTTP

HttpServer as a: Flow[HttpRequest, HttpResponse]

HTTP Entity as a: Source[ByteString, _]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Page 50: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming in Akka HTTP

HttpServer as a: Flow[HttpRequest, HttpResponse]

HTTP Entity as a: Source[ByteString, _]

Websocket connection as a: Flow[ws.Message, ws.Message]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Page 51: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Resource efficiency and dynamic fan-out

Page 52: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Resource efficiency and dynamic fan-out

Page 53: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

It’s turtles buffers all the way down!

xkcd.com

Page 54: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming from Akka HTTP

Page 55: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming from Akka HTTP

Page 56: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

}

Page 57: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

}

Page 58: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Streaming from Akka HTTP (Scala)object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer()

implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet)

def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080");}

Page 59: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Next steps for Akka

Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron).

More integrations for Akka Streams stages, project Alpakka.

Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski

Akka Typed progressing again, likely towards 3.0.

Akka HTTP 2.0 Proof of Concept in progress.

Collaboration with Reactive Sockets

Page 60: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Ready to adopt on prod?

Page 61: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Totally, go for it.

Page 62: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Akka <3 contributionsEasy to contribute tickets:

https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

Akka Stream Contrib https://github.com/akka/akka-stream-contrib

Mailing list: https://groups.google.com/group/akka-user

Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka

Page 63: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

More resources:

Read more tutorials and deep-dives:

http://blog.akka.io/ https://www.lightbend.com/resources

Page 64: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive PlatformReactive Platform

Reactive Platform

Page 65: Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Further reading:

Reactive Streams: reactive-streams.org Akka documentation: akka.io/docs Free O’Reilly report – bit.ly/why-reactive

Example Sources: ktoso/akka-streams-alpakka-talk-demos-2016

Contact: Konrad [email protected] Malawski http://kto.so / @ktosopl