stream-style messaging development with rabbit, active, zeromq & apache kafka by vyacheslav...

1

Stream-style messaging development in JavaThe NoJMS way

October 1, 2015

2

Problems of traditional way

Integration system working is…

3

Problems of traditional way

Integration system working is…

4

Problems of traditional way

Integration system working is…

5

Problems of traditional way

Integration system working is…

6

Problems of traditional way

«There's no such thing as problems, Mr. Green, only situations...»

Avi, «Revolver»

7

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

8

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster

9

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster …

Second another application ServerCluster

10

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster …

Second another application ServerCluster

11

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster …

Second another application ServerCluster

12

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster …

Second another application ServerCluster

Problems of traditional way - Architecture

13

Internet/Intranet

Application ServerCluster

Database Cluster

First another application Server Cluster …

Second another application ServerCluster

Problems of traditional way - Architecture

14

Problems of traditional way

«Actually, the cause of computer revolution is not that calculator become faster and get a lot of memory, get

possibility to show as advanced graphics, and so on; but that cause is that computer can connect to another computer. So, computer is not, as usually described, the result of calculator

evolution – it`s result of telephone evolution!»

Andrey Verbitsky,«SoftwarePeople 2009»

15

Problems of traditional way

16

Problems of traditional way

WSDL/SOAP web-services (XML)1

17

Problems of traditional way

WSDL/SOAP web-services (XML)1

RESTful web-services (JSON)2

18

Problems of traditional way

WSDL/SOAP web-services (XML)1

RESTful web-services (JSON)2

JMS (Java standard Serialization)3

19

1. Messages size

Problems of traditional way - WSDL/SOAP web-services

20

1. Messages size

2. Validation via schema

Problems of traditional way - WSDL/SOAP web-services

21

1. Messages size

2. Validation via schema

3. Synchronous model

Problems of traditional way - WSDL/SOAP web-services

22

1. Messages size

2. Validation via schema

3. Synchronous model

4. HTTP(S)

Problems of traditional way - WSDL/SOAP web-services

23

1. Messages Size

Problems of traditional way – RESTful/JSON web-services

24

1. Messages Size

2. Validation?

Problems of traditional way – RESTful/JSON web-services

25

1. Messages Size

2. Validation?

3. Synchronous model

Problems of traditional way – RESTful/JSON web-services

26

1. Messages Size

2. Validation?

3. Synchronous model

4. HTTP(S)

Problems of traditional way – RESTful/JSON web-services

27

1. Java only!

Problems of traditional way – JMS

28

1. Java only!

2. Messages Size

Problems of traditional way – JMS

29

1. Java only!

2. Messages Size

3. Lots of code (except JMS 2.0 in JEE7)

Problems of traditional way – JMS

30

1. Java only!

2. Messages Size

3. Lots of code (except JMS 2.0 in JEE7)

4. Lost messages

Problems of traditional way – JMS

31

1. Java only!

2. Messages Size

3. Lots of code (except JMS 2.0 in JEE7)

4. Lost messages

5. Duplicates

Problems of traditional way – JMS

32

1. Enterprise Integration Patterns

2. Apache Camel

3. Spring Integration

Modern alternatives

33

Modern alternatives – Apache Camel

@Override public void configure() throws Exception { from("direct:start") .split().method("sectionRequestSplitterBean", "split") .aggregationStrategy(new ReportAggregationStrategy()) .transform().method("sectionRequestToXMLBean", "transform") .to(serviceURL) .transform().method("sectionResponseXMLToSectionBean", "transform"); }

public void setServiceURL(String serviceURL) { this.serviceURL = serviceURL; }

34

Modern alternatives

35

1. File Transfer

2. Shared Database

3. Remote Procedure Call

4. Messaging

Modern alternatives

36

Problems of coding

1. Synchronous vs Asynchronous

2. Systems Integration or Distributed System?

3. Do we really need a Bus (Mediator)?

4. Back pressure (Reactive Streams – Erlang, Scala, Java)

Modern alternatives

37

1. http://queues.io (~26 decisions)

2. Local BlockingQueue (“seda” in Apache Camel, JdbcChannelMessageStore in Spring Integration)

3. ZooKeeper

4. JCache (Hazelcast)

5. In-memory NoSQL DB (Redis, DertyDB,…)

6. AMQP (ActiveMQ, RabbitMQ)

7. ZeroMQ

8. Amazon SQS

9. Apache Kafka (ZooKeeper inside!)

Modern alternatives

38

Modern alternatives - AMQP (ActiveMQ, RabbitMQ)

39

Modern alternatives – Apache Kafka

40

Modern alternatives – Apache Kafka

1. Shared write-ahead log• partitions

41

Modern alternatives – Apache Kafka

42

Stream-style code

43

Stream-style code

@Autowired ParametrizedConsumer<OneFieldBean> simpleAmqpReceiver;

@Test public void priyomSoobshcheniy() throws Exception { simpleAmqpReceiver.objectStream(OneFieldBean.class) .forEach(object -> out.println(object.getS())); }

44

Stream-style code

private static int ITERATIONS_NUMBER = 1_000;

@Autowired private Sender<OneFieldBean> sender; private Benchmark benchmark = new Benchmark(out::println);

@Test public void prostaiaPosilkaSoobshcheniy() throws Exception { benchmark.benchmarkOperation(() -> { for (int i = 1; i < ITERATIONS_NUMBER; ) sender.sendMessage(new SimpleOneFieldBean().setS("тест " + i++)); return "done!"; }, "Послано за (сек.): "); }

45

Stream-style code

private static int ITERATIONS_NUMBER = 1_000;

@Test public void posilkaSoobshcheniyCherezStream() { benchmark.benchmarkOperation(() -> { sender.streamSending( new Iterator<OneFieldBean>() { AtomicInteger i = new AtomicInteger(0); @Override public boolean hasNext() { return i.get() < ITERATIONS_NUMBER; } @Override public OneFieldBean next() { return new SimpleOneFieldBean().setS("тест " + i.getAndIncrement()); } } ); return "done!"; }, "Послано за (сек.): "); }

46

Stream-style code

@POST@Produces({“application/x-msgpack; qs=1”, “application/json; qs=0.75”}) public Order getOrder() { // … }