stream-style messaging development with rabbit, active, zeromq & apache kafka by vyacheslav...

1

Stream-style messaging development in JavaThe NoJMS way

October 1, 2015

2

Problems of traditional way

Integration system working is…

3



4



5



6


«There's no such thing as problems, Mr. Green, only situations...»

Avi, «Revolver»

7

Problems of traditional way - Architecture

Internet/Intranet

Application ServerCluster

Database Cluster

8


Internet/Intranet


Database Cluster

First another application Server Cluster

9


Internet/Intranet


Database Cluster

First another application Server Cluster …

Second another application ServerCluster

10


Internet/Intranet


Database Cluster



11


Internet/Intranet


Database Cluster



12

Internet/Intranet


Database Cluster




13

Internet/Intranet


Database Cluster




14


«Actually, the cause of computer revolution is not that calculator become faster and get a lot of memory, get

possibility to show as advanced graphics, and so on; but that cause is that computer can connect to another computer. So, computer is not, as usually described, the result of calculator

evolution – it`s result of telephone evolution!»

Andrey Verbitsky,«SoftwarePeople 2009»

15


16


WSDL/SOAP web-services (XML)1

17



RESTful web-services (JSON)2

18



RESTful web-services (JSON)2

JMS (Java standard Serialization)3

19

1. Messages size

Problems of traditional way - WSDL/SOAP web-services

20

1. Messages size

2. Validation via schema


21

1. Messages size


3. Synchronous model


22

1. Messages size



4. HTTP(S)


23

1. Messages Size

Problems of traditional way – RESTful/JSON web-services

24

1. Messages Size

2. Validation?


25

1. Messages Size

2. Validation?



26

1. Messages Size

2. Validation?


4. HTTP(S)


27

1. Java only!

Problems of traditional way – JMS

28

1. Java only!

2. Messages Size


29

1. Java only!

2. Messages Size

3. Lots of code (except JMS 2.0 in JEE7)


30

1. Java only!

2. Messages Size


4. Lost messages


31

1. Java only!

2. Messages Size


4. Lost messages

5. Duplicates


32

1. Enterprise Integration Patterns

2. Apache Camel

3. Spring Integration

Modern alternatives

33

Modern alternatives – Apache Camel

@Override public void configure() throws Exception { from("direct:start") .split().method("sectionRequestSplitterBean", "split") .aggregationStrategy(new ReportAggregationStrategy()) .transform().method("sectionRequestToXMLBean", "transform") .to(serviceURL) .transform().method("sectionResponseXMLToSectionBean", "transform"); }

public void setServiceURL(String serviceURL) { this.serviceURL = serviceURL; }

34

Modern alternatives

35

1. File Transfer

2. Shared Database

3. Remote Procedure Call

4. Messaging

Modern alternatives

36

Problems of coding

1. Synchronous vs Asynchronous

2. Systems Integration or Distributed System?

3. Do we really need a Bus (Mediator)?

4. Back pressure (Reactive Streams – Erlang, Scala, Java)

Modern alternatives

37

1. http://queues.io (~26 decisions)

2. Local BlockingQueue (“seda” in Apache Camel, JdbcChannelMessageStore in Spring Integration)

3. ZooKeeper

4. JCache (Hazelcast)

5. In-memory NoSQL DB (Redis, DertyDB,…)

6. AMQP (ActiveMQ, RabbitMQ)

7. ZeroMQ

8. Amazon SQS

9. Apache Kafka (ZooKeeper inside!)

Modern alternatives

http://queues.io/

38

Modern alternatives - AMQP (ActiveMQ, RabbitMQ)

39

Modern alternatives – Apache Kafka

40


1. Shared write-ahead log• partitions

41


42

Stream-style code

43

Stream-style code

@Autowired ParametrizedConsumer<OneFieldBean> simpleAmqpReceiver;

@Test public void priyomSoobshcheniy() throws Exception { simpleAmqpReceiver.objectStream(OneFieldBean.class) .forEach(object -> out.println(object.getS())); }

44

Stream-style code

private static int ITERATIONS_NUMBER = 1_000;

@Autowired private Sender<OneFieldBean> sender; private Benchmark benchmark = new Benchmark(out::println);

@Test public void prostaiaPosilkaSoobshcheniy() throws Exception { benchmark.benchmarkOperation(() -> { for (int i = 1; i < ITERATIONS_NUMBER; ) sender.sendMessage(new SimpleOneFieldBean().setS("тест " + i++)); return "done!"; }, "Послано за (сек.): "); }

45

Stream-style code

private static int ITERATIONS_NUMBER = 1_000;

@Test public void posilkaSoobshcheniyCherezStream() { benchmark.benchmarkOperation(() -> { sender.streamSending( new Iterator<OneFieldBean>() { AtomicInteger i = new AtomicInteger(0); @Override public boolean hasNext() { return i.get() < ITERATIONS_NUMBER; } @Override public OneFieldBean next() { return new SimpleOneFieldBean().setS("тест " + i.getAndIncrement()); } } ); return "done!"; }, "Послано за (сек.): "); }

46

Stream-style code

@POST@Produces({“application/x-msgpack; qs=1”, “application/json; qs=0.75”}) public Order getOrder() { // … }

stream-style messaging development with rabbit, active, zeromq & apache kafka by vyacheslav...

Technology