Real-time Analytics with Netty, Apache Kafka and StormCase study with “lambda architecture”

http://nguyentantrieu.infoUpdate: 07/06/2013

Agenda1. Overview Architecture2. Log HTTP-Handler and producer: Netty 43. Kafka 0.8 (Stream Data Log Storage)4. Storm Analytics Cluster

Overview System Architecture

Concept Flow

Concept FlowJavaScript Tracking

Mobile SDK

Http Log Server

Kafka

S2 HTTP Log Servernetty framework 4

Netty.ioNetty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming.

http://en.wikipedia.org/wiki/Netty_(software)http://nguyentantrieu.info/blog/backend-system-with-netty-io

Apache Kafka (version 0.8)https://cwiki.apache.org/confluence/display/KAFKA/Index

In Production● Clustering ( 4 nodes)● Partitions

○ user-activity: 24 partitions● Producer and consumer are replication

aware

Core Producer Class (S2 HTTP Log Server)

Storm Analytics Cluster

The Storm Topology

Kafka Clustertopic: user-activity

Tokenizer Bolt

Parser Bolt

Aggregate Bolt

Redis Statistics Bolt

Save DWH Bolt

Raw Data

Kafka Consumer Spout

Useful Links● https://storm.incubator.apache.org● https://www.youtube.com/watch?

v=DRgs8L4AOgc● https://cwiki.apache.

org/confluence/display/KAFKA/Index● http://nguyentantrieu.info/blog/backend-

system-with-netty-io●