© 2016 DataTorrent

Chinmay KolhatkarCommitter, Apache Apex

Engineer, DataTorrentMarch 23, 2017

Apache Apex-Bigtop

© 2016 DataTorrent

Agenda

2

•About Apache Apex•Apex Platform Overview•Apex - Native Hadoop Integration•Apex Malhar Library•Apex as a Bigtop component• Installing Bigtop Apex•Apex Docker sandbox•Apex Docker sandbox Demo

© 2016 DataTorrent

About Apache Apex

3

•Platform and runtime engine that enables development of scalable and fault-tolerant distributed applications

•Hadoop native (Hadoop >= 2.2)No separate service to manage stream processingStreaming Engine built into Application Master and

Containers•Process streaming or batch big data•High throughput and low latency•Library of commonly needed business logic•Write any custom business logic in your application

© 2016 DataTorrent

Apex Platform Overview

4

An Apex Application is a DAG(Directed Acyclic Graph)

A DAG is composed of vertices (Operators) and edges (Streams).A Stream is a sequence of data tuples which connects operators at end-points called PortsAn Operator takes one or more input streams, performs computations & emits one or more output streams

● Each operator is USER’s business logic, or built-in operator from our open source library● Operator may have multiple instances that run in parallel

© 2016 DataTorrent

Apex - Native Hadoop Integration

6

• YARN is the resource manager

• HDFS used for storing any persistent state

© 2016 DataTorrent

Apex Malhar Library

7

RDBMS• Vertica• MySQL• Oracle• JDBC

NoSQL• Cassandra, Hbase• Aerospike, Accumulo• Couchbase/ CouchDB• Redis, MongoDB• Geode

Messaging• Kafka• Solace• Flume, ActiveMQ• Kinesis, NiFi

File Systems• HDFS/ Hive• NFS• S3

Parsers• XML • JSON• CSV• Avro• Parquet

Transformations• Filters• Rules• Expression• Dedup• Enrich

Analytics• Dimensional Aggregations

(with state management for historical data + query)

Protocols• HTTP• FTP• WebSocket• MQTT• SMTP

Other• Elastic Search• Script (JavaScript, Python, R)• Solr• Twitter

© 2016 DataTorrent

Apex as Bigtop component

8

•Uses Bigtop framework for ease of deploymentDeployment using puppet recipes and VagrantCan spawn multiple node clusters for docker, VM &

OpenStack

•Generates a deployable binaries for Apex engineRPM - CentOS 6 & 7, Fedora 25, OpenSuse 42.1DEB - Ubuntu 16.04, Debian 8

•Allows validating installationsPackage TestSmoke Test

© 2016 DataTorrent

•Add Bigtop Repositoryhttp://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/

• Install bigtop-hadoopFor Debian: apt-get install hadoop\*For RPM: yum install hadoop\*

•Download bigtop-apex from bigtop CIhttps://ci.bigtop.apache.org/job/Bigtop-trunk-packages/

• Install Apex:For Debian: dpkg -i apex_3.4.0-1_all.debFor RPM: rpm -i apex-3.5.0-1.el7.noarch.rpm

Installing Bigtop Apex Bigtop 1.1.0 (Current)

9

© 2016 DataTorrent

•Add Bigtop Repository (Future URL)http://www.apache.org/dist/bigtop/bigtop-1.2.0/repos/

• Install apexFor Debian: apt-get install apexFor RPM: yum install apex

Installing Bigtop Apex Bigtop 1.2.0 (Next Release)

10

© 2016 DataTorrent

•A quick starter Apex docker image: https://hub.docker.com/r/apacheapex/sandbox/

•Preconfigured and running componentsHDFS (namenode, secondarynamenode, datanode)YARN (resourcemanager, nodemanager, timelineserver)

•Preconfigured and installed componentApex

•Get started:Step1: docker pull apacheapex/sandboxStep2: docker run -it apacheapex/sandbox

Apex Docker sandbox

11

© 2016 DataTorrent

Apex Docker sandbox (contd.)

12

© 2016 DataTorrent

Resources

13

• Apache Apex website - http://apex.apache.org/• Subscribe - http://apex.apache.org/community.html• Download - http://apex.apache.org/downloads.html• Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex• Facebook - https://www.facebook.com/ApacheApex/• Meetup - http://www.meetup.com/topics/apache-apex•SlideShare - http://www.slideshare.net/ApacheApex/presentations•More Examples - https://github.com/DataTorrent/examples• Startup Program – Free Enterprise License for Startups, Educational

Institutions, Non-Profits - https://www.datatorrent.com/startups/•Cloud Trial - https://www.datatorrent.com/download/cloud-trial/