introduction to apache apex
TRANSCRIPT
Pramod Immaneni <[email protected]>PPMC Member, Architect @DataTorrent IncApr 5th, 2016
The next generation native Hadoop platform
Introduction to Apache Apex (incubating)
© 2015 DataTorrent2
What is Apex• Platform and framework to build scalable and fault-tolerant distributed applications
• Hadoop native• Build any custom logic in your application
• Unobtrusive API to facilitate distributed application development
• Runtime engine to ensure fault tolerance, scalability and data flow
• Process streaming or batch big data• High throughput and low latency
• Realtime applications
© 2015 DataTorrent3
Applications on Apex• Distributed processing
• Application logic broken into components called operators that run in a distributed fashion across your cluster
• Natural programming model• Code as if you were writing normal Java logic• Maintain state in your application variables
• Scalable• Operators can be scaled up or down at runtime according to the load and SLA
• Fault tolerant• Automatically recover from node outages without having to reprocess from
beginning• State is preserved• Long running applications
• Operational insight – DataTorrent RTS• See how each operator is performing and even record data
© 2015 DataTorrent4
Apex Platform Overview
© 2015 DataTorrent5
Apache Malhar Library
© 2015 DataTorrent6
Native Hadoop Integration
• YARN is the resource manager
• HDFS used for storing any persistent state
© 2015 DataTorrent7
Application Development Model
A Stream is a sequence of data tuplesA typical Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library• Operator has many instances that run in parallel and each instance is single-threaded
Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Filtered
Stream
Output StreamTuple Tuple
Filtered Stream
Enriched Stream
Enriched
Stream
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
© 2015 DataTorrent8
Advanced Windowing Support
Application window Sliding window and tumbling window
Checkpoint window No artificial latency
© 2015 DataTorrent9
Application in Java
© 2015 DataTorrent10
Operators
© 2015 DataTorrent11
Operators (contd)
© 2015 DataTorrent12
Partitioning and unificationNxM PartitionsUnifier
0 1 2 3
Logical DAG
0 1 2
1
1 Unifier
1
20
Logical Diagram
Physical Diagram with operator 1 with 3 partitions
0
Unifier
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneck
Unifier
Unifier0
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier
© 2015 DataTorrent13
Advanced Partitioning
0
1a
1b
2 3 4Unifier
Physical DAG
0 4
3a2a1a
1b 2b 3b
Unifier
Physical DAG with Parallel Partition
Parallel Partition
Container
uopr
uopr1
uopr2
uopr3
uopr4
uopr1
uopr2
uopr3
uopr4
dopr
dopr
doprunifier
unifier
unifier
unifier
Container
Container
NIC
NIC
NIC
NIC
NIC
Container
NIC
Logical Plan
Execution Plan, for N = 4; M = 1
Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiers
Cascading Unifiers
0 1 2 3 4
Logical DAG
© 2015 DataTorrent14
Dynamic Partitioning
• Partitioning change while application is runningᵒ Change number of partitions at runtime based on statsᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitionsᵒ Supports re-distribution of state when number of partitions changeᵒ API for custom scaler or partitioner
2b
2c
3
2a
2d
1b
1a1a 2a
1b 2b
3
1a 2b
1b 2c 3b
2a
2d
3a
Unifiers not shown
© 2015 DataTorrent15
How tuples are partitioned• Tuple hashcode and mask used to determine destination
partitionᵒ Mask picks the last n bits of the hashcode of the tupleᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuplesᵒ Can also be used for specifying custom serialization
tuple: {Name, 24204842, San Jose}
Hashcode: 001010100010101
Mask (0x11)
Partition
00 1
01 2
10 3
11 4
© 2015 DataTorrent16
Custom partitioning• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{Name, 24204842, San Jose}
Hashcode: 001010100010101
Mask (0x00)
Partition
00 1
00 2
00 3
00 4
© 2015 DataTorrent17
Fault Tolerance• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint stateᵒ Frequency configurable per operatorᵒ Asynchronous and distributed by defaultᵒ Default store is HDFS
• Automatic detection and recovery of failed operatorsᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data
• Application master state checkpointed
© 2015 DataTorrent18
Processing GuaranteesAtleast once• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lostᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following mechanisms to achieve exactly-once behavior in fault recovery scenariosᵒ Transactions with meta information, Rewinding output, Feedback
from external entity, Idempotent operationsAtmost once• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient
Exactly once• At least once + state recovery + operator logic to achieve
end-to-end exactly once
© 2015 DataTorrent19
Stream Locality• By default operators are deployed in containers (processes)
randomly on different nodes across the Hadoop cluster
• Custom locality for streamsᵒ Rack local: Data does not traverse network switchesᵒ Node local: Data is passed via loopback interface and frees up
network bandwidthᵒ Container local: Messages are passed via in memory queues
between operators and does not require serializationᵒ Thread local: Messages are passed between operators in a same
thread equivalent to calling a subsequent function on the message
© 2015 DataTorrent20
Data Processing Pipeline ExampleApp Builder
© 2015 DataTorrent21
Monitoring ConsoleLogical View
© 2015 DataTorrent22
Monitoring ConsolePhysical View
© 2015 DataTorrent23
Real-Time DashboardsReal Time Visualization
© 2015 DataTorrent
Resources
24
• Apache Apex - http://apex.apache.org/• Subscribe - http://apex.apache.org/community.html• Download - https://www.datatorrent.com/download/• Twitter
ᵒ @ApacheApex; Follow - https://twitter.com/apacheapexᵒ @DataTorrent; Follow – https://twitter.com/datatorrent
• Meetups - http://www.meetup.com/topics/apache-apex• Webinars - https://www.datatorrent.com/webinars/• Videos - https://www.youtube.com/user/DataTorrent• Slides - http://www.slideshare.net/DataTorrent/presentations • Startup Accelerator Program - Full featured enterprise product
ᵒ https://www.datatorrent.com/product/startup-accelerator/
© 2015 DataTorrent
We Are Hiring
25
• [email protected]• Developers/Architects• QA Automation Developers• Information Developers• Build and Release• Community Leaders
End
26
© 2015 DataTorrent
Extra Slides
© 2015 DataTorrent28
Application Programming Model
A Stream is a sequence of data tuplesAn Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library• Operator has many instances that run in parallel and each instance is single-threaded
Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Filtered Stream
Output StreamTuple Tuple
Filtered Stream
Enriched Stream
Enriched
Stream
er
Operator
er
Operator
er
Operator
er
Operator
© 2015 DataTorrent29
Partitioning and Scaling Out
• Operators can be dynamically scaled• Flexible Streams split• Parallel partitioning
• MxN partitioning • Unifiers
© 2015 DataTorrent30
Fault Tolerance OverviewStateful Fault Tolerance Processing Semantics Data Locality
Supported out of the box– Application state– Application master state– No data loss
Automatic recovery Lunch test Buffer server
At least once At most once Exactly once
Stream locality for placement of operators
Rack local – Distributed deployment
Node local – Data does not traverse NIC
Container local – Data doesn’t need to be serialized
Thread local – Operators run in same thread
Data locality
© 2015 DataTorrent31
Machine Data ApplicationLogical View
© 2015 DataTorrent32
Machine Data ApplicationPhysical View