Core concepts and Key technologies - Big Data Analytics!!Big Data Business has solved the problem of 'Big data acquisition and persistence' using daily ETL and Batch Analysis through the Hadoop Eco-system.!!Now lets now see how the Big Data Market evolved beyond batch processing (Hadoop , RDBMS - world) to extract intelligence from global data streams in real time !!!Tremendous Research work underway , last few years , to challenge conventional wisdom and create the augmented reality of Big Data Analytics !!!'Dynamic Decision Making' is no longer driven by 'Traditional Business Intelligence' but involves 'Fast exploration of data patterns', 'performing complex deterministic or stochastic approximations or accurate queries' !!!Lets glance through some of the core concepts and key technologies that are driving this Renaissance.!!Need to preserve Data Locality !!

Traditional Hadoop MR does not preserve 'data locality' during Map Reduce transition or between iterations. In order to send data to the next job in a MR workflow, a MR job needs to store its data in HDFS. So it incurs communication overhead and extra processing time. Bulk Synchronous Parallel - concept was implemented in (Pregel, Giraph, Hama) to solve this very important MR problem. !Hama initiates a peer2peer communication only when necessary and peers focus on keeping locally processed data in local nodes.!!BSP manages the synchronization and communication in the middle layer as opposed to file-system based parallel random access pattern ! It uses the K-Means Clustering algorithm. Apache Hama provides a stable reference implementation for analyzing streaming events or big data with graph/network structure by implementing deadlock-free 'message passing interface' and 'barrier synchronization' (reduces significant n/w overheads)!!

Need for Real-time processing and streaming ETL!!Hadoop was purpose-built for 'distributed batch processing' using static input, output and processor configuration. But fast ad-hoc machine learning query requires real-time distributed processing and real-time updates based on dynamically changing configuration without requiring any code change. Saffron Memory Base (SMB) is a technology that offers real-time analytic on hybrid data.!!SQLstream Connector for Hadoop provides bi-directional, continuous integration with Hadoop HBase. DataTorrent Apex is another front-runner in Stream Processing.!!With SciDB, one can run a query the moment it occurs to the user. By contrast, arguably Hadoop enforces a huge burden of infrastructure setup, data preparation, map-reduce

configuration and architectural coding. Both SciDB and SMB positions themselves as a complete replacement of Hadoop-MR when it comes to complex data analysis.!!

Need for complex analytic functions!!Its not suitable for increasingly complex mathematical and graphical functions like Page-Ranking, BFS, Matrix-Multiplication which require repetitive MR Jobs.!!So many interesting research works spawned in recent time; BSP, Twister, Haloop, RHadoop!!

Need for rich Data Model and rich Query syntax!!Existing MR Query API has limited syntax for relational joins and group-bys. They do not directly support iteration or recursion in declarative form and are not able to handle complex semi-structured nested scientific data. Here comes MRQL (Map-Reduce Query Language) to the rescue of Hadoop MR by supporting nested collections, trees, arbitrary query nesting, and user-defined types and functions.!!Impala : read directly from HDFS and HBase data. it will add a columnar storage engine, cost-based optimizer and other distinctly database-like features.!!Need to optimize data flow and query execution!!All the 'Big data Analytics datastores' both proprietery and open-sourced trying their best to redefine the 'traditional Hadoop MR'!!Well MapReduce does not make sense as an engine for querying !!!So here comes Shark is a Distributed In-Memory MR framework with great speed of Execution!!

Need for speed and versatility of MR Query!!Apache Hadoop is designed to achieve very high throughput, but is not designed to achieve the sub-second latency needed for interactive data analysis and exploration. Here comes Google Dremel and Apache Drill. Columnar query execution engine offers low latency interactive reporting using DrQL which encompasses a broad range of low latency frameworks like Mongo Query, Cascading, Plume. It adheres to Hadoop philosophy of connecting to multiple storage systems, but broadens the scope and introduces enormous flexibility through supporting multiple query languages, data formats and data sources. Spark extends Hive syntax but employs a very efficient column-storage with interactive multi-stage computing flow.!!

Ability to cache and reuse intermediate map outputs!!HaLoop introduces recursive joins for effective iterative data analysis by caching and reusing loop-independent data across iterations. A significant improvement over conventional general-purpose MapReduce.!

!Twister offers a modest mechanism for managing configurable and cacheable.mr tasks and implements effective pub/sub based communication and off course special support for 'iterative MR computations'.!!

Leverage CPU cache and distributed memory access patterns!!There are quite a few frameworks to store data in distributed memory instead of HDFS like GridGain, HazelCast, RDD, Piccolo!!Hadoop was not designed to facilitate interactive analytics. So it required few game changers to exploit 'CPU cache' and 'Distributed Memory’!!

Single-bus Multi-core CPU!!It's a well-known fact how Single-bus Multi-core CPU offers simultaneous multi-threading that significantly reduces latency for certain type of algorithm and data structures ! So DBMS being re-architected from ground up to leverage the 'cpu-bound partitioning-phase hash-join' as opposed to 'memory-bound hash-join' !!!Its the ever-increasing speed of CPU caches and TLBs which allow blazing fast computation and retrieval of hashed result. Also its noteworthy how modern multi-core CPU and GPGPU offer cheap compression schemes at virtually no CPU cost. As we know access to memory becoming pathetically slower compared to the ever galloping processor clock-speed!!!ElasticCube from SiSense leverages 'query plans optimized for fast response time and parallel execution based on multi-cores' and continuous 'instruction recycling' for reusing pre-computed results.!!Few other Business Analytics database/tools like VectorWise, Tableau Data Engine and SalesEdge also make great use of CPU cache to offer blazing fast ad-hoc query.!!!

Implementing Parallel Vectorization of compressed data through SIMD (Single-Instruction, Multiple Data) : !!

VectorWise efficiently utilized the techniques of vectorization, cpu compression and 'using cpu as execution memory'. This is also a core technology behind many leading analytics column stores.!!

Driving Positional-Delta-Tree (PDT) : !!PDT stores both position and the delta are stored in memory and effectively merged with data during optimized query execution. Ad-hoc query in most cases is about identifying the 'difference'. VectorWise makes effective use of PDT. More can be found in its white paper.!!

!Need for directly query compressed data residing in heavily indexed columnar files : !!

SSDs and Flash storages will get cheaper (means more cheaper Cloud service) with innovative compression and de-duplication on file system. All the analytics datastores are gearing up to make most of this feature. Watch out for Pure Storage .!!

Need for Dynamic Memory Computation and Resource Allocation!!Output of batched job need to be dumped in secondary storage. Rather it would be good idea to constantly compute the data size and create peer processors and make sure collective memory does not exceed entire data size. So its important to understand Hadoop-MR is not the best-fit for processing all types data structures ! BSP model should be adopted for massive Graph processing where bulk of static data can remain in filesystem while dynamic data processed by peers and result kept in memory.!!!

Usage of Fractal Tree Indexes : !!Local processing is the key ! Keep enough buffers and pivots in the Tree node itself in order to avoid frequent costly round trips along the tree for individual items! That means keep filling up your local buffer and then do bulk flush! New age drives love bulk updates (more changes per write) to avoid fragmentation !!!TokuDB replaces MySQL and MongoDB binary tree implementation with Fractral tree and achieved massive performance gain.!!

Distributed Shared Memory Abstraction !!It provides a radical performance improvement over disk-based MR ! Partial DAG execution (Directed Acyclic Graph) model to describe parallel processing for for in-memory computation (aggregate result set that fits in memory e.g. like intermediate Map outputs).!!Spark uses 'Resilient Distributed Dataset' architecture for converting query into 'operator tree' ! Shark keeps on reoptimizing a running query after running first few stages of the task DAG, thereby selecting better Join strategy and right degree of parallelism. Shark also offers 'co-partitioning multiple tables based on common key' for faster join query!!!It leverages SSD, CPU cores and Main Memory to the fullest extent !!!Its worth mentioning how DDF (Distributed DataFrame - nexgen extension of Spark RDD) , H2O DataFrame and Dato SFrame expanding the horizons machine learning at massive scale.!!

Parallel Array Computation !!

It is a very simplistic yet powerful mathematical approach to embed Big Math functions directly inside database engine. SciDB has mastered this concept by embedding statistical computation using distributed, multidimensional arrays.!!

Semi-computation and instant approximation!!'fast response to ad-hoc query' through Continuous learning from experience and instant approximation as opposed to waiting for the end of processing and computation of final result . A bunch of analytics products coming to market with built-in 'data science capabilities' - for example H20 from 0xdata .!!!

Need for Push-based Map Reduce Resource Management!!Though Hadoop's main strength is distributed processing over clusters, but at peak load the utilization drops due to scheduling overhead. Its well-known how map-reduce cluster is divided into fixed number of processor slots based on static configuration. !So Facebook introduced Corona - a 'push-based scheduling where a cluster-manager tracks nodes, dynamically allocates a slot making it easier to utilize all the slots based on cluster workload for both map-reduce and non-mr applications.!!

Topological Data Analysis !!It is a giant leap forward ! It treat data model as a topology of nodes and discovers patterns and results by measuring similarity ! Ayasdi has pioneered this idea to build the first 'Query-free exploratory analytics tool' ! Its a true example of 'analytics based on unsupervised learning without requiring a priori algebraic model'.!!Ayasdi Iris is a mind-boggling insight discovery tool !!!

Learn from KnowledgeBase!!InfoBright offers fast analytics based on the Knowledge-base built by continuously updating metadata about data, data access patterns, query, aggregate result. !This type of innovative 'dynamic introspective' approach helps columnar storage to avoid requirement for indexing and costly sub-selects and allows to decompress only the required data !!!!

Reduction in Cluster size!!'Data Analysis' processors do not need the same number of machines like Hadoop nodes. For example ParAccel can handle data analysis in one node as opposed to (avg) 8 nodes required by Hadoop to perform same type of analysis due to advanced storage schema optimization. Enhanced File systems like QFS offer much lower replication factor (1.5) and higher throughput.!!!

Avoid Data duplication !!Both ParAccel and Hadapt share a similar vision of analyzing the data as close to the data node as possible without moving data to different BI layer.!!As opposed to Hadoop Connector strategies of MPP analytic datastores, Hadapt processes data in an RDBMS layer sitting close to HDFS and load-balances queries in a virtualized environment of adaptive query engines.!!

Ensure Fault-tolerance & Reliability !!Apache Hama ensures reliability through Process Communication and Barrier Synchronization. Each peer uses checkpoint recovery to occasionally flush the volatile part of its state to the DFS and allows rollback to the last checkpoint in the event of failure.!!Per Shark documentation, "RDDs track the series of transformations used to build them (their lineage) to recompute lost data "!! Shark/Spark with its compact Resilient Data Set -based distributed in-memory computing offers a great hope to startups and open source enthusiasts for building a lightning fast data warehouse system.!!

References : !!** Single-Bus-Multi-Core CPU : http://pages.cs.wisc.edu/~jignesh/publ/hashjoin.pdf !** Bulk Synchronous Parallel : http://www.staff.science.uu.nl/~bisse101/Book/PSC/psc1_2.pdf !!Spark : http://spark-project.org/research/ !Shark : https://github.com/amplab/shark !Pregel : http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html !Giraph : http://giraph.apache.org/ !Hama : http://people.apache.org/~edwardyoon/papers/Apache_HAMA_BSP.pdf !Saffron Memory Base : http://www.slideshare.net/paulhofmann/big-data-and-saffron !SQL Stream : http://www.sqlstream.com/applications/hadoop/ !RHadoop : https://github.com/RevolutionAnalytics/RHadoop/wiki !MRQL : http://lambda.uta.edu/mrql/ !Apache Drill : http://incubator.apache.org/drill/ !HaLoop : http://code.google.com/p/haloop/wiki/UserManual !Twister : http://www.iterativemapreduce.org/ !BSP : http://en.wikipedia.org/wiki/Bulk_synchronous_parallel !Corona : https://github.com/facebook/hadoop-20/tree/master/src/contrib/corona !ParAccell : http://img.en25.com/Web/ParAccel/%7Be72a7284-edb0-4e58-bb75-ff1145717d2b%7D_Hadoop-Limitations-for-Big-Data-ParAccel-Whitepaper.pdf !QFS : https://github.com/quantcast/qfs/wiki/Performance-Comparison-to-HDFS !Hadapt : http://hadapt.com/assets/Hadapt-Product-Overview1.pdf !ElasticCube : http://pages.sisense.com/elasticube-whitepaper.html?src=bottom !VectorWise : http://fastreporting.files.wordpress.com/2011/03/vectorwise-whitepaper.pdf !

SalesEdge : http://edgespring.com/product.php !TokuDB : http://www.tokutek.com/resources/technology/ !Ayasdi Iris : http://www.ayasdi.com/rethink-data/ !InfoBright : http://support.infobright.com/Support/Resource-Library/Whitepapers/ !SciDB : http://www.paradigm4.com/2013/01/terabyte-scale-parallel-processing-with-r-and-scidb/ !Saffron Memory Base : http://www.slideshare.net/paulhofmann/saffron-for-cloud-con !Hama : http://www.slideshare.net/teofili/machine-learning-with-apache-hama !H2O : http://www.0xdata.com/faq.html !!http://pages.cs.wisc.edu/~jignesh/publ/hashjoin.pdf !http://www.sisense.com/documentation/prism-elasticube-manager/introduction-to-elasticube-manager , !http://kowshik.github.com/JPregel/pregel_paper.pdf!http://en.wikipedia.org/wiki/Topological_sorting!http://fastreporting.files.wordpress.com/2011/03/vectorwise-whitepaper.pdf!http://www.slideshare.net/paulhofmann/big-data-and-saffron!http://www.paradigm4.com/2013/01/terabyte-scale-parallel-processing-with-r-and-scidb/!http://en.wikipedia.org/wiki/Associative_Memory_Base!http://www.staff.science.uu.nl/~bisse101/Book/PSC/psc1_2.pdf!http://calab.kaist.ac.kr/~swseo/papers/IEEE_CLOUDCOM2010_HAMA.pdf!http://en.wikipedia.org/wiki/Bulk_synchronous_parallel!http://gigaom.com/2012/07/05/want-to-ditch-your-data-scientists-heres-are-7-startups-that-can-help/!http://pages.sisense.com/elasticube-whitepaper.html?src=bottom!http://fastreporting.files.wordpress.com/2011/03/vectorwise-whitepaper.pdf!http://support.infobright.com/Support/Resource-Library/Whitepapers/!http://www.cse.buffalo.edu/faculty/tkosar/datacloud2012/papers/datacloud2012_paper_4.pdf!