gridgain’in3memory’data’fabric:’ · gridgain’in3memory’data’fabric:’’...
TRANSCRIPT
© 2014 GridGain Systems, Inc.
DMITRIY SETRAKYAN Founder & EVP Engineering @dsetrakyan
GridGain In-‐Memory Data Fabric: UlCmate Speed and Scale for TransacCons and AnalyCcs
www.gridgain.com #gridgain
© 2014 GridGain Systems, Inc.
Agenda
• EvoluCon of In-‐Memory CompuCng • GridGain In-‐Memory Data Fabric • Distributed Cluster & Compute
– Coding Example
• Distributed Data Grid – Coding Examples
• Distributed Streaming & CEP • Plug-‐n-‐Play Hadoop Accelerator
© 2014 GridGain Systems, Inc.
What is In-‐Memory CompuFng
• High Performance & Low Latencies • Faster than Disk and Flash • Cost EffecCve • Distributed or Not • Caching, Streaming, ComputaCons • Data Querying – SQL or Unstructured • VolaCle and Persistent • OLAP and OLTP Use Cases
© 2014 GridGain Systems, Inc.
EvoluFon of In-‐Memory CompuFng
Caching
Distributed Caching
In-‐Memory Data Grids IMDBs
Database IM opCons Hadoop
accelerators
Streaming
BI accelerators
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
© 2014 GridGain Systems, Inc.
ExisFng Market is Fragmented
Company Product Proprietary/ Open Source CharacterizaFon
Oracle In-Memory Option for Oracle Database Proprietary Cost Option
Oracle Times Ten Proprietary Point Solution IMDB
Oracle Coherence Proprietary Point Solution IMDG
SAP Hana Proprietary Point Solution - IMDB
Microsoft SQL Server 2014 Proprietary Feature Upgrade
DataBricks Apache Spark Open Source Point Solution - Hadoop
VoltDB VoltDB Open Source Point Solution – IMDB
Aerospike Aerospike Open Source Point Solution – NoSQL DB
IBM DB2 with BLU Acceleration Proprietary Feature Upgrade
Software AG Terracotta Open Source Point Solution - IMDG
Hazelcast Hazelcast Open Source Point Solution - IMDG
GridGain In-‐Memory Data Fabric: Strategic Approach to IMC
• Supports all Apps
• Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware
• Supports existing & new data sources
• No need to rip & replace
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
© 2014 GridGain Systems, Inc.
© 2014 GridGain Systems, Inc.
Clustering & Compute • Direct API for MapReduce • Direct API for Fork/Join • Zero Deployment • Cron-‐like Task Scheduling • State Checkpoints • Early and Late Load Balancing • AutomaCc Failover • Full Cluster Management • Pluggable SPI Design
© 2014 GridGain Systems, Inc.
AutomaFc Cluster Discovery
© 2014 GridGain Systems, Inc.
Closure ExecuFon
© 2014 GridGain Systems, Inc.
Closure ExecuFon
© 2014 GridGain Systems, Inc.
In-‐Memory Caching and Data Grid
• Distributed In-‐Memory Key-‐Value Store • Replicated and ParCConed • TBs of data, of any type • On-‐Heap and Off-‐Heap Storage • Backup Replicas / AutomaCc Failover • Distributed ACID TransacCons • SQL queries and JDBC driver • CollocaCon of Compute and Data
© 2014 GridGain Systems, Inc.
Cache OperaFons
© 2014 GridGain Systems, Inc.
Cache TransacFon
© 2014 GridGain Systems, Inc.
Distributed Java Data Structures
• Distributed Map (cache) • Distributed Set • Distributed Queue • CountDownLatch • AtomicLong • AtomicSequence • AtomicReference • Distributed ExecutorService
© 2014 GridGain Systems, Inc.
Client-‐Server vs Affinity ColocaFon
Client-‐Server Affinity ColocaCon
© 2014 GridGain Systems, Inc.
In-‐Memory Streaming & CEP
• Streaming Data Never Ends • Branching Pipelines • CEP Sliding Windows • Pluggable RouCng • Real Time Analysis • At Least Once Guarantee
© 2014 GridGain Systems, Inc.
Plug-‐n-‐Play Hadoop Accelerator
• Up to 100x AcceleraCon • In-‐Memory NaCve MapReduce
– In-‐Process Data ColocaCon – Eager Push Scheduling
• GGFS In-‐Memory File System – Pure In-‐Memory – Write-‐Through to HDFS – Read-‐Through from HDFS
• Sync and Async Persistence
© 2014 GridGain Systems, Inc.
In-‐Memory NaFve MapReduce
• In-‐Memory NaCve MapReduce – Zero Code Change – Use exisCng MR code – Use exisCng Hive queries
• No Name Node • No Network Noise • In-‐Process Data ColocaCon • Eager Push Scheduling
© 2014 GridGain Systems, Inc.
DevOps Management and Monitoring
© 2014 GridGain Systems, Inc.
THANK YOU
www.gridgain.com #gridgain @dsetrakyan