how to run real time processing on big data / ron zavner (gigaspaces)

Copyright © GigaSpaces 2015. All rights reserved.Copyright © GigaSpaces 2015. All rights reserved.

How to run Real Time processing on Big DataRon ZavnerTechnical Director, EMEA Gigaspaces

Copyright © GigaSpaces 2015. All rights reserved. 2

About GigaSpaces

GigaSpaces provides software middleware for deployment, management and scaling of mission-critical applications on

cloud environments.

GigaSpaces serves more than 500 large enterprises & ISVs, over 50 of which are Fortune-listed.

Direct customers300+

Fortune / Organizations40+ / 500+

Cloud Customers75+

ISVs25+


Agenda• The challenge of real time analytics today• Introduction to In Memory Data Grids• Meet XAP• XAP on Big Data (plus advanced features)• Use cases + Demo


In today’s realityinformation is gathered from numerous sources


• Number of transactions grows exponentially• Tolerance for system response time reduces significantly


In order to gain a competitive edge, organizations need real time processing of large data sets


Alongside this challenge, there are many other challenges for management of enterprise applications

Such as:


Current tier based architecture cannot meet these challenges


Lets go over these challenges and see why


Peak Loads ChallengesYour messaging tier can only scale so far…

Result: Costly over-provisioning

Massive over-provisioning of resources only to meet peak loads

Your database can only scale so far…


Performance Challenges

Disk throughput limitations and bottlenecks

Network bottlenecks

Result: Higher latency for your business transactions, which can be very costly


Business Continuity Challenges

Too many moving parts

Result: Application becomes more error-prone and harder to troubleshoot


TCO Reduction Challenges

Result: Having many moving parts means higher operational costs (purchase, update, maintain, etc).


Real-Time Business Insight Challenges

Result: If performance is not ideal, than real time response isn’t either.

In today’s reality, real time event processing of constantly growing data sets gives your company the competitive edge it needs.


Tier-based architecture cannot solve today's data & application processing issues


Meets All These Challenges


XAP scales the Data Tier using its In-Memory Data Grid so you can access your data in real time


XAP scales the Data Tier using its In-Memory Data Grid

The database goes to the background

Partition your data and store it in memory


Same goes for the Messaging Tier

Partitioned, co-located in-memory messaging


XAP scales the entire application – so you get extreme processing of your big data and get real time insights

Business logic, data & messaging co-located & partitioned into processing units


XAP ensures High Availability

Hot backup for each partition for high availability


Scale the Web Tier

Host your web application on the XAP infrastructure


XAP enables auto-scaling of the entire application on demand

Auto-scale out & in based on real-time performance & load


Result:Real time processing & analytics of your big data with XAP IMC


What can XAP do for you?

Scaling the Data Tier

Multi-site deployment & DR across remote sites

Batch processing of large data sets

Online transaction processing

Real time querying and analysis of large datasets

Real time processing of large event stream

Scaling the Web Tier

Copyright © GigaSpaces 2015. All rights reserved.

Some features…Complex Queries Support

Scala Support

Data Consistency Level

Increase developer productivity and application scalability using new syntax for complex nested conditions

Comprehensive Scala support including Tasks, Objects, APIs and Queries

Update Multiple Entries to different partition with timeout to cope with locks

Optimize XAP to your business requirements setting the Data Consistency Level

Optimized Cross Partition Update

IPv6 LRMI Filters

Support for IPv6 Network with XAP deployments

Web GUI can serve hundreds of nodes with good UX

Allow for encryption and compression for selected LRMI connection

Scalable Web GUI

Better support for customized user credentials

Extended Custom Security

http://wiki.gigaspaces.com/wiki/display/XAP96/Query+Nested+Properties

http://wiki.gigaspaces.com/wiki/display/XAP96/Consistency+Level

http://wiki.gigaspaces.com/wiki/display/XAP96/Scala+Predicate+Based+Queries

http://wiki.gigaspaces.com/wiki/display/XAP96/Change+API

http://wiki.gigaspaces.com/wiki/display/XAP96/Communication+Filter+Over+the+Wan+(SSL,ZIP)

http://wiki.gigaspaces.com/wiki/display/XAP96/Custom+Security


and more…

27

Microsoft Linq providerSupport for queries written using Linq API and syntax

MongoDB External Data Source

Bi directional data and metadata exchange between MongoDB and XAP

Unique constrains in IndexSupport for unique validation of objects in index in each partition

Change API EnhancementsAdditional Change API returns the valuethat had been changed

Advanced Projections supportPartial queries on nested objects

Immutable Objects Support

Zone controlled deploymentSupport for mapping primary nodes into specific zone

http://wiki.gigaspaces.com/wiki/display/XAP95/Change+API+Advanced

http://wiki.gigaspaces.com/wiki/display/XAP95/Getting+Partial+Results+Using+The+Projection+API

Copyright © GigaSpaces 2015. All rights reserved.28

AggregationFramework

Custom Change Operations

Enhanced Initial Data Load

Management & Deployment Automation

Query Analysis

Memory Xtend for SSD

Global HTTP Session Sharing

and more…


Aggregations – Java

29


Aggregations – .Net (using LINQ)

30


Change Operation

31


Custom Change Operation – Java

32


The In-Memory Computing Platform Extreme scaling solution across multiple verticles


Multi-Site OperationsBy Synchronization of dynamic data

across remote sites

Multi-Site Data Replication Across Remote Data Centers

Real Time Data Processing & AnalysisBy providing a secured private data cloud

Out of the box Disaster Recovery PlanningData is up-to-date across all sites

Near real-time replication of massive data streams – synchronize data across your different sites in a consistent, failure-proof,

scalable way.


Selected Customers


E-Commerce

Industry

“GigaSpaces XAP performed beautifully, easily withstanding Kohl’s Black Friday load. In fact, we are quite confident that GigaSpaces could handle Kohl’s growth for years to come”

Handle peak loads without over-provisioning for maximum traffic (following 2009 system crash resulting in loss of millions of $$)

Challenge

Implemented inventory management on top of XAP within 4 months

Solution

* Kohl’s was N. America’s best performing e-commerce website on Black Friday 2010Results


Banking

Industry

“GigaSpaces technology allows us to increase customer satisfaction by facilitating better, user-friendly services, as well as new services, which ultimately enhances our bottom line”

Scalable transaction processing for trading platform; increase performance and efficiency of banking processes.

Challenge

GigaSpaces provides infrastructure for Avanza’s core services, including trading, customer and data processing, and storage.

Solution

* Zero system cost for data storage (vs. half system capacity before)* Massive cost reduction* Major performance improvement

Results


Premium Edition

XAP In-MemoryLite Edition

Extreme app scaling & RT insights

Try for free!

Try and let us know what you think…


Check us out:

Email us:Call us:Follow us:

www.gigaspaces.comwww.getcloudify.orginfo@gigaspaces.com646-421-2830@GigaSpaces, @CloudifySource

how to run real time processing on big data / ron zavner (gigaspaces)

Engineering