vmware vfabric gemfire for high performance, resilient distributed apps
DESCRIPTION
Learn how VMware vFabric GemFire helps buildTRANSCRIPT
![Page 1: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/1.jpg)
© 2010 VMware Inc. All rights reserved
Building High Performance, Data Intensive Resilient, Distributed Applications
![Page 2: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/2.jpg)
2
The ChallengeData Explosion
Decision Time Compression
Critical to act time frame that matters•Milliseconds•Seconds•Minutes or even hours
![Page 3: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/3.jpg)
3
Today’s Modern Architecture’s
Web
Application
Database Tier
Storage Tier
Stateless
Stateful
Load Balancer
Web Applications
![Page 4: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/4.jpg)
4
Today’s Modern Architecture’s
Data Ingest Applications
![Page 5: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/5.jpg)
5
Agenda - Challenges and Solutions1.High Performance & Data Intensive
Latency Scale
2.Resilient Reliability Availability
3.Solutions
![Page 6: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/6.jpg)
6
Sources of LatencyDisk access
Serial access
Network time
Sockets (open/close)
Marshalling and unmarshalling
Security overhead
![Page 7: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/7.jpg)
7
Sources of Latency - Disk Access
Mitre Public Release: 10-0861. Distribution Unlimited
![Page 8: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/8.jpg)
8
Network time
Keep sockets open to all members Helps with security performance
Minimize network hops
Push computing to data
![Page 9: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/9.jpg)
9
Marshalling/Unmarshalling
Lazy deserialization
Index serialized data
Shared compact data format
![Page 10: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/10.jpg)
10
Security overhead
Mutual authentication at socket time
Process/user level (optional)
![Page 11: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/11.jpg)
11
Agenda - Challenges and Solutions1.High Performance & Data Intensive
Latency Scale
2.Resilient Reliability Availability
3.Solutions
![Page 12: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/12.jpg)
12
Architecting Infinitely Scalable Systems
http://blogs.msdn.com/b/pathelland/
A seminal paper on the architecture of elastic applications by Pat Helland (Tandem Computing, Amazon.com, Microsoft)
“Life Beyond Distributed Transactions: an Apostate’s Opinion”http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
Application architectures need to change to achieveinfinite scalability and elasticity without using large hardware
![Page 13: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/13.jpg)
13
Scale Aware Code
Scale - Layered Code
Programming Abstraction
Scale Agnostic Code
Bottom layer understands application is distributed
Top layer
Abstraction layer
Common layered architecture in largest scale applications
![Page 14: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/14.jpg)
14
Scale
Shared nothing
Partition/Sharding
Collocated relations
Replicated reference
14
![Page 15: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/15.jpg)
15
Agenda - Challenges and Solutions1.High Performance & Data Intensive
Latency Scale
2.Resilient Reliability Availability
3.Solutions
![Page 16: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/16.jpg)
16
Reliability
No data loss No data corruption Consistency
Race condition Synchronous vs Asynchronous
![Page 17: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/17.jpg)
17
No Data Loss, No Data corruption, Consistency
Distributed semaphore - lightweight
Primary copy
Distributed transaction(s) – heavy weight
MVCC – Acronyms are annoying
![Page 18: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/18.jpg)
18
Race Conditions, Consistency
Application
Data Tier
Stateless ?
Stateful
Eventually Consistent
Controllably Consistent
![Page 19: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/19.jpg)
19
Agenda - Challenges and Solutions1.High Performance & Data Intensive
Latency Scale
2.Resilient Reliability Availability
3.Solutions
![Page 20: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/20.jpg)
20
Availability - on Server
Protect data• Extra copies• Disk?
Data Center crashes
Network Splits• Split Brain detection
![Page 21: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/21.jpg)
21
Availability – Between Client/Server
Slow Consumers• HA Queues
Client Network drops• Durable subscribers
![Page 22: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/22.jpg)
22
Agenda - Challenges and Solutions1.High Performance & Data Intensive
Latency Scale
2.Resilient Reliability Availability
3.Solutions
![Page 23: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/23.jpg)
23
Latency & Reliability - Memory-based Performance
Perform Memory on a peer machine to make data updates durable, Writes return 10x to 100x faster than disk, 10s to 100s of Microseconds vs 10s to 100s Milliseconds
Keep redundant copies of dataUpdate thru primary0 data lossOptionally write updates to disk, Optional write to data warehouse asynchronously and reliably.
Customers
Orders
Product
Protect
![Page 24: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/24.jpg)
24
Memory-based Performance
Perform In Situ data processingReal-time controls
Calculate: current total fuel left
![Page 25: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/25.jpg)
25
Latency - Data-Aware Access
Perform Application Client
Java, C++, .Net, SQL
![Page 26: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/26.jpg)
26
Latency & Reliability - HA Data-Aware Function
Execute
Move behavior to data
ClientData Aware Function
![Page 27: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/27.jpg)
27
Parallel Queries
Client
Scatter-Gather Queries & Functions
Compute
![Page 28: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/28.jpg)
28
Data Distribution
Distribute
Keep clusters synchronized in real-time Operate reliably Disconnected, Intermittent and Low-Bandwidth network environments.
![Page 29: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/29.jpg)
29
Distributed Events
Targeted, Guaranteed delivery. Event notification &Continuous Queries
Notify
Disconnected, Intermittent and Low-Bandwidth network environments
![Page 30: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/30.jpg)
30
Cloud Ready
Web TierApplication Tier
Load Balancer
Optional reliable, asynchronous feed to Data Warehouse or Archival Database
GemFire Jar 11MB (or less)
Soar
![Page 31: VMware vFabric gemfire for high performance, resilient distributed apps](https://reader036.vdocuments.us/reader036/viewer/2022062614/546c80cdaf795956388b5d7a/html5/thumbnails/31.jpg)
31
Thank you