distributing data the aerospike way

DISTRIBUTING DATA THE

AEROSPIKE WAYYoung PaikDirector, Sales EngineeringAerospikeyoung@aerospike.com

July 24, 2013

…OR Why Is This Line Taking So Long?

© 2013 Aerospike. All rights reserved. Pg. 2

A Database Is Like A Conference Registration Line

➤ The goal is to get as many people through the line as quickly as possible.

➤ Registrants must get their own registration package, not just anyone’s.

© 2013 Aerospike. All rights reserved. Pg. 3

Scaling Throughput

All databases (and registration systems) have limits to performance. The real question is how do you go beyond your current limits.

There are two basic strategies: ➤ Vertical scaling – upgrade single server➤ Horizontal scaling – distribute to multiple

servers

© 2013 Aerospike. All rights reserved. Pg. 4

Vertical Scaling

Vertical scaling means that if a small server can handle some traffic … a big one can handle more. This is true … to a point.

© 2013 Aerospike. All rights reserved. Pg. 5

What’s Wrong With Vertical Scaling?➤ Expensive➤ Still need to deal with failover.

What happens if your DB goes down? What happens if you need to upgrade?

➤ May still not meet the storage/speed requirements

➤ Still a single point of failure

© 2013 Aerospike. All rights reserved. Pg. 6

Horizontal Scaling

Horizontal Scaling means that in some way the load will be distributed among many servers.

© 2013 Aerospike. All rights reserved. Pg. 7

What Do You Want From Horizontal Scaling?➤ Hide the complexity of distribution.➤ Linear scalability.➤ Better service availability.➤ Deal with meteor strike on your data

center.

© 2013 Aerospike. All rights reserved. Pg. 8

Different Distribution Models

Distributed databases will place different data on different nodes.

Some common methods:➤ Simple sharding➤ Hashed sharding➤ Master-slave ➤ Smart partitioning

© 2013 Aerospike. All rights reserved. Pg. 9

Simple Sharding

© 2013 Aerospike. All rights reserved. Pg. 10

Simple Sharding

Clients know which node has the data.

© 2013 Aerospike. All rights reserved. Pg. 11

Simple Sharding

What happens if a node fails?

© 2013 Aerospike. All rights reserved. Pg. 12

Simple Sharding

Pros Cons+ Easy to set up. Clients are

written with a knowledge of how the data is distributed.

+ Servers aren’t coordinated, so no intra-cluster communication is necessary.

- May lead to imbalance and hot nodes

- If a node fails, the data on that node is unavailable.

- Adding new nodes requires reconfiguration on the clients and re-shuffling of data on the server, resulting in service down time.

- Replication must be handled separately.

© 2013 Aerospike. All rights reserved. Pg. 13

Hashed Sharding

© 2013 Aerospike. All rights reserved. Pg. 14

Hashing Sharding

➤ The key can be hashed using a hashing algorithm to create a seemingly random string

➤ The first several characters of the hash can be used to determine the node for that data.

PaikPaik

C820G3KH15HH3KASD43S

C820G3KH15HH3KASD43S

© 2013 Aerospike. All rights reserved. Pg. 15

Instead of using the actual key value, use a hash to randomize how the data is distributed.

Hashed Sharding

Hashed sharding will balance data and load.

© 2013 Aerospike. All rights reserved. Pg. 16

Hashed Sharding

But has the same problem on a node failure.

© 2013 Aerospike. All rights reserved. Pg. 17

Hashed Sharding

Pros Cons+ Easy to set up. Clients are

written with a knowledge of how the data is distributed.

+ Servers aren’t coordinated, so no intra-cluster communication is necessary.

+ Data/traffic is now balanced.

- If a node fails, the data on that node is unavailable.

- Adding new nodes requires reconfiguration on the clients and re-shuffling of data on the server, resulting in service down time.

- Replication must be handled separately.

© 2013 Aerospike. All rights reserved. Pg. 18

Master-Slave

© 2013 Aerospike. All rights reserved. Pg. 19

Master-Slave

Master coordinates connection with slave nodes.

© 2013 Aerospike. All rights reserved. Pg. 20

Master-Slave Sharding

Pros Cons+ Relatively simple setup with

master controlling distribution.

+ Replication can be set up to go to backup node. Master is responsible for coordinating.

- Adding new nodes requires reconfiguration on the master and often manual re-shuffling of data on the server, resulting in service down time.

- Requires multiple network connections.

- Single point of failure: the master. Some databases like Mongo require 3 masters (called configuration servers) where 2 will be backups for the main one.

© 2013 Aerospike. All rights reserved. Pg. 21

Smart PartitioningThe Aerospike Way

© 2013 Aerospike. All rights reserved. Pg. 22

Smart Partitioning

Every registrant knows where to go.

© 2013 Aerospike. All rights reserved. Pg. 23

Map

Smart Partitioning

And, every registrant knows where to go if a node fails.

© 2013 Aerospike. All rights reserved. Pg. 24

Map

Smart Partition Architecture

© 2013 Aerospike. All rights reserved. Pg. 25

Cluster creates a map of how data is distributed, called a partition map.

Combine features from other architectures to create a map.

Smart Partitioning

➤ Every key is hashed using the RIPEMD160 hash function

➤ The creates a fixed 160 bits (20 bytes) string.

➤ 12 bits of this hash are used to identify the partition id

➤ There are 4096 partitions

➤ Are distributed among the nodes

PaikPaik

182023kh15hh3kahdjsh182023kh15hh3kahdjsh

PartitionID

Master node

Replica node

… 1 4

1820 2 3

1821 3 2

4096 4 1

© 2013 Aerospike. All rights reserved. Pg. 26

Aerospike uses a partition table

Smart Partitioning

For simplicity, let’s take a 3 node cluster with only 9 partitions and a replication factor of 2.

© 2013 Aerospike. All rights reserved. Pg. 27

Smart Partitioning

Pros Cons+ Relatively simple setup, with the

cluster determining data distribution.

+ Balanced distribution.

+ No single point of failure.

+ Replication is automatic and immediate.

+ Failover is automatic and immediate.

+ Rebalancing is automatic and immediate.

+ An arbitrary number of nodes can be added to increase capacity.

+ True 24x7 uptime. Cluster can be upgraded on a rolling basis.

- Application must be written using smart API.

© 2013 Aerospike. All rights reserved. Pg. 28

What Do You Want From Horizontal Scaling?➤ Hide the complexity of distribution.

Balanced data distribution. No “hot nodes.” Automatic client reconfiguration. No need to manually

reconfigure/restart clients.➤ Linear scalability.

Easy to calculate needed capacity. Cluster can be an arbitrary number of nodes.

➤ Better service availability. 24 x 7 uptime. No downtime, even for “routine”

maintenance. No single point of failure. Automatic replication of data. Automatic failover. Automatic rebalancing when nodes fail. Automatic rebalancing when adding nodes.

➤ Deal with a catastrophe on your data center.© 2013 Aerospike. All rights reserved. Pg. 29

So can you deal with a meteor hitting my data center?

© 2013 Aerospike. All rights reserved. Pg. 30

But what about a meteor strike?

Multi-Datacenter Architecture

© 2013 Aerospike. All rights reserved. Pg. 31

Data Center 1

Data Center 2

Data Center 3

Cross Data Center Replication (XDR)➤ Asynchronous replication for long link delays

and outages

➤ Namespaces configured to replicate to a destination cluster – master / slave, including star and ring

➤ Replication process Transaction journal on partition master and

replica XDR process writes batches to destination Transmission state shared with source replica Retransmission in case of network fault When data arrives back at originating cluster,

transaction ID matching prevents subsequent application and forwarding

➤ In master / master replication, conflict resolution via multiple versions, or timestamp

© 2013 Aerospike. All rights reserved. Confidential Pg. 32

Caveats for Evaluating Distributed Databases

When testing new databases:➤ Make sure to test to an appropriate scale.➤ Beware of inconsistent performance in a

cloud environment.➤ If a database has caching, make sure your

use is realistic.

© 2013 Aerospike. All rights reserved. Pg. 33

THANK YOU

© 2013 Aerospike. All rights reserved. Pg. 34