accordion - vldb 2014
DESCRIPTION
Slides of the VLDB 2014 presentation of the paper "Accordion: Elastic Scalability for Database Systems Supporting Distributed Transactions"TRANSCRIPT
Accordion: Elastic Scalability for Database Systems
Supporting Distributed Transactions
!Marco Serafini
Qatar Computing Research Institute !
joint work with: Essam Mansour, Ashraf Aboulnaga Qatar Computing Research Institute Kenneth Salem University of WaterlooTaha Rafiq Amazon.com Umar Farooq Minhas IBM Research Almaden
Leveraging the Cloud
P1P2P3
P4P5P6
P7P8
Applications cannot always leverage the cloud
Make partitioned DBMSes scale out and in!
Cloud layer
DBMS layer
Leveraging the Cloud
P1P2
P3P4P5 P6
P7P8
Applications cannot always leverage the cloud
Make partitioned DBMSes scale out and in!
Cloud layer
DBMS layer
Online Solution
Online = Handle workload with unanticipated skews
Partitions suddenly become hot
Overall database load grows/shrinks
Skews change over time
No prior knowledge, no workload trace analysis
Accordion
Goal: run a partitioned DBMS on a variable set of servers
DBMS supports ACID distributed transactions
Necessary in many OLTP workloads
Major performance bottleneck
Problems we address: Where & When to move data
New!
Accordion Architecture
�
�Y<hjQEQjs� ]Zd][I[jh
0RQLWRULQJ��6HUYHU�FDSDFLW\�
HVWLPDWRU
3DUWLWLRQ�SODFHPHQW�PDSSHU
7UDQVDFWLRQ�UDWHV��0HPRU\�XWLOL]DWLRQ��SHU�SDUWLWLRQ�5HVSRQVH�ODWHQF\��SHU�VHUYHU�$IILQLW\�PDWUL[��SHU�SDUWLWLRQ�SDLU�
3URYLVLRQLQJ��3DUWLWLRQ�PLJUDWLRQ
6HUYHU�FDSDFLW\�IXQFWLRQ�F
1HZ�PDSSLQJSDUWLWLRQV�ĺ�VHUYHUV
$FFRUGLRQ
3DUWLWLRQHG�'%06
3HUIRUPDQFHPHWULFV
The dangers of scaling out
with distributed transactions
Before scale out: After:
Full DB
1/2 DB
1/2 DB
Does scaling out increase throughput?
Scaling Out: How Effective is it?
0 5000
10000 15000 20000 25000 30000 35000
8 16 32 64
Serv
er c
apac
ity (t
ps)
Overall number of partitions
16 partitions per server8 partitions per server
No Distributed Transactions
YCSB Max throughput
per server constant
Overall throughput
grows linearlyN nodes
2*N nodesBars: Smaller DB <-
Larger DB ->
Distributed Transactions
0 5000
10000 15000 20000 25000 30000
8 16 32 64
Serv
er c
apac
ity (t
ps)
Overall number of partitions
16 partitions per server8 partitions per server
TPC-C Max throughput
per server decreases
Overall throughput
can still increase N nodes
2*N nodesBars: Smaller DB <-
Larger DB ->
Bin Packing Partition Placement
Bin Server
Volume of Bin Maximum throughput of server
Item Database partition
Volume of Item Transaction rate of partition
Distributed Transactions = Circular Dependency
Bin Volume - not constant!
Packing
Constraints Determines
Two Problems
1. Model for server capacity (maximum throughput)
Capacities varies based on placement
Learn model online
2. Partition placement using this model
1: Server Capacity Models
Affinity: Likelihood of co-access among partitions
!
Max throughput capacity per server depends on affinity
Null affinity (YCSB) -> Constant capacity
Uniform affinity (TPC-C) -> f (# local partitions)
Arbitrary affinity -> Must model affinity explicitly
Aff(P1,P2) = Prob (P2 accessed | P1 accessed)
2: Partition Placement
Minimize data migration
s.t. servers are not overloaded
Server capacity function Can be nonlinear
Accordion’s planner uses linear programming
Evaluation
Setup
Target DBMS: H-Store
Horizontally partitioned
Single-partition transactions execute sequentially
Distributed transactions use 2PC
Three benchmarks
YCSB (null affinity)
TPC-C (uniform affinity)
Clustered TPC-C (arbitrary affinity)
Cost Reduction (TPC-C)
Arbitrary Affinity up to 9x cost reduction
Uniform Affinity up to 1.7x cost reduction
0 5
10 15 20 25 30
64 256 1024
Num
ber o
f ser
vers
use
d
Number of partitions
AccordionKairos-SP
GreedyStatic
0 5
10 15 20 25 30 35 40
64 256 1024
Num
ber o
f ser
vers
use
dNumber of partitions
AccordionKairos-SP
GreedyStatic
Impact on Migration (TPC-C)
Shorter reconfiguration time, fewer servers
0 5000
10000 15000 20000 25000 30000 35000 40000
0 10 20 30 40 50
Tran
sact
ions
per
sec
Time (min)
AccordionKairos-SP
Cold
