an approach for hybrid-memory scaling columnar in-memory

!!!!

*Bernhard Höppner, °Ahmadshah Waizy, *Hannes Rauhe !

* SAP SE ° Fujitsu Technology Solutions GmbH

!!!!

ADMS’14 in conjunction with 40th VLDBHangzhou, China

September 1, 2014

An Approach for Hybrid-Memory Scaling Columnar In-Memory Databases

© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.

Memory – Linking Desire and Reality

2

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s



3

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s

DRAM HDD



4

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s

DRAM HDD

SSD



5

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s

DRAM HDD

SSD Storage Class Memory ( )

Cheaper than DRAM

Denser than DRAMBandwidth close to DRAM

Latency close to DRAM

+Non-Volatile +Byte-addressable

+More energy efficient than DRAMSCM



6

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s

DRAM HDD

SSD Storage Class Memory ( )SCM

Available 2018+

Cheaper than DRAM

Denser than DRAMBandwidth close to DRAM

Latency close to DRAM

+Non-Volatile +Byte-addressable

+More energy efficient than DRAM



7

LatencyBandwidth

Capacity

Cost

nsμs

GB

TB

10 $/GB

$/GB

GB/s MB/s

DRAM HDD

SSD SCM

Hybrid-Memory

Fixed amount of DRAM to buffer data from SSD

PCIe SSD as backing storage

Offer separate memory allocator !Accessed data via load/store operations

Use paging to buffer data of SSD on DRAM !Increase addressable amount of memory


Main Store

Persistence

Data Volumes Log Volumes

Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge

Read Operations Data Modifying Operations

Mem

oryStorage

IMDBMS – SAP HANA based Architecture

8


Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage


Main MemoryLoaded

Column(s)

Data Request on Colum

nLi

mite

d am

ount

of M

emor

y

UnloadedTable

9


Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage


Main MemoryLoaded

Column(s)

Data Request on Colum

nLi

mite

d am

ount

of M

emor

y

UnloadedTable

0 2 4 6 8 10

20

40

60

80

In-Memory Table Size [GB]

Table

Load

Tim

e[s]

10


IMDBMS – Scaling the Traditional Way

Scale-Up • Increase the amount of available memory

in a single server by adding DRAM

• Limitations:

• Fujitsu RX-600 S6 2 TB

• Fujitsu RX-900 S2 4 TB

• Fujitsu 2800B 12 TB !Scale-Out • Add additional cooperating instances

• Instances are separate (shared nothing)

• Additional complexity and cost e.g. on the infrastructure level!

11


Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage

IMDBMS – Scaling w/ Hybrid-Memory

Main MemoryLoaded

Column(s)

Data Request on Column

Lim

ited

amou

nt o

f Mem

ory

UnloadedTable

Hybrid-MemoryLoaded

Column(s)

Load

Unload

On Dem

and

12


Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage


Main MemoryLoaded

Column(s)

Data Request on Column

Lim

ited

amou

nt o

f Mem

ory

UnloadedTable

Hybrid-MemoryLoaded

Column(s)

Load

Unload

On Dem

and

0 2 4 6 8 10 12

100

200

300

400

500

600

Database Size [TB]

App

roxi

mat

edH

ardw

are

Cos

t[1000Euro]

IMDBMS using DRAM onlyIMDBMS using DRAM and Hybrid-Memory

13


Hybrid-Memory

Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage

321101

Values


DRAM

SSD14


Main Store

Persistence


Delta Store

AlphaBeta

CharlieDelta

0123

Dictionary101232

Values

BetaDeltaAlpha

Charlie

0123

Dictionary

1312

Values

Merge


Mem

oryStorage

Hybrid-Memory – Integrating

Hybrid-Memory

321101

Values

DRAM

SSD

• Hybrid-Memory is introduced on column level if data does not fully fit into DRAM !

• Minimize latency overhead by storing less frequently used columns on Hybrid-Memory first !

• Higher access skew on column level increases the probability of cache hits !

• Reduce large range selects on Hybrid-Memory as data is paged in 4kB chunks !• Avoid scan operations as sequential operations are not

beneficial on Hybrid-Memory

15


Fetch SQLPlan Cache

Parse SQL Plan Cache,

generate ASTGet next node

from AST

Analyze column access

Column used for

Projection

Column used for Selection

Not indexed

Indexed

Add to SeqAcc

Add to RndAcc

Add to RndAcc

While nodes in AST

Aggregate SeqAcc, RndAcc

SortSeqAcc, RndAcc

ascending

Append RndAcc to AccFreq

Calc. usage frequency for

SeqAcc, RndAcc

Append SeqAcc to AccFreq


16

Fetch SQLPlan Cache


generate AST

Use existing information, do not generate additional statistics

Parse SQL statements executed in the past

Generate LFU list for columns


Evaluation – Partitioning TPC-C

17

80% of all columns used in less than 2% of all queries

Less frequently used columns cause 60% of the data volume


Evaluation – TPC-C Hybrid-Memory 80% : DRAM 20%

18

1008060402010560

70

80

90

100

Hybrid-Memory Buffer Size [%]

DB

MS

Thr

ough

put[%

]80% of all columns on Hybrid-Memory,20% on DRAM

DRAM buffer size for Hybrid-Memory is adjusted

Upper touchpoint, no paging

TPC-C performance in tpm% compared to unchanged system

10% DRAM buffer reduces tpm by 12%, index vector size reduced by 54%


Conclusion and Future Work

Conclusion:

• DRAM + SSD combined to Hybrid-Memory

• Hybrid-Memory made available to applications via a separate allocator

• Extended SAP HANA to use Hybrid-Memory as an additional data store

• Analyzed TPC-C to be partitioned for two data stores

• Ran TPC-C based evaluation reducing the index vector size by 54%

while keeping the IMDBMS performance at 88%

19


Conclusion and Future Work

Future Work:

• Evaluate our approach based on real data and workload

• Integrate column partitioning directly into HANA

• Introduce paging to other data stores (e.g. dictionary)

• Design a cost model for columns to be stored on Hybrid-Memory

including row level information

20

© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.©

Thank You! Questions? Comments?

An Approach for Hybrid-Memory Scaling Columnar In-Memory Databases !

*Bernhard Höppner, °Ahmadshah Waizy, *Hannes Rauhe !

* SAP SE ° Fujitsu Technology Solutions GmbH

!ADMS’14 in conjunction with 40th VLDB

Hangzhou, China September 1, 2014


Fetch SQLPlan Cache



from AST


Column used for

Projection


Indexed

Not indexed

Add to SeqAcc

Add to RndAcc

Add to RndAcc

While nodes in AST


SortSeqAcc, RndAcc

ascending



SeqAcc, RndAcc



22

Analyze SQL Plan Cache on column level

Separate into random and sequential access

Maintain statistics in two independent lists

Get next node from AST


Column used for

Projection


Not indexed

Indexed

Add to SeqAcc

Add to RndAcc

Add to RndAcc

While nodes in AST



23



SeqAcc, RndAcc

Fetch SQLPlan Cache



from AST


Column used for

Projection


Not indexed

Indexed

Add to SeqAcc

Add to RndAcc

Add to RndAcc

While nodes in AST


SortSeqAcc, RndAcc

ascending



SeqAcc, RndAcc




24

SortSeqAcc, RndAcc

ascending



Add randomly accessed columns first as they are more suited for Hybrid-Memory

Reduce access on Hybrid-Memory by keeping frequently used columns on DRAM

Workload adapted priority list for Hybrid-Memory

Fetch SQLPlan Cache



from AST


Column used for

Projection


Not indexed

Indexed

Add to SeqAcc

Add to RndAcc

Add to RndAcc

While nodes in AST


SortSeqAcc, RndAcc

ascending



SeqAcc, RndAcc



© 2014 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

an approach for hybrid-memory scaling columnar in-memory

Documents