Benchmark setup

Centos 6.5, Datastax 4.5.3, jmeter with custom

sampler using the java driver

/dev/shm (100G), HP on, -XX:+UseLargePages,

Disabled THP, IRQBalance

Scaling Horizontally - Latency

1918

1010

16

97

15

78

14

7

0

5

10

15

20

25

INSERT AVG response time (us) SELECT AVG response time (us) UPDATE AVG response time (us)

Ave

rage

response tim

e (

ms)

sm

alle

r is

bette

r

1 node 2 nodes 3 nodes 4 Nodes

Scaling Horizontally - Throughput

90 96

168151

106

193202

120

252

223

130

249

0

65

130

195

260

325

INSERT throughput (k) SELECT throughput (k) UPDATE throughput (k)

KR

eq/s

ec -

Big

ger

is B

etter

1 node 2 nodes 3 nodes 4 Nodes

52 53

70

9096

171

0

45

90

135

180

INSERT SELECT UPDATE

kR

eq/s

-big

ger

is b

ette

r

FMCI 4.32 FMCI 20.192

33

40

1919 18

10

0

10

20

30

40

50

INSERT SELECT UPDATE

Avera

ge

Re

spo

nse t

ime

(m

s)

Sm

alle

r is

Bette

r

FMCI 4.32 FMCI 20.192

Scaling Vertically

Scaling Economics - Math

The score was computed by comparing:

Response times against the slowest

Number of requests/second against the fastest

2.3

1.5

1.0

0.40.3

0.0

0.6

1.2

1.8

2.4

3.0

1 node -FMCI 4.32 1 node 2 nodes 3 nodes 4 Nodes

Price

-to

-pe

rform

ance

-ratio (

big

ge

r is

be

tte

r)

Datastax’s Scaling Economics

• Amdhal’s Law

• Hardware prices

Why?

Amdhal’s Law

8,855 10,493

15,825 14,63817,304

22,249 23,963

3,918

10,493

20,986

26,505

35,107

53,010

70,215

0.

20000.

40000.

60000.

80000.

1x E3-1230v2

1x E5-2630v2

2x E5-2630v2

1x E5-2670v2

1x E5-2690v2

2x E5-2670v2

2x E5-2690v2

Pe

rfo

rma

nce

(h

igh

er

is b

ette

r)

Configuration

Performance relative to price

WHAT IT IS WHAT IT SHOULD BE

Specs PRICE ($) CPUMARK Est.

CPUMARK

1x E3-1230v2 4 cores,

3.3Ghz

$230.00 8855 3918

1x E5-2630v2 6 cores,

2.6Ghz

$616.00 10493 10493

2x E5-2630v2 2x6 cores,

2.6Ghz

$1232.00 15825 20986

1x E5-2670v2 8 cores,

2.6Ghz

$1556.00 14638 26505

1x E5-2690v2 10 cores,

3Ghz

$2061.00 17304 35107

2x E5-2670v2 2x8 cores,

2.6Ghz

$3112.00 22249 53010

2x E5-2690v2 2x10 cores,

3Ghz

$4122.00 23963 70215

CPU Prices

Money spending efficiency

0.000s

432000.000s

864000.000s

1296000.000s

1728000.000s

2160000.000s

2592000.000s

3024000.000s

sysbench memory 1TB read (1Mbs), write total time

Native Virtual

0.000s

86400.000s

172800.000s

259200.000s

345600.000s

432000.000s

518400.000s

sysbench multi-threadingperformance

Native Virtual

Virtualisation vs Native

Source:VIRTUAL MEMORY SYSTEMS AND TLB STRUCTURES Univ. Maryland 2001

Virtual Memory

Source:VIRTUAL MEMORY SYSTEMS AND TLB STRUCTURES Univ. Maryland 2001

Memory address translation with and without a TLB

Virtual Address Virtual Address

Physical Address Physical Address

• TLB: Translation Lookaside Buffers

• Memory pointers in OS = address in virtual memory not real memory, need an offset to get to the real

memory. Offset needs to be calculated (and this is very expensive) so it is cached in TLB.

• TLB miss normally=150 cycles

• Hardware assisted virtualisation makes normal translation faster in VMs but introduces high penalty on

TLB miss.

TLB: Translation Lookaside Buffers

TLB Misses

Source: “Memory System Characterization of Big Data Workloads” by Martin Dimitrov et al. - Intel Corp. [2013]

Centos 6.5, Datastax 4.5.3, jmeterdocker run -m 16G -d --privileged=true

Docker setup

19 18

10

2119

11

20

26

13

40

3028

0

10

20

30

40

50

INSERT SELECT UPDATE

Avera

ge R

esponse T

ime (

ms)

-S

malle

r Is

Better

1 Node native 1 Node Native 1 docker container 1 node native with 2 docker containers 1 native with 4 docker containers

Docker vs Native - Latency

Docker vs Native - Throughput

9096

168

8292

149

7868

81

45

60 56

0

45

90

135

180

INSERT SELECT UPDATE

KR

eq/s

-big

ger

is b

etter

1 Node native 1 Node Native 1 docker container 1 node native with 2 docker containers 1 native with 4 docker containers