aerospike: comparing nosql databases for real-time bidding

40
WEBINAR: Comparing NoSQL Databases for Real-time Bidding Presented by Sergey Zhemzhitsky, CTO of CleverDATA

Upload: aerospike-inc

Post on 15-Jul-2015

3.551 views

Category:

Data & Analytics


3 download

TRANSCRIPT

WEBINAR:

Comparing NoSQL Databases

for Real-time Bidding

Presented by

Sergey Zhemzhitsky,

CTO of CleverDATA

cleverdata.ru | [email protected]

International market business development since 2012

One of three leading IT companies in Russia

43 branches in Russia and abroad

+5500 employees

100K projects for 10K customers

Data management innovative

platform (Data Exchange Service)

Cloud Service

In-house development

Internet advertising solutions

Data Management Platforms

Customers Base Management

Web Analytics

Marketing automation

Big DataData Mining

Digital IntelligenceOperational Intelligence

Low Latency and NoSQLCloud Computing

cleverdata.ru | [email protected]

Agenda

• RTB intro;• Challenges;• Choice difficulties;• Results;• Do’s and Don’ts.

cleverdata.ru | [email protected]

RTB intro

cleverdata.ru | [email protected]

Real Time Bidding (RTB)

publishers

AD NETWORKAD NETWORK

AD NETWORK

AD NETWORKAD NETWORK

AD NETWORK

advertisers

cleverdata.ru | [email protected]

How RTB works

publishers

AD NETWORKAD NETWORK

AD NETWORK

AD NETWORKAD NETWORK

AD NETWORK

advertisers

cleverdata.ru | [email protected]

Demand Side Platform (DSP)

advertiserspublishers

AD NETWORKAD NETWORK

AD NETWORK

AD NETWORKAD NETWORK

AD NETWORK

DSP

cleverdata.ru | [email protected]

Supply Side Platform (SSP)

publishers

AD NETWORKAD NETWORK

AD NETWORK

AD NETWORKAD NETWORK

AD NETWORK

advertisers

SSP

TRACKING DATA

cleverdata.ru | [email protected]

Data Management Platform (DMP)

publishers

COOKIE SYNCs

ACCESS LOGS

PARTNERS’ DATA

3rd PARTY DATA

CLICK STREAMS

advertisers

SSP

DSP

WHATEVER ELSE DATA

cleverdata.ru | [email protected]

RTB challengesvisitors publishers

advertisers

10

0 m

s

20 ms

10.000+ rps

TRACKING DATA

cleverdata.ru | [email protected]

publishers

COOKIE SYNCs

ACCESS LOGS

PARTNER’S DATA

3rd PARTY DATA

CLICK STREAMS

advertisers

SSP

DSP

DMP in the RTB ecosystem

DMP

cleverdata.ru | [email protected]

Wishes…

cleverdata.ru | [email protected]

Choice criteria

• Linear scalability;• Sharding “out of the box”;• Distributed by design;• Data redundancy & replication;• Low latency.

cleverdata.ru | [email protected]

Which NoSQL … again

cleverdata.ru | [email protected]

Aerospike Cassandra Mongo Redis

Scalability

Sharding

Redundancy

Latency

SPoF

Maintainability

Monitoring tools

Features

cleverdata.ru | [email protected]

Public results

cleverdata.ru | [email protected]

nginx 1.2.xngx-http-redislua-nginx-modulelua-resty-mongolngx-aerospike

aerospike 2.xredis 2.6.xmongodb 2.4.xwrkiperfnmon

• Intel® Core™ i7-920 Quad-Core• 48 GB RAM • 1 Gbit/s NICs (*)• 20 * 106 msgs• 512 bytes msg

* 200 Mbit/s guaranteed

Tools & Materials

cleverdata.ru | [email protected]

• No logging;• No plugins;• CPU Affinity.

NGINX

WRK

Measuring Nginx

cleverdata.ru | [email protected]

Nginx results. Base line

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

1.35 1.46 1.64 1.90 68851

cleverdata.ru | [email protected]

Measuring Redis

NGINX

WRK

Node 1Shard 1

Shard 2

Slave 3

Slave 4

Node 2Shard 3

Shard 4

Slave 5

Slave 6

Node 3Shard 5

Shard 6

Slave 2

Slave 1

cleverdata.ru | [email protected]

Redis results

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

2.68 3.10 3.25 3.90 34769

cleverdata.ru | [email protected]

Redis results

cleverdata.ru | [email protected]

Measuring Mongo

NGINX :: MONGOS

WRK

Node 1mongod :: cfg

mongod :: repl

mongod :: repl

mongod :: repl

Node 2mongod :: cfg

mongod :: repl

mongod :: repl

mongod :: repl

Node 3mongod :: cfg

mongod :: repl

mongod :: repl

mongod :: repl

Shard 1Replica Set 1

Shard 2Replica Set 2

Shard 3Replica Set 3

cleverdata.ru | [email protected]

Mongo results

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

6.70 8.22 10.22 15.46 14220

cleverdata.ru | [email protected]

Mongo results

cleverdata.ru | [email protected]

Node 1

Chunk 1

Replica 2

Node 2

Chunk 2

Replica 3

Node 3

Chunk 3

Replica 1

NGINX

WRK

Measuring Aerospike

cleverdata.ru | [email protected]

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

8.93 14.99 26.83 106.48 3402

Aerospike results

cleverdata.ru | [email protected]

Aerospike results

cleverdata.ru | [email protected]

Wks L, 50% L, 75% L, 90% L, 99% T, r/s Ngx, CPU Ngx, p/s As, CPU As, p/s

4 8.22 10.22 15.46 97.3 3402 7 8 K 1 1 K

8 7.89 15.16 92.4 89.8 6028 10 13 K 1 2 K

12 4.76 8.74 103.1 121.6 10233 15 20 K 2 3 K

16 3.91 6.21 99.6 111.3 13178 22 26 K 3 6 K

24 2.13 2.87 4.68 76.81 25744 21 60 K 5 10 K

32 2.01 2.60 4.42 81.29 28925 25 70 K 6 11 K

64 2.54 3.66 112.4 118.1 26468 27 70 K 6 11 K

… more Aerospike results

cleverdata.ru | [email protected]

nginx+ blocking I/O

How to shoot yourself in the foot…

cleverdata.ru | [email protected]

Libevent to the rescue (2nd attempt)

Node 1

Chunk 1

Replica 2

Node 2

Chunk 2

Replica 3

Node 3

Chunk 3

Replica 1

LIBEVENT :: LIBEVHTP

WRK

cleverdata.ru | [email protected]

Finally…

L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s

2.64 3.09 3.27 3.95 35746

cleverdata.ru | [email protected]

Finally…

cleverdata.ru | [email protected]

Nginx Redis Mongo Aerospike, b Aerospike, nb

Latency, 50% 1.35 2.68 6.70 8.93 2.64

Latency, 75% 1.46 3.10 8.22 14.99 3.09

Latency, 90% 1.64 3.25 10.22 26.83 3.27

Latency, 99% 1.90 3.90 15.46 106.48 3.95

Throughput, m/s 68851 34769 14220 3402 35746

CPU, Http, % 29 20 70 7 33

Network, Http, p/s, K 71 71 37 8 71

CPU, Db, % - 6 25 1 6

Network, Http, p/s, K - 12 21 1 12

Unofficial results

cleverdata.ru | [email protected]

Aerospike

cleverdata.ru | [email protected]

Let’s start building…

cleverdata.ru | [email protected]

• Do use non-blocking I/O;

• Do computations locally;

• Be lazy. Don’t do more than needed;

• Don’t afraid of using other’s good ideas;

• Don’t trust anyone.

Do’s & Don’ts

cleverdata.ru | [email protected]

Thanks a lot for your questions!

[email protected] :: [email protected]

cleverleaf.co.uk :: cleverdata.ru

1dmp.io/en :: crawler.1dmp.io/en

facebook.com/CleverData :: +7 (495) 967-66-50

To get started now with Aerospike, visit:

aerospike.com/get-started

Tweet us at @Aerospikedb