large scale nosql db migration under fire - aerospike

A E RO S P I K E US E R S UM M I T 2018

Large Scale NoSQL DB Migration Under Fire

Ido Barkan

Software Architect

AppsFlyer

2 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc

Ag

en

da

Motivation and limitations

What is AppsFlyer?

What have we learned?

War plan


What is AppsFlyer?

~25B events per day


Installs DB- Zoom in

app installs writer

2K write IOPS

app events enrichments 180k read

IOPS


We need to replace CouchBase

● Scale out is out of control(45 r3.4xl instances)

● XDCR can’t keep up● Daily backups take more than a day● Paid support is not good enough

Chosen solution: Aerospike


Starting point

● # of records: ~ 2,000,000,000 (repl. factor 3)● Read intensive: 180K read + 2K write IOPS● # of machines: 45 r3.4xlarge● Cost: ~ 30,000 $ per month● Time frame: a few months.Mission critical


Why is data migration harder than in a relational DB?

Relational DB

NoSql DB

Data types conventional

blob, JSON, specialized bins

Vendors a few 100+

Query Language

SQL N1QL, SQL, JavaScript, HTTP/REST-- creating views/ MapReduce style

Schema Yes Maybe

Migration tools

Yes DIY


Rules of the game

No DowntimeNo data loss


War plan- easy part

● Write new installs/updates to BOTH● Continue reading from the OLD DB Application


Breathing (testing) time

test test test● Sizing RAM/Disk● Expected load● Metrics● Alerting● Logs● Backups


Easy part (Cont.)● Write new installs/updates to NEW DB● Read from BOTH DBs

● A fraction of records (not VIP)

● Do NOT delete yet!● Have metrics on

misses


Breathing time

Test Test Test● Sizing (again)● Expected load

(with reads)● Metrics (again)


Is that it?

If your data is short lived- you are mostly done.

Otherwise...


Dumperview (query)dump keys/records

Kafka as a buffer

Loader-> write(do not overwrite)

values

‘Manual’ partMigrating the long tail


Detour: DB views

function (doc, meta) {emit(meta.expiration, doc.id);}

key (sorting) value

Map function (JS)

● Record itself is NOT a part of the view


‘Manual’ partMigrating the long tail

Dumperview (query)dump keys/records

Kafka as a buffer

Loader-> write(do not overwrite)

values


New Cluster

● # of records: ~ 2,000,000,000 ● 2AZ- repl. factor 2 in each one● IOPS, same and growing● # of machines: 2 x 5 i3.4xlarge● Cost: ~ 10,000 $ per month (⅓)


What have we learned?

1. Plan, Plan, Plan.2. Go easy: test a sample before you start

(1 app / API / Customer).3. Breath (read: test) between stages.4. Do NOT delete until you have to.5. slower -> safer (-> $$$).

[email protected]

large scale nosql db migration under fire - aerospike

Documents