barns: backup and recovery for nosql databases backup and recovery for nosql databases atish...

BARNS: Backup and Recovery for NoSQL DatabasesAtish Kathpal, Priya SehgalAdvanced Technology Group, NetApp

© 2017 NetApp, Inc. All rights reserved.1

Why Backup/Restore NoSQL DBs?

Customers are directly ingesting into NoSQL

Security breach are on the rise e.g. ransomware attackson MongoDB [1] and recent WannaCrypt exploits

“Fat-finger” errors eventually propagate to replicas

Sandbox deployments for test/dev

§ Bring up shadow clusters of different cardinality (from production cluster snapshots)

Compliance and regulatory requirements

IDC, 2016 report [2] lists data-protection and retention as one of the top infrastructural requirements for NoSQL

© 2017 NetApp, Inc. All rights reserved.2[1] http://thehackernews.com/2017/01/secure-mongodb-database.html [2] Nadkarni A., Polyglot Persistence: Insights on NoSQL Adoption and the Resulting Impact on Infrastructure. IDC. 2016 Feb.

Ransomware

From Backup/Restore Service PerspectiveNoSQL Database Classes

§ Master-slave§ Authoritative copy of each partition is contained in the master node

that we can backup.§ Loss of primary node leads to shard/partition-unavailability until new

leader is elected. § Example: MongoDB, Redis, Oracle NoSQL, MarkLogic

§ Master-less§ Data is scattered across nodes using consistent hashing techniques,

no single node has all data for a given partition§ Eventual consistency: Unavailability of a destination node does not

lead to write-failure, data is eventually replicated§ Example: Cassandra, Couchbase


https://www.slideshare.net/mongodb/sharding-v-final, https://blog.imaginea.com/consistent-hashing-in-cassandra/

NoSQL DBs Hosted on Shared StorageHigh-level, conceptual deployment architecture


NoSQL DB software(Node1)

Filesystem(DB Journal, Logs,

Data Files)



Data Files)



Data Files)

Shared Storage Array(Snapshots, Cloning, Compression, Deduplication, Encryption, Cloud Integrations)

LUN-1 LUN-2 LUN-3

Backup: Leverage storage snapshotsRestore: Leverage

cloning

Backup/Restore Challenges§ Cluster-consistency at scale

§ Cluster/App quiesce – significantly hampers application performance. Cross node consistency not guaranteed§ Take crash consistent snapshots§ Post process crash consistent snapshots (in a sand-box) using NoSQL DB stack

to reach an cluster-consistent state

§ Space Efficiency§ Replica set data copies do not de-duplicate – small row sizes, scattered

across nodes (Cassandra) and unique ids added by storage engines (MongoDB)

§ DB performs compression and encryption§ Remove replicas logically (application aware backup)

§ Topology Changes § Commodity nodes, at scale of 10-100s of nodes. E.g., Primary node

might be unreachable while taking backup in case of MongoDB§ Storage snapshots do not have context about cluster topology§ Use cases may require restore to a test/dev cluster of different cardinality§ Save Cluster topology and storage mapping as part of backup


Cassandra N1

Cassandra N2

Cassandra N3

Write:(K1, V1), (K3, V3)

Write:(K1, V1), (K2, V2)

Write:(K2, V2), (K3, V3)

Shared Storage clusterLUN1: K1, K3LUN2: K1, K2LUN3: K2, K3

Insignificant block based

deduplication

Replication factor = 2

Existing open source utilities like Mongodump and Cassandra snapshots suffer from above challenges.

NoSQL Data Protection – Challenges

Challenges:

§ Fault tolerance§ Backup may capture stale data due

to eventual consistency§ Higher restore times, since

Cassandra will perform repairs during restore

Master-Less Databases


Cassandra N1

Cassandra N2

Cassandra N3

Update:(K1, V11, Tnew)

Update:(K1, V11, Tnew)

Shared Storage clusterLUN1: (K1, V1, Told), …LUN2: (K1, V1, Told), …LUN3: …

Client, Update K1 (CL.ONE)

(K1, V11, Tnew)

Ack

Snapshot of LUN2 will point to stale

data

http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html, https://wiki.apache.org/cassandra/HintedHandoff

BARNS Architecture


Light Weight Backup Process(LWB)

Post-Process

Restore Process

Lib_DB

Lib_Storage

Lib_MongoDB Lib_Cassandra

Lib_Store1 Lib_Store2

Store1 Store2

APIs APIs

MongoDB Cluster

Cassandra Cluster

Mongo APIs Cassandra APIs

(PP)

Stop_balancer()Get_topology()Check_cluster_health()Prepare_Backup_Topo()Etc.

Get_lun_mappings()Take_snapshots()Lun_clone()Vault_backup()Etc.

Addresses challenges of:1. Taking cluster-consistent

backup at scale2. Taking storage efficient

backups (through replica removal)

3. Enables recovery/cloning to different cluster topologies

Master-less Distributed DatabaseBARNS Solution: Cassandra


Phase 1: Light-weight Backup Phase


C1

CL1D1

C4

CL4D4

C3

CL3D3

C2

CL2D2

L1 L2 L3 L4

BARNS:LWB

Production Cluster

CL1

D1

sn1

CL2

D2

sn2

CL3

D3

sn3

CL4

D4

sn4

1. Capture token assignment of each node 2. Store mapping of LUNs à Tokens

3. Take snapshots of L1 to L4

Token1 Token4Token3Token2

Backup Metadata example

Phase2: Post Process Phase Part1: Flush Commitlogs


P1

CL1D1

P4

CL4D4

P3

CL3D3

P2

CL2D2

CL_sn1 CL_sn2 CL_sn3 CL_sn4

BARNS:PP-Phase

PP Node(s)

1. Clone the LWB Snapshot LUNs2. Mount on different PP processes or different nodes3. Start Cassandra processes P1 to P4

4. Flush CommitLogs

E.g. Data

K1, V1, T1

K2, V2, T1

K1,V11, T2

Phase2: Post Process Phase Part2: Compaction


CL_sn1 CL_sn2 CL_sn3 CL_sn4

1. Using UnionFS mount all snapshot clones as read-only2. Create a new volume and LUN for full backup3. Mount it as RW through UnionFS4. Let PP node have all tokens of prod cluster5. Start Cassandra6. Start compaction process7. Final compacted files will be stored on

Fullbackup LUN

Data 1

K1, V1, T1

K2, V2, T1

K1,V11, T2

/cassandra

Data 2

K1, V1, T1

K1, V11, T2

K3,V3, T3

Data 3

K1, V1, T1

K2, V2, T1

Data 4

K3,V3, T3

K2, V22, T4

DataCompact

K1, V11, T2

K2, V22, T1

K3,V3, T3

Full_bk_lun

unionFS

PP Node

Keeping single copy of data in the backup => ~66% reduction in backup storage requirements

BARNS:PP-Phase

Cassandra RestoreThe post-process step enables cloning to different restore/clone topologies


Full Backup

K1, V11, T2

K2, V22, T1

K3,V3, T3

Token1 Token2

N1 N2

Full_bk_lun

Clones

Token3 Token4

Evaluation - Cassandra Backup and Restore § Production cluster§ 4 nodes§ 4 iSCSI LUNs§ Commitlog and SSTables for a

node on same LUN§ Cassandra 4.0

§ Post Process Node§ 2 CPUs§ 8GB RAM

§ YCSB to ingest data

Full Backup


• LWB – <10 secs• pp-flush - ~40 secs• pp-compact – time increases by 35-40% à incremental backup• Restore time – less than ~2 mins (irrespective of cluster size and data set size)

Master-Slave Distributed Database

Ø Check the paper or just attend the poster session J

BARNS Solution: MongoDB


Summary

• Tracking replicas and cluster topologies is important for taking backups and performing flexible topology restores

• Existing open-source solutions have several inefficiencies like need for repairs after restore, lack of storage efficiency in backup and poor integrations with shared storage

• Opportunity to provide efficient backup and restore through light-weight snapshots and clones



Thank You.

barns: backup and recovery for nosql databases backup and recovery for nosql databases atish...

Documents