hbase snapshots

HBase Snapshots

HBase User Group Meetup10/29/12

Jesse Yates

So you wanna….

• Prevent data loss

• Recover to a point in time

• Backup your data

• Sandbox copy of data

Problem!

a BIG Problem…

• Petabytes of data

• 100’s of servers

• At a single point in time

• Millions of writes per-second

Solution!

Solutions!

(Obvious) Solutions!

Built-in• Export

– MapReduce job against HBase API– Output to single seqeunce file

• Copy Table– MapReduce job against HBase API– Output to another table

Yay• Simple• Heavily tested • Can do point-in-time

Boo• Slow• High impact for running cluster

(Less Obvious) Solution!

Replication

• Export all changes by tailing WAL

YAY• Simple• Gets all edits• Minimal impact on running cluster

Boo• Turn on from beginning• Can’t turn it off and catch up• No built-in point-in-time• Still need ETL process to get multiple copies

(Facebook) Solution!1

Mozilla did something similar2

1. issues.apache.org/jira/browse/HBASE-55092. github.com/mozilla-metrics/akela/blob/master/src/main/java/com/mozilla/hadoop/Backup.java

Facebook Backup

• Copy existing hfiles, hlogs

Yay• Through HDFS

– Doesn’t impact running cluster• Fast

– distcp is 100% faster than M/R through HBase

Boo• Not widely used• Requires Hardlinks• Recovery requires WAL replay• Point-in-time needs filter

Backup through the ages

HBaseHDFS

Export

Copy Table

Replication

Facebook

HBASE-50

Maybe this is harder than we thought…

We did some work…

Hardlink workarounds

• HBASE-5547– Move deleted hfiles to .archive directory

• HBASE-6610– FileLink: equivalent to Windows link files

Enough to get started….

Difficulties

• Coordinating many servers

• Minimizing unavailability

• Minimize time to restore

• Gotta’ be Fast

HBASE-50HBASE-6055

Snapshots

• Fast- zero-copy of files

• Point-in-time semantics– Part of how its built

• Built-in recovery– Make a table from a snapshot

• SLA enforcement– Guaranteed max unavailability

Coming in HBase-0.96!

Snapshots?

We’ve got a couple of those…

Snapshot Types

• Offline– Table is already disabled

• Globally consistent– Consistent across all servers

• Timestamp consistent– Point-in-time according to each server

Offline Snapshots

• Table is already disabled• Requires minimal log replay– Especially if table is cleanly disabled

• State of the table when disabled• Don’t need to worry about changing state

YAY• Fast!• Simple!

But I can’t take my table offline!

Globally Consistent Snapshots

• All regions block writes until everyone agrees to snapshot– Two-phase commit-ish

• Time-bound to prevent infinite blocking– Unavailability SLA maintained per region

• No Flushing – its fast!

What could possibly go wrong?

Cross-Server Consistency Problems

• General distributed coordination problems– Block writes while waiting for all regions– Limited by slowest region– servers = P(failure)

• Stronger guarantees than currently in HBase

• Requires WAL replay to restore table

I don’t need all that, what else do you have?

Timestamp Consistent Snapshots

• All writes up to a TS are in the snapshot

• Leverages existing flush functionality

• Doesn’t block writes

• No WAL replay on recovery

Timestamp Consistent?

Put/Get/Delete/Mutate/etc.

Timestamp in snapshot?

Snapshot Store Future Store

Yes No

MemStore

I’ve got a snapshot,now what?

Recovery

• Export snapshot– Send snapshot to another cluster

• Clone snapshot– Create new table from snapshot

• Restore table– Rollback table to specific state

Export Snapshot

• Copy a full snapshot to another cluster– All required HFiles/Hlogs– Lots of options

• Fancy dist-cp– Fast!– Minimal impact on running cluster

Clone Table

• New table from snapshot

• Create multiple tables from same snapshot

• Exact replica at the point-in-time

• Full Read/Write on new table

Restore

• Replace existing table with snapshot

• Snapshots current table, just in case

• Minimal overhead– Handles creating/deleting regions– Fixes META for you

Whew, that’s a lot!

Even more awesome!

Goodies

• Full support in shell

• Distributed Coordination Framework

• ‘Ragged Backup’ added along the way

• Coming in next CDH

• Backport to 0.94?

Special thanks!

• Matteo Bertozzi– All the recovery code– Shell support

• Jon Hsieh– Distributed Two-Phase Commit refactor

• All our reviewers…– Stack, Ted Yu, Jon Hsieh, Matteo

Thanks!Questions?

Jesse Yates@jesse_yates

jesse.k.yates@gmail.com