Exo-clonesInstantaneous Change Tracking for the Whole File SystemRichard P. Spillane, Wenguang Wang, Luke Lu, Maxime Austruy, Christos Karmanolis, Rawlinson Rivera
May 10th, 2016
VDFS (POSIX)
VSAN
VDFS (POSIX)
DeveloperLaptop
Backups, Documents,Applications
Exo Support Exo Support
1Hot Storage 2016
rsync/diff -R
Exo-clone
What is an Exo-clone?
2
A
FS SnapshotUpdatedSnapshot
A’
File Data
Meta Data A
FS SnapshotUpdatedSnapshot
A’
File Data
Meta Data
VDFS on VSAN VDFS on Developer Laptop
Copy of A’
Scan whole FS
• “free” to store• “free” to create• “free” to import
skipped
Hot Storage 2016
Benefits of Exo-clone
3
No duplication
No FS Scanning
diff -R ✖ ✖
git pack-objects ✔ ✖
Docker FS layers ✖ ✔
ZFS/BtrFS send/receive ✖ ✔
Exo-clones ✔ ✔
Hot Storage 2016
Exo-clone Use Cases
4
Changes in last hour
Application changes
Backup/replication
Application Revision Control
Improve with Exo-clones?
Hot Storage 2016
A Common Problem…
5
MyApp 1.6
Sources…git
Images, 3D models,icons, sounds, …?100sMB—10sGB
OS & Libraries…Docker
Lots of changes to large andsmall files
Backup MyApp
Hot Storage 2016
git? Sadly, no…
6
WORKING OBJECT STORE
A B
Redundant copy, 5-10x bigger thancurrent working state
checkout/commitcopy GBs of data
Hot Storage 2016
Docker?
7
Ubuntu14.04.2
MyApp1.0
network
Docker pullDocker push
Dockerfile:FROM 14.04.2apt-get install MyLibcp build/ /opt/mycp assets/ /opt/assets
Hot Storage 2016
Docker FS Layer
… = FS Layer 0x7dfb…
… = FS Layer 0xe9a0…
Other filesare the same
8Hot Storage 2016
Update With Docker Pull
9
Ubuntu14.04.3
MyApp1.1
network
Ubuntu14.04.3
Ubuntu14.04.2
Ubuntu14.04.2
MyApp1.0
200MB
MyApp1.1
MyApp1.0
…downstream
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.3
MyApp1.1
Ubuntu14.04.3
MyApp1.1
Security update
50MB250MB250MB
Hot Storage 2016
Update With Docker Pull
10
Ubuntu14.04.3
MyApp1.1
network
Ubuntu14.04.3
Ubuntu14.04.2
Ubuntu14.04.2
MyApp1.0
200MB
MyApp1.1
MyApp1.0
…downstream
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.3
MyApp1.1
Ubuntu14.04.3
MyApp1.1
Security update
50MB250MB250MBdocker build, push, pull, Dockerfile, …
Change Tracking FS
AUFS and LayersExo-clones
Hot Storage 2016
Updating Base Layers With Exo-clones
11
Ubuntu14.04.3
network
Docker pullDocker push
E-cUbuntu14.04.2
.3 - .2
network
Security update
Docker workflow Exo-clones
Hot Storage 2016
Update With Docker Utilizing Exo-clones
12
Ubuntu14.04.3
MyApp1.1
network
Ubuntu14.04.3
Ubuntu14.04.2
Ubuntu14.04.2
MyApp1.0
200MB
MyApp1.1
MyApp1.0
…downstream
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.2
MyApp1.0
Ubuntu14.04.3
MyApp1.1
Ubuntu14.04.3
MyApp1.1
Security update
50MB250MB250MB
.3 - .2
25MB
.3 - .2.3 - .2
75MB75MB
Still sending all of MyApp
.1 wrt .0
Dedup wrt to non-parental
volumes
5MB30MB30MB
Hot Storage 2016
Network Overhead of Docker FS Layers vs. Exo-clonesTransfer size of Docker FS layers compared to Exo-clones using whole-file dedup and Exo-clones using either block or 6-9KB variable length chunking
13
0
50
100
150
200
250
300
350
400
LTS Upgrade PXE Upgrade (zipped)
Docker FS Layer Exo Whole-file Exo Blk-chunk Exo Var-chunk
Laye
r Siz
e (M
B)
88%
25%
Hot Storage 2016
Exo-clone Format
14
1. rm /a/12. curl http://img.com/huge > /new/foo3. commit /.exo/clone1
clone1:
rm /a/1
touch /new/foo
map /new/foo to yellow data block
Ref Count=2Ref Count=1Ref Count=2
Meta-dataSection
DataSection
/new/foo/a/1 /a/2 /.exo/clone1
Ref Count=2Ref Count=1
Ref Count=1
Parent SS…
Child SS:
Only additional space used by
Exo-clone
Hot Storage 2016
Summary• Instantly create and share diffs between any two FSes with
– No space overhead– No sending the same block twice over the wire
• Better backend for Docker, rsync, git• Easy but efficient backup/replication between any storage environment• What becomes simpler when we make revision control on the whole FS cheap and standard?
– What if every FS (e.g., ZFS, BtrFS, LVM, NASes) supported exo-clones?
15Hot Storage 2016
Thank [email protected]://telegram.me/necro351
16Hot Storage 2016
Zero-copy Requires Block Alignment
17
A
SnapshotUpdatedSnapshot
A’
BlockStore
Meta Data
Existing FS
send/receive
A
SnapshotUpdatedSnapshot
A’
BlockStore
Meta Data
Zero-copyExo-clone
Without Zero-copy With Zero-copy
Small files/changes increase exo-
clone size
Zero-copy export (and
import)
Hot Storage 2016
How do exo-clones compare to ZFS and BtrFS send/receive?
0
100
200
300
400
500
600
700
800
Linux Django Ubuntu
tar.gztarBtrFSZFSVDFS
Volumes full of small files are encoded less efficiently in exchange for zero-copy
We rely on an additional gzip pass when transferring over network
18
Laye
r Siz
e (M
B)
Hot Storage 2016
Exo-Clone Prototype doing Cassandra Migration
WAN
VSAN
VDFS
Deployment Cluster
VDFS
VSAN
VDFS
Test/Dev or Backup Cluster
VDFS
19Hot Storage 2016
Application Revision Control
20
MyApp 1.1
MyApp 1.6 mobile
MyApp 1.4
MyApp 1.6, 1.5, …
MyApp 1.7a
MyApp 1.7b
MyApp 1.6
Bugs & Feature RequestsFeatures & Perf & Fixes & Bugs
Hot Storage 2016
VDFSExo Support
VDFSExo Support
Implementation of Exo-clones on VDFS
21
VSAN AWS/EBS (?)POSIX (e.g., Ext4)
Exo-clon
e VDFSExo Support
Hot Storage 2016
git? Sadly, no…
22
WORKING OBJECT STORE
A B
1.Checkoutcopies GBs
2.Modify3.Add
4.Commitcopies GBs
git add…
Redundant copy, 5-10x bigger thancurrent working state
Hot Storage 2016