backup uuasc 9/6/12
DESCRIPTION
A presentation on rsync and DuplicityTRANSCRIPT
![Page 1: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/1.jpg)
backups
using rsync and duplicity
![Page 2: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/2.jpg)
Who needs to backup?
There are 2 kinds of people in this world:1. Those who keep backups.2. Those who lose their data.
![Page 3: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/3.jpg)
What is a backup?
The copying and archiving of computer data to assist in recovering from a data loss event.
![Page 4: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/4.jpg)
What causes data loss?
● Failure● Disaster● Crime● Intentional action● Unintentional action
![Page 5: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/5.jpg)
![Page 6: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/6.jpg)
Why keep a backup?
● Keeps important files safe● Helps us recover from stupid mistakes● Helps recover from malicious attacks● CYA ( Cover Your Ass )
![Page 7: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/7.jpg)
Types of backups
● RAID!?
![Page 8: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/8.jpg)
RAID?!
RAID is not a backup!
ಠ_ಠ
![Page 9: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/9.jpg)
Types of backups
● RAID!?● Unstructured
![Page 10: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/10.jpg)
Types of backups
● RAID!?● Unstructured● Differential● Incremental
![Page 11: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/11.jpg)
Rsync
● extremely efficient file transfer protocol for Unix● open source (GNU license)● written by Andrew Tridgell and Paul Mackerras at the
Australian National University in 1996
![Page 12: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/12.jpg)
Rsync
We can backup to either local or remote server
LOCALGo to external disk for local backup. Backing up to same hard drive is not sufficient for a comprehensive backup solution!
EXTERNALGoes to rsync daemon (can be standalone or started from rsync client by inetd). Route through SSH shell for security.
![Page 13: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/13.jpg)
Rsync
Local locations:~/docs/important
Remote locations:Remote Shell[user@]server.com:~/docs/important
Rsync Daemon[user@]server.com::~/docs/importantrsync://[user@]server.com/~/docs/important
![Page 14: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/14.jpg)
Rsync
common flags:● -v : verbose
![Page 15: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/15.jpg)
Rsync
common flags:● -v : verbose
● -a : archive
![Page 16: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/16.jpg)
Rsync
common flags:● -v : verbose
● -a : archive
● -z : compress
![Page 17: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/17.jpg)
Rsync
common flags:● -v : verbose
● -a : archive
● -z : compress
● --rsh=/usr/bin/ssh
![Page 18: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/18.jpg)
Rsync
common flags:● -v : verbose
● -a : archive
● -z : compress
● --rsh=/usr/bin/ssh
● --delete : delete extraneous files
![Page 19: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/19.jpg)
Rsync
common flags:● -v : verbose
● -a : archive
● -z : compress
● --rsh=/usr/bin/ssh
● --delete : delete extraneous files
● --link-dest : create hard links (incremental backup)
![Page 20: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/20.jpg)
Incremental backups with Rsync
Hard links allow us to create extra nodes for our files so we can make sure at least one copy exists without using extra space.
![Page 21: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/21.jpg)
Advantages of Rsync over cp
● Mirroring data● Transmitting lots of data● Only copy what is needed● Use with --link-dest to create hard links for
incremental backups
![Page 22: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/22.jpg)
Duplicity
● First developed in 2002.● Current version is 0.6.19 released in May
2012● GNU licensed● Written in Python & C● Leverages Rsync algorithm● Available via package managers, or via
RPM, binaries, or source downloadable from http://duplicity.nongnu.org/
![Page 23: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/23.jpg)
Duplicity
Produces tar-format volumes on local or remote servers
![Page 24: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/24.jpg)
Duplicity files
● Backup● Manifest● Signature
![Page 25: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/25.jpg)
Duplicity filesystems
Local File Storage, scp/SSH, Rsync, FTP, HSI, WebDAV, Tahoe-LAFS, Amazon S3, with others easily added
![Page 26: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/26.jpg)
Duplicity
Remote locations are specified using URIs:● ftp://[user@]archive.business.com/backups● rsync://[user@]newspeak.marketing.biz/outside_box/● s3+http://awesometown-socool
![Page 27: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/27.jpg)
Using Duplicity
By default, Duplicity doesn't need any command line options to make encrypted, incremental backups to a local or remote location. However, for a sane backup strategy we should:
● Perform periodic full (snapshot) backups● Delete old backups● Ignore certain files
![Page 28: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/28.jpg)
Using Duplicity
● --full-if-older-than
![Page 29: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/29.jpg)
Using Duplicity
● --full-if-older-than● remove-all-but-n-full
![Page 30: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/30.jpg)
Using Duplicity
● --full-if-older-than● remove-all-but-n-full● --include/exclude-globbing-filelist
![Page 31: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/31.jpg)
Using Duplicity
● --full-if-older-than● remove-all-but-n-full● --include/exclude-globbing-filelist● PASSPHRASE environment variable
![Page 32: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/32.jpg)
Moar Duplicity
Also used to verify backups and restore backups
Verify:duplicity verify [backup] [source]can also be used to determine changes since backup
![Page 33: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/33.jpg)
Moar Duplicity
Also used to verify backups and restore backups
Verify:duplicity verify [backup] [source]can also be used to determine changes since backup
Restore:duplicity restore [backup] [destination]Can use --file-to-restore and --time flags to specify more precise restoration
![Page 34: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/34.jpg)
Why use Duplicity over Rsync?
● Encrypted and signed archives● Built in incremental backup● Ability to use a wide range of local and
remote filesystems
![Page 35: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/35.jpg)
Best Practices
● Trust, but verify● Use strong passphrase● Backup to multiple locations● Automate
![Page 36: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/36.jpg)
Automation
For the best backup strategy, we take the human out of the equation.
● write a Bash script and cron it● Use a GUI
![Page 37: Backup UUASC 9/6/12](https://reader034.vdocuments.us/reader034/viewer/2022042613/547e90b3b479597f508b4b71/html5/thumbnails/37.jpg)
Related LinksJamie Zawinski on backupshttp://www.jwz.org/doc/backups.htmlSnapshot Backups with Rsync by Mike Ruebelhttp://www.mikerubel.org/computers/rsync_snapshots/Epydocs for Duplicityhttp://duplicity.nongnu.org/epydoc/index.htmlDuplicity to Amazon S3 Bash Scripthttp://blog.damontimm.com/bash-script-incremental-encrypted-backups-duplicity-amazon-s3/RTFM!http://linux.die.net/man/1/rsynchttp://duplicity.nongnu.org/duplicity.1.htmlLots of Horror Stories about not Backing Uphttp://reddit.com/r/talesfromtechsupport