backup automation in kakao
TRANSCRIPT
Backup AutomationSUNG DONGCHAN From KAKAO
Backup is for..
2. Replication
1. Recovery
3. etc...
Backup Method?
2. xtrabackup
1. mysqldump
3. snapshot
Backup Storage?
Local Disk
Easy, No scheduling
But.. Recovery?
Remote Storage
Network Traffic?
Rare affect on Recovery
NIC
Backup Start Time?
• Schedule with CRONTAB
• Start from manager server on DEFINED TIME (BUT, Everyday it’s same time..)
Same time
No Problem if it's on LOCAL BACKUP
Think about
REMOTE BACKUP
A lot of Servces
Multi-IDC
IDC1 IDC2
IDC3
Frequently Master/Slave Switch
X
Schedule Manually?
We need..
Scheduling Automatically
TODAY!! I’ll talk about remote backup scheduling
Backup Files?
• Big.. but Rarely Used
• Transaction Log exists 2~3 Days
THEN!!!!
MUST "Same Backup Server" As Yesterday?
Question 1.
NO
Question 2.
MUST "Today's Backup File"?
NO
Question 3.
MUST “Backup on Same Time"?
NO
Think about..
Backup File is in Cloud
A B
C D E F
Today Yesterday
The day before yesterday
Everyday, Different
Let's start backup on
Unhurried Server
Consists of..
• Service Configuration
• Backup Servers
• Backup History
Service Configuration
Queue Time?
Backup job could be started
ANY TIME after QUEUE TIME
Backup Server
Quota?
0/3
3/3
1/3
Quota Full I'm So Busy!!
Backup ServerLocated on “Specific RACK”
But.. Database could be on.. “Different” IDC!!
need logical Concept..
Service Zone!!Virtual Service Group
ex) Common Zone, Security Zone, ThirdParty Zone..
IDC1 IDC2 IDC3
Common Service Zone
Security Zone
ThirdParty Zone
Service Zone
Backup History
Today? Not Yesterday Server!!
BACKUP001.kr
Yesterday
NOT!! BACKUP001.kr
so.. BACKUP001.kr
BACKUP002.kr
BACKUP003.kr
Service
Candidate Today's Backup Server
Backup Step
• Choose Service base on Queue Time
• Choose Target Backup Server (Service Zone + IDC Location + Quota)
• Start Backup
Choose Service (1/3)
SVC01 SVC02 SVC03 SVC0400:00
Smaller Queue Time, Backup First01:59 02:00 02:00 03:00
Choose Service (2/3)
SVC01 SVC02 SVC03 SVC0400:00
Bigger Data File, Backup First
02:00 02:00 02:00 02:00
500G 300G 200G 100G
Choose Service (3/3)
SVC01 SVC02 SVC03 SVC0400:00
Different Switch, Backup First
02:00 02:00 02:00 02:00
100G 100G 100G 100G
SW01 SW02 SW03 SW01
Choose Backup Server (1/3)
Target Backup Server List?
• Same Service Zone • Same IDC
Choose Backup Server (2/3)
Less than Quota
0/3
3/3
1/3
Quota Full I'm So Busy!!
First Candidate
Choose Backup Server (3/3)
Not Used YesterdayBackuped Yesterday
Candidate Today's Backup Server
So.. Backup Process is..
Start Backup
At Service after queue time on Unhurried and Valid Backup Server
Backup ServerService Zone
Backup Service ArchitectDatabases Zone1 Databases
Databases
Zone2
Databases
Manager
Schedule Automatically
Backup Servers Backup Servers
Backup Servers Backup Servers
IDC1 IDC2
Backup Timeline
Restore Sample
ssh root@BACKUP_SERVER \
'cat /data/backup/SERVICE_HOST/DATE.xbs' | xbstream -x
ssh root@backup001 \
'cat /data/backup/svc01/20150427.xbs' | xbstream -x
Generated By Manager
With LZOP? Recommand! If InnoDB :)
Compressed
After Queue Time Same IDC
Same Service Zone Unhurried Backup Server
Backup Scheduled..
Q/A
