2©Bacula Systems S.A. - 2014
● First Bacula patch in Feb 2003● French Translation● BWeb GUI● Accurate Mode, RunScript, Batch Insert, Base Jobs, Deadlock detection system...
● First Deployment– Version 1.39 custom, 500 Clients, 2 SD, 1 DIR using
PostgreSQL, DRBD and HALinux
4©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
From simple disk (40 MB/s → 80MB/s), to large array (100MB/s → ∞ MB/s)
Fragmented FS = Sequential I/O → Random I/O
6©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
100Mb/s ? (35GB/h)1Gb/s ?10 Gb/s ?
Using Bonding ? Dedicated Link ?Duplex problem ?
FD
9©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
Interface shared between X jobs
Ex : 1Gb/s, 6 jobs16MB/s per job.
SD
11©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
1Gb/s shared (100MB/s) between X jobs
Tape Drive(80MB/s → 200MB/s)
Disks (80MB/s)
SD
12©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
- Special case with Data Spooling (Disk to Tape using large chunks)
Multiple Jobs writingat the same time
Despooling to tape at constant high speedrate
Disk structure should allow to Read AND Write with a good throughput
15©Bacula Systems S.A. - 2014
Journey to the Center of your Backup
- Test all compoments- Believe only what you see- Test the whole chain
Network : netcat, ntopDisks : tar to /dev/null, iostat, benchCPU : topTape : btape
16©Bacula Systems S.A. - 2014
Resolve a bottleneckNot always easy
● Client Side– Split the FileSet– Change Compression to LZO
● Network– Configure properly– Add interfaces, switches – Avoid firewalls and routers
● Storage Daemon – Add a new Storage Daemon
17©Bacula Systems S.A. - 2014
Bacula Servers System Tuning
● Memory (> 8GB)– vm.dirty_ratio = 2– vm.dirty_background_ratio = 1– vm.swappiness = 10– vm.zone_reclaim_node = 0
● Disks– Use good RAID (ARECA) controler with battery– Use deadline disk scheduler– Use XFS, stay away from ext3 on Redhat5
18©Bacula Systems S.A. - 2014
Network Tuning
● Use network bonding and Jumbo Frames● Configure Kernel
– net.core.wmem_max– net.core.rmem_max– net.ipv4.tcp_rmem– net.ipv4.tcp_wmem
● Use a Storage Daemon in your File Daemon LAN/VLAN, avoid Firewall and routers
- net.ipv4.tcp_sack- net.core.netdev_max_backlog- net.ipv4.tcp_timestamps- net.ipv4.tcp_window_scaling
19©Bacula Systems S.A. - 2014
Bacula Tuning 1/2
● Turn off « AutoPruning » in Client and Job ressources
● Prune volumes once a day● Turn on « Attribute Spooling » for all Jobs● Do not try VTL to Tape Job migration
20©Bacula Systems S.A. - 2014
Bacula Tuning 2/2
● Use concurrent jobs, specially when using spooling and tapes
● Use batch insert and configure your database● Run more Incremental, less Full● For tapes, « Maximum Block Size » (256 to 512k), « Maximum File Size » (10 to 20GB)
24©Bacula Systems S.A. - 2014
Catalog Tuning : Why
● PostgreSQL – work_mem = 1MB– wal_buffer = 64k
● MySQL – sort_buffer_size = 2MB– innodb_buffer_pool = 128MB
● File record is about 100 bytes● Jobs can despool several GB of attributes● Even a small Catalog can contain GB of data
25©Bacula Systems S.A. - 2014
Catalog Tuning : Why
● Typical Bacula Job– Large, sequential insertion
File
Batch
PathFilename
Job
One record per file
26©Bacula Systems S.A. - 2014
Catalog Tuning : PostgreSQL
– shared_buffers = 1GB # up to 8GB– work_mem = 64MB– effective_cache_size = 2GB – checkpoint_segments = 64– checkpoint_timeout = 20min– checkpoint_completion_target = 0.9– maintenance_work_mem = 256MB – wal_buffers = 4MB # No longer needed 9.1
27©Bacula Systems S.A. - 2014
Catalog Tuning : MySQL
● Good Question !– key_buffer_size– sort_buffer_size– join_buffer_size– myisam_sort_buffer_size– innodb_log_buffer_size– innodb_log_file_size– innodb_buffer_pool_size– binlog_cache_size
– thread_cache_size– query_cache_size– query_cache_limit– tmp_table_size– max_heap_table_size– innodb_additional_mem_p
ool_size– innodb_rollback_segments– ...
29©Bacula Systems S.A. - 2014
Some Catalog Numbers
● Big Hosting Company– 3.3 billion records– 1.4 TB of Postgresql data– 1200 client defined– 4 to 10 weeks of retention– 8 SAS disks, no SSD, standard raid controler