bacula performance & tuningbacula.us/wp-content/uploads/2015/10/ericbollengier.pdf · ©bacula...

Eric BollengierDirector of EngineeringBacula Systems

Bacula Performance & Tuning

2©Bacula Systems S.A. - 2014

● First Bacula patch in Feb 2003● French Translation● BWeb GUI● Accurate Mode, RunScript, Batch Insert, Base Jobs, Deadlock detection system...

● First Deployment– Version 1.39 custom, 500 Clients, 2 SD, 1 DIR using

PostgreSQL, DRBD and HALinux


Journey to the Center of yourBackup


Journey to the Center of your Backup

From simple disk (40 MB/s → 80MB/s), to large array (100MB/s → ∞ MB/s)

Fragmented FS = Sequential I/O → Random I/O



Compression ?Encryption ?



100Mb/s ? (35GB/h)1Gb/s ?10 Gb/s ?

Using Bonding ? Dedicated Link ?Duplex problem ?

FD



?

100 Mb/s ?ADSL link ?

FD



?

SDFD



Interface shared between X jobs

Ex : 1Gb/s, 6 jobs16MB/s per job.

SD



Packing, Checksum, etc...

SD



1Gb/s shared (100MB/s) between X jobs

Tape Drive(80MB/s → 200MB/s)

Disks (80MB/s)

SD



- Special case with Data Spooling (Disk to Tape using large chunks)

Multiple Jobs writingat the same time

Despooling to tape at constant high speedrate

Disk structure should allow to Read AND Write with a good throughput



● Done !



- Test all compoments- Believe only what you see- Test the whole chain

Network : netcat, ntopDisks : tar to /dev/null, iostat, benchCPU : topTape : btape


Resolve a bottleneckNot always easy

● Client Side– Split the FileSet– Change Compression to LZO

● Network– Configure properly– Add interfaces, switches – Avoid firewalls and routers

● Storage Daemon – Add a new Storage Daemon


Bacula Servers System Tuning

● Memory (> 8GB)– vm.dirty_ratio = 2– vm.dirty_background_ratio = 1– vm.swappiness = 10– vm.zone_reclaim_node = 0

● Disks– Use good RAID (ARECA) controler with battery– Use deadline disk scheduler– Use XFS, stay away from ext3 on Redhat5


Network Tuning

● Use network bonding and Jumbo Frames● Configure Kernel

– net.core.wmem_max– net.core.rmem_max– net.ipv4.tcp_rmem– net.ipv4.tcp_wmem

● Use a Storage Daemon in your File Daemon LAN/VLAN, avoid Firewall and routers

- net.ipv4.tcp_sack- net.core.netdev_max_backlog- net.ipv4.tcp_timestamps- net.ipv4.tcp_window_scaling


Bacula Tuning 1/2

● Turn off « AutoPruning » in Client and Job ressources

● Prune volumes once a day● Turn on « Attribute Spooling » for all Jobs● Do not try VTL to Tape Job migration


Bacula Tuning 2/2

● Use concurrent jobs, specially when using spooling and tapes

● Use batch insert and configure your database● Run more Incremental, less Full● For tapes, « Maximum Block Size » (256 to 512k), « Maximum File Size » (10 to 20GB)


Catalog Tuning : Why ?


Catalog Tuning : Why

Is what youExpect fromBacula ?


Catalog Tuning : Why ?

Is what youget with defaultsettings...



● PostgreSQL – work_mem = 1MB– wal_buffer = 64k

● MySQL – sort_buffer_size = 2MB– innodb_buffer_pool = 128MB

● File record is about 100 bytes● Jobs can despool several GB of attributes● Even a small Catalog can contain GB of data



● Typical Bacula Job– Large, sequential insertion

File

Batch

PathFilename

Job

One record per file


Catalog Tuning : PostgreSQL

– shared_buffers = 1GB # up to 8GB– work_mem = 64MB– effective_cache_size = 2GB – checkpoint_segments = 64– checkpoint_timeout = 20min– checkpoint_completion_target = 0.9– maintenance_work_mem = 256MB – wal_buffers = 4MB # No longer needed 9.1


Catalog Tuning : MySQL

● Good Question !– key_buffer_size– sort_buffer_size– join_buffer_size– myisam_sort_buffer_size– innodb_log_buffer_size– innodb_log_file_size– innodb_buffer_pool_size– binlog_cache_size

– thread_cache_size– query_cache_size– query_cache_limit– tmp_table_size– max_heap_table_size– innodb_additional_mem_p

ool_size– innodb_rollback_segments– ...


Catalog Tuning : MySQL

mysqltuner


Some Catalog Numbers

● Big Hosting Company– 3.3 billion records– 1.4 TB of Postgresql data– 1200 client defined– 4 to 10 weeks of retention– 8 SAS disks, no SSD, standard raid controler


Some Numbers 1/3

Jobs / day


Some Numbers 2/3

Files / Day

200,000,000 files in one day


Some Numbers 3/3

bacula performance & tuningbacula.us/wp-content/uploads/2015/10/ericbollengier.pdf · ©bacula...

Documents