Linux‐HA+DRBD+PostgreSQL

JesseYoung

TargetAudience

•  SystemAdministrators•  DBAslookingintoreplicaBon

Goals

•  InspiremorereplicaBondiscussions•  InformSystemAdmins/DBAs

•  Describecomponents

•  SimpleSetup

•  DemonstraBon

•  PerformanceTips

Intro

•  ZonarSystems,Inc.– 45,000vehiclessendingdatain24/7andgrowing– 15databaseservers– 5TBdata,growing~25GBdaily– PostgreSQL+PostGIS

WhyuseLinux‐HA+DRBD?

•  SimplereplicaBon•  GPSdatacomingintothesystem24/7

•  MiBgateproblemsfromserverhardware

•  UsersandIntegraBonsoYwareaccess

Linux‐HA

•  “Provide a high availability (clustering) solu6on for Linux which promotes reliability, availability, and serviceability (RAS) through a community development effort.”

•  SystemAdmins:YoucanconBnueto(half)sleep,evenifaserverdies.

DRBD•  DistributedReplicatedBlockDevice

•  Mirroredstorageacrossservers

Imagefromh_p://www.drbd.org

PostgreSQL

•  Databaseengineweallknowandlove

Setup

•  Dual‐procQuad‐coreIntelXeon@2.66GHz•  16GBRAM

•  2x74GBSCSI@15kRPM(RAID1‐OS)

•  6x300GBSCSI@10kRPM(RAID10–DRBD)

•  LSIMegaraid–2Channel,512MBCachew/BBU

QuickInstall

•  Versions– heartbeat‐2.0.8– drbd‐8.0.5– PostgreSQL–Anyversion

InstallDRBD

•  Compilefromsourceor

•  yuminstalldrbdkmod‐drbd

ConfigureDRBDresourcepostgres{onnode‐001{device/dev/drbd0;disk/dev/sdb1;address10.40.0.1:7788;meta‐diskinternal;}

onnode‐002{device/dev/drbd0;disk/dev/sdb1;address10.40.0.2:7788;meta‐diskinternal;}}

BeginDRBD

•  Eachnode:– modprobedrbd

– drbdadmcreate‐mdpostgres– drbadmupall

CreateParBBononDRBDDevice

•  Primarynode:– drbdadm‐‐‐‐overwrite‐data‐of‐peerprimaryall

– mke2fs‐j/dev/drbd0– mountdevice– Edit/etc/init.d/postgresql•  PGDATAtoDRBDdevice

–  initdborsync/copyexisBngdatabase

DRBDstatus

•  cat/proc/drbd

version:8.0.5(api:86/proto:86)

SVNRevision:3011buildbyjesse@node‐001.zonarsystems.net,2007‐09‐0223:16:51

0:cs:Connectedst:Primary/Secondaryds:UpToDate/UpToDateCr‐‐‐

ns:191681144nr:108784dw:191798660dr:1228217841al:962834152bm:9054lo:0pe:0ua:0ap:0

resync:used:0/31hits:34633misses:4367starving:0dirty:0changed:4367

act_log:used:0/127hits:11433119293misses:1051777270starving:11558178dirty:81149096changed:962834152

InstallLinux‐HA

•  Compilefromsourceor

•  yuminstallheartbeat

ConfigureLinux‐HA

•  ha.cf•  haresources•  authkeys

ha.cf

deadBme120auto_failbackoff

udpport694

ucasteth110.30.0.10

nodenode‐001.zonarsystems.net

nodenode‐002.zonarsystems.net

haresources

cluster‐001.zonarsystems.netIPaddr::10.30.0.101/24drbddisk::postgres\

Filesystem::/dev/drbd0::/mnt/rod1::ext3\

postgresql\MailTo::jesse@zonarsystems.com::'Zonar_Cluster‐001_Switch'

authkeys

auth1

1sha1f93e5e562d430dae92b7fd7b272c13bb0c5de0e2

StartLinux‐HA

•  /etc/init.d/heartbeatstart

DEMO

Problems?

•  Secondaryserveralwaysstand‐by•  Failednodecancauseslowdowns•  Kernelmodule

Performance?

•  ddif=/dev/zeroof=/mnt/drbd/pgwest/testbs=8192count=2000000

•  DRBDdevice– ~85MB/s

•  Diskdevice– 121MB/s

30%write performancedegradaBon

Problems?

PerformanceImprovements

Ethernet

•  Directcrossovercable•  GigE,10GigE•  1500MTUvs9000MTU

RAIDCards

•  CapableofWRITEBACK•  Writecaching

•  Ba_erybackupunit(BBU)

WritebackVS.Writethrough

•  Writeback– 80MB/s–94MB/s

•  Writethrough– 61MB/s–71MB/s

0102030405060708090

100

Run1

Run3

Run5

Run7

Run9

Run11

Run13

Run15

DRBDProtocols

•  ProtocolA– DRBDwaitsforlocaldiskandlocalTCPsendbuffer

•  ProtocolB– DRBDwaitsforlocaldiskandremotebuffercache

•  ProtocolC– DRBDwaitsforbothlocalandremotedisk

– SAFEST

YMMV

•  Remembertotestyourself•  Fedora/e1000networkstackissues•  DRBDissoYwaretoo,bugscomeup

OtherUsesHA‐Linux+DRBD+???

•  Mail•  NFS•  MySQL

QuesBons?

Links

•  Linux‐HA:h_p://www.linux‐ha.org•  DRBD:h_p://www.drbd.org

Demo(backupplan)

Host:c6a

Host:c6a

Host:c6a

Host:c6a

Host:c6a

Host:c6b

Host:c6b

Host:c6a