avoidingdown+meusinglinux( ( high(availability( rust - avoiding downtime using...• raid or raid...
TRANSCRIPT
Introduction & Agenda
• Downtime is not cheap • What is High Availability = not a back up! • Raid or Raid over the network (DRBD) • SANs and clustered applications • The Linux cluster stack • Cluster management with Pacemaker • Disaster Recovery / Linking sites • DRBD and the Cloud
DRBD HA and DR
Down+me = $$$
• Lost revenue • Lost reputa+on • Almost every business these days has a
cri+cal database or file system that they could not do without.
• HP es+mates $31,705 per hour 3.8 hours a year totaling $481,900/ year
• 40% internet traffic stops when Google goes down
Survey: Cri+cal “IT-‐Systems in the medium-‐sized business“, 2013 Techconsult, on behalf of HP Germany Basis: 300 medium-‐sized companies from Germany
Down+me = $$$
“YOU LOST THE DATABASE?!?!”
• “Ummm, can you ping ____?” • “I can’t seem to reach our inventory system.” • “Can you try pulling up this record?”
Devo+on to Duty -‐ xkcd
Why Monitor?
• Hardware dies • DDOS afacks • Set it and forget it mentality • Internet connec+on • Security programs
Hos+ng / XaaS
• Reliability • Security • Mul+-‐tenant architecture • Scalability • Up+me
The Pillars of IT Security
Integrity
Types of Clustering Solu+ons
• Hardware redundancy • SAN solu+ons • NAS boxes • External hard drives or JBODS • So#ware Solu+ons
Recovery Time/Point Objec+ves
What is Raid? Is it enough?
RAID
Microsok Library hfp://msdn.microsok.com/en-‐us/library/aa226166(v=sql.70).aspx#sql7perkune_moreinfo
What Could Go Wrong
• Your shiny new hardware will fail • Single points of failures are dangerous • Dropped alerts • Internet outage • Power outage
• Easy to implement -‐ high cost per TB • Large SLAs -‐ quality of technicians • Management via GUI • Scalable -‐ with the right packages • SAN maintenance -‐ learning curve • Off site replica+on is expensive
SAN/NAS
Single Point of Failure
SAN
NFS
MySQL
VM’s
Piualls
• High ini+al and ongoing costs • Vender lock in is required • Ongoing worry of voiding the warrantee • Maintenance is tricky and ongoing • It is a black box, typically Solaris based • Cannot add or remove features • It is s+ll a single point of failure
Sokware Only Solu+ons
Things to look for: • Synchronous or Asynchronous replica+on • Stability / maturity • Time to recovery • Chance of data loss • Onsite / offsite • Is it real +me (live) or snap shots
Asynchronous Architecture
Synchronous Architecture
Secondary
Layer Cake of Replica+on
• Virtualiza+on
• Applica+on
• File system
• Object store
• Block layer
hfp://images.pinkcakebox.com/cake696.jpg
Cluster Cake Fail
Common Issues / Piualls
• File locking • Network conges+on • Data consistency / data corrup+on • High overhead and/or addi+onal CPU cycles • Asynchronous or even back up based • Require ongoing licensing and royal+es
DRBD
• Completely hardware and applica+on agnos+c • German engineering • In development since 2001 • Created by LINBIT founder and CEO
Phillip Reisner • DRBD built into the na+ve Linux kernel as of
2.6.33 • Ships in all major Linux distribu+ons • Does not void RHEL or Oracle support
DRBD Users
A DRBD Cluster Stack
LAN
Server
RAID
High Speed RAID-Controller
High Speed NIC
Replication Network
shared nothing
Storage
Fully Redundant System
Storage 1 Storage 2
Ac+ve Passive
MySQL.com
Fully Redundant System
Storage 1 Storage 2
Ac+ve Passive
Passive Ac+ve
MySQL.com Ausweb.com
Heartbeat/Corosync: The Comm Layer
• These are the communica+on tools of the cluster
• “Are you dead?” • “Are you alive?” • Heartbeat is seasoned and stable
(reliability = HA) • Corosync is newer and under
development
Pacemaker
The Linux Cluster Resource Manager • The powerful and bossy cluster manager • Manages all aspects of system • Decides who is alive and primary • Well known • Widely deployed • Does not require applica+ons have specific
plugins
Pacemaker : Sleep All Night
• It lets you sleep though the night even if there’s a failure.
• Highly Configurable • Used with a number of clustering
tools / File Systems • Very powerful if done well
Disastrous if done wrong
Linux HA Stack
Disaster Recovery / Offsite Replica+on
• True Disaster Recovery happens live • Interval based snapshots no longer meet
todays SLA requirements • DRBD does real-‐+me replica+on on-‐site and
off-‐site • DRBD Proxy tool mi+gates throughput
constraints and latency-‐ highly configurable
Real-‐+me Disaster Recovery
Scaling DRBD
• DRBD Proxy is typically done in 3 node configura+ons.
• Extremely configurable • Proxy mi+gates bandwidth constraints and
latency • Can replicate across 4 machines even across
distances
3 node HA / DR
Location A Live Site
Location B DR Site
Proxy Proxy Proxy
4 node DR + Ac+ve-‐Ac+ve HA
Location A Live Site
Location B DR Site
Proxy Proxy Proxy Proxy
Dedicated Proxy-‐Many Resources
Location A Live Site
Location B DR Site
Proxy Proxy
Dedicated Server Dedicated Server
How to apply this in your cloud
hfp://www.gamesparks.com/wp-‐content/uploads/2013/07/the-‐cloud.jpg
DRBD works in the cloud and AWS VPC
DRBD can be used as backing storage for ISCSI
On na+ve bare hardware or as part of your hardware or sokware appliance
HA with Nagios!
• Filesystem (which has many symlinks in it) • MySQL • PostgreSQL • Crond • Ndo2db • The Nagios applica+on itself • A Virtual IP
Q+A Jeremy Rust
[email protected] @NerdHacker
877-DRBD247
www.linkedin.com/in/RustJeremy
DRBD.org Linbit.com
Linux-HA.org
DRBD 9 the future
DRBD 8 Branch build structure
DRBD 9 Branch build structure
2 Full redundant systems
Storage 1 Storage 2
Ac+ve Passive
Passive Ac+ve
MySQL.com Ausweb.com