architecture and design mysql powered applications by peter zaitsev meetup sao paulo may2014
DESCRIPTION
Palestra do Peter Zaitsev no Meetup MySQL-BR de maio-2014 sobre Arquiteturas e Projeto de Aplicações com MySQLTRANSCRIPT
Architecture and Design of MySQL Powered Applications
Peter ZaitsevMay 8, 2014
About Presentation
What's in your Toolbox ?
What Questions you should ask ?
Common MySQL Architecture
Patterns
2
LiveJournal Architecture 10 Years Ago3
10 Years ago we see
Load Balancers
FrontEnd vs BackEnd Web Server Separation
MemcacheD (Caching)
Global Database for Shared Data
Read Write Splitting
Sharding “Database Clusters”
4
History Lesson
MySQL is quite Mature
Battle tested Architecture Patterns
Most Problems have proven solutions
5
YOUR TOOLBOX NOW
Well… certain things have changed
6
Very Powerful Hardware
• 16GB to 512GBLots of
Memory
• 1000 IOPS to 100.000 IOPS
• 10ms latency to 0.1ms IO Latency
Very Fast Storage
• 4 Cores to 32 CoresMany CPU
Cores
7
Reasonable Software
•MySQL
•Linux Kernel
•File Systems
•Etc..
Can use this
Hardware Pretty Well
8
Cloud
Dynamic Scalability
Throw Away Server Mentality
Agility
Less involved Operations
9
Automation
Development Testing Deployment Operations
10
New Languages and Frameworks
More Agile
Less “system” code
More Client Side Development
Less Database Dependence
ORM
11
Multiple “Clients”
Desktop Web Browsers
Mobile Web Browsers
“Apps”
API
12
Not Only “MySQL”
• Memcache, Redis
• RabbitMQ
• Cassandra, MongoDB
• Hadoop
• Column Store Engines, Vertica
Right Tools
for Right Job!
13
More HA Options
Mature MySQL Replication
Percona XtraDB Cluster, Galera
MySQL Cluster
Proprietary cloud HA
14
Ready solutions for Replication Management
MHA
PRM
Continuent Tungsten
15
Making Sharding Better
• Clustrix
• MemSQL
• ScaleDB
Not Quite MySQL
• ScaleArc, ScaleBase, Tesora
• MySQL Proxy, MaxScale, ProxySQL
Proxy Solutions
• Vitess
• JetPants
• MySQL Fabric
OpenSource Frameworks
16
ARCHITECTURE QUESTIONS
17
Basics
Make some Right decisions Early
Your first Architecture will not be perfect
Some poor choices are very expensive to fix
18
Consultants are like Lawyers
You can waste a lot of money with them
Getting right advice on the right topics will save you a lot of trouble in the end
19
Dimensions to consider
Application Scale
High Availability
Team Experience
20
Approach to Scaling
Avoid Over-Engineering
Give yourself some
runway
Do Some Capacity Planning
21
High Availability
Real High Availability is Really Expensive
Medium Availability is getting more affordable
People are often leading cause of downtime
22
Team Experience
Chose Solutions which your Team can Support
Often Developers know little about Databases
Right Development, Operational Process Is a Key
Tools to Support your Process
• Percona Toolkit, Percona Cloud Tools
• Systematically find and review new queries
23
Simplicity
• Less complexity lower costsUse fewer
components when possible
• Monitor
• Troubleshoot
Each components needs experience
to
• “Follow the Crowd”
• Unless you have a reason not toThere is safety in
numbers
24
World of “Small” Applications
• Really
Majority of Applications are
Small
• Especially with support of caching
May well run on single database
instance
• Active intranet site for company with 200K employees
• Ecommerce site supporting $10M+ in Sales
Such “Small” Applications can
be significant
25
Lets do Some Math
Modern MySQL
Instance Handle 100K
QPS
20 Queries per User
Interaction
5K User Interactions Per Second
144M User Interactions a
Day (Considering
3x peak to average)
10M Daily Active users
(with Medium
Engagement)
26
PRACTICAL CHOICES
Basic Architecture Patterns
27
Baseline
Single MySQL Instance
No Caching
No HA
No Supplemental Technologies
28
Simple High Availability
HA with DBaaS (Amazon RDS)
HA with MySQL Replication
HA with Percona XtraDB Cluster
29
What do you need to Scale ?
Reads
Writes Data Size
30
Load Management Tricks
Move the load
In Time (Doing Batch Processing
at Night)
In Space (Use Slave for Reports)
31
Queueing
Used by Most high volume architectures
Deal with Load Spikes
Increase Reliability
Scale parts of the system which need to
Many solutions
• RabbitMQ
• Redis
• Gearman
32
Scaling Reads
Replication (and Read Write Splitting)
Caching (Memcache, Redis etc)
Percona XtraDB Cluster (Read Anywhere)
33
Scaling Writes
• Using Different Servers for Different Parts of Application
Functional Partitioning
• Horizontal Partitioning over many servers Sharding
34
Scaling Data Size
•Depends on the Application and Needs
•10TB good rule of thumb
How Much We Can
Practically Store in Single
MySQL Instance
these days
35
How Is it Possible
“Online” Schema Change
MySQL 5.6
Pt-online-schema-change (Percona
Toolkit)
Fast Backup
Percona Xtrabackup
MySQL Enterprise Backup
Modern Filesystem/Volume
Managers
Fast Network
10Gbps affordable
High Performance
Storage
Flash/SSD
Compression
Innodb
TokuDB
36
Summary
MySQL is Mature. Solutions Available for Many Problems
Make few choices right in the beginning
Hardware and Software advances allow to go a long way with simple architectures
37