increasing mysql productivity

Post on 19-May-2015

6.550 Views

Category:

Technology

0 Downloads

Preview:

Click to see full reader

DESCRIPTION

What sets apart the every day web developer from the knowledgeable and highly desired developer that can provide a better return on investment (ROI) for their business or organization?This session will provide MySQL development tips, tricks and best practices that elevate your knowledge and use of MySQL from the perspective of a MySQL Expert. In this master class we will touch on many aspects that make an impact for a successful application and minimize additional work or re-work from commonly observed poor development practices. From architectural design, coding, development practices, testability, debugging, security, application instrumentation, optimal deployment and monitoring we will touch much of the software development lifecycle.This class is all about improving your knowledge and maximizing the strengths of the MySQL product for increasing productivity and increasing ROI the MySQL way.

TRANSCRIPT

Increasing MySQL Productivity

From design to implementation

Improving MySQL Productivity from Design to Implementation - 2010.06

Title

Ronald Bradfordhttp://ronaldbradford.com

2010.06

Improving MySQL Productivity from Design to Implementation - 2010.06

Objectives

MySQL development tips and tricks

Identify poor development practices

Recommended best practices

Improve knowledge and MySQL skills

Optimal deployment & support

Productivity ROI• Increase your productivity• Improve development quality

Improving MySQL Productivity from Design to Implementation - 2010.06

Session Scope

Design

Development

Security

Testing

Instrumentation

Implementation

Support

R & D

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Data Integrity, Storage Engines, Diagnostics, SQL Syntax, Replication, Optimizing SQL, Meta Data, Query Analyzer, Workbench, Data Import/Export, Caching with Memcached, MySQL joins, Database Administration

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

MySQL Idiosyncrasies that BITE

What are MySQL Defaults

The impact of using defaults

Ensuring better compatibility

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Storage Engines in ReviewWhat is the pluggable storage architectureBuilt In EnginesPlug in EnginesEngine CharacteristicsHow to create a storage engine

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

InnoDB Usage and Diagnostics

MySQL transactional storage engine

Clustered and Secondary Indexes

Understanding innoDB threads

Performance metrics

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

What do you mean "SQL Syntax Error"

The ANSI standard

Similarities that produce differences

What's missing

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Everything you didn't know about MySQL Replication

Replication Basics

Asynchronous

Topology options

Replication usages

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Importing and Exporting Data with MySQL

Loading Data (csv, tsv, fixed, report)

Performance benefits

Dumping Data

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Capturing, Analyzing and Optimizing your SQL

Various capture methods

Bulk and per query analysis

Optimization techniques and examples

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Query Analysis with MySQL Enterprise Monitor

GUI SQL Analysis

Monitoring

Alerts and Advisors

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Using MySQL Meta Data Effectively

the INFORMATION_SCHEMA

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Navigating MySQL Stored Procedures & Funtions, Views and Triggers

The MySQL Syntax

Limitations

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Data Caching with Memcached and MySQL

Leveraging Caching

Write thru/Write back

UDF support

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

Design and Development using MySQL Workbench

GUI Schema Development

Re-engineering schemas

Query browser integration

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

JoinFu - The Art of SQL

Join Basics

And/Or conditions

Hierarchal data

Aggregation and ranking

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

ODTUG Presentations Review

The Five Minute DBA

Quick DBA tips

Configuration

Monitoring

http://www.odtugkaleidoscope.com/MySQL.html

Improving MySQL Productivity from Design to Implementation - 2010.06

Design

Improving MySQL Productivity from Design to Implementation - 2010.06

Design Best Practices

Data Types

Schema Management

Sharding 101

Higher Availability

Improving MySQL Productivity from Design to Implementation - 2010.06

About MySQL Data Types

Numeric Data Types

Oracle has 1

MySQL has 9

TINYINT,SMALLINT,MEDIUMINT,INT, BIGINT,FLOAT,DOUBLE,DECIMAL,BIT

Improving MySQL Productivity from Design to Implementation - 2010.06

Optimal Auto Increment Primary Key

Don't use BIGINT AUTO_INCREMENT

Use INT UNSIGNED AUTO_INCREMENT

BIGINT is 8 bytes

INT is 4 Bytes

INT UNSIGNED stores 4.3 billion values

Productivity ROI• Can reduce index space by 50+% • Better memory usage, less I/O

Improving MySQL Productivity from Design to Implementation - 2010.06

INT(1)

This is not 1 byte, it's 4 bytes

(1) is only for client display only

Client with 10+ flags using INT(1)

40 bytes reduced to 10 bytes

2 bytes using bit operators

INT(1) is not what it looks like

Horror Stories

Improving MySQL Productivity from Design to Implementation - 2010.06

Dates

MySQL supports

DATE, DATETIME, TIMESTAMP, YEAR

TIMESTAMP for Epoch values

TIMESTAMP is 4 bytes

DATETIME is 8 bytes

Supports DEFAULT CURRENT_TIMESTAMP

Neither store milliseconds

Improving MySQL Productivity from Design to Implementation - 2010.06

The beauty of ENUM

Values Check Constraint

Ideal for static codes

Compact - i.e. 1 byte for 'True', 'False'

Human readable values

5.1 Non blocking ALTER

Improving MySQL Productivity from Design to Implementation - 2010.06

TRUE/FALSE Datatype Examples

CREATE TABLE enums ( flag1 CHAR(1) NOT NULL COMMENT 'T or F, Y or N', flag2 TINYINT NOT NULL COMMENT '0 or 1', flag3 BIT NOT NULL COMMENT 'True or False', flag4 ENUM ('True','False') NOT NULL);

INSERT INTO enums(flag4) VALUES ('True', 'False');SELECT flag4 FROM enums;

Improving MySQL Productivity from Design to Implementation - 2010.06

InnoDB Primary Key

Primary key is clustered index15/16 fill factor for sequential50% for random

Secondary Indexes hold primary key value

Best Practice - Keep primary keys shortexcept for high Disk I/O

Improving MySQL Productivity from Design to Implementation - 2010.06

Schema Best Practices

Have pristine schema

Use different schemas

Backup copies of tables

Temporary objects

Improving MySQL Productivity from Design to Implementation - 2010.06

Schema Management

Always have current schema.sql

Use Patch/Revert SQL for upgrades

See http://schemasync.org

Productivity ROI• Reproducibility • Upgrade/Downgrade path

Improving MySQL Productivity from Design to Implementation - 2010.06

Best Practices

Separate Schema changes and application code

Productivity ROI• Developers develop • Administrators administer

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

Sharding 101

Design

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

A database table consists of

Columns

Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7

Columns

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7

Columns

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7

Columns

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

A database consists of

Tables

Table Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Table Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Table Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Table Rows

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

Partitioning is?

Grouping like rows in a table together

e.g.

By Date

By Local

By parent grouping

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Partition

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Partition

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

col1 col2 col3 col4 col5 col6 col7row1row2row3row4row5row6row7row8row9row10row11row12row13row14

Partition

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

Sharding is?

Grouping like tables and/or like rows together

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables + Rows

1 1

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables + Rows

2 2

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Tables + Rows

3 3

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Shards

1

2

3

1

2

3

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Shards

1

2

3

1

2

3

4 4 4

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

table1 table2 table3 table4 table5 table6 table7

Shards

1

2

3

1

2

3

4 456

89

7

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

How do you group tables?

That’s the secret

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 - First Steps

First Steps

Consider schema separation

Define a good partition key

Balanced solution

Support rebalancingProductivity ROI

• Write Scalability• Done before application grows in complexity and usage

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101 - Next Steps

Next Steps

Managing Joins

Searching Data

Unique Constraints

Aggregated ReportingProductivity ROI

• Write Scalability• Done before application grows in complexity and usage

Improving MySQL Productivity from Design to Implementation - 2010.06

Manual Partitioning

Ideal for Write Once Data

MySQL supports Atomic table rename

RENAME TABLE current TO old, new TO current;

Improving MySQL Productivity from Design to Implementation - 2010.06

Data Partitioning

Feature of MySQL 5.1

RANGE

LIST

HASH

KEY

http://dev.mysql.com/doc/refman/5.1/en/partitioning.html

Improving MySQL Productivity from Design to Implementation - 2010.06

Sharding 101

Supporting Growth

Design

Improving MySQL Productivity from Design to Implementation - 2010.06

Supporting Growth

Availability

Scalability

Resilience

Seek ProfessionalAdvice

Improving MySQL Productivity from Design to Implementation - 2010.06

Availability

How do you support maintenance?

Software upgrades

Application releases

24x7 operations

Eliminate single points of failure

Productivity ROI• Less Downtime • Greater Sales / Better Reputation

Improving MySQL Productivity from Design to Implementation - 2010.06

Scalability

Read ScalabilityReplication

Write ScalabilitySharding

CachingImplement from day 1

Use best product for purpose

Productivity ROI• Support Demand

Improving MySQL Productivity from Design to Implementation - 2010.06

Scalability Implementation

Implementation

H/W acquisition time

Zero new human resource need

Zero application changes

Productivity ROI• Seamless growth support

Improving MySQL Productivity from Design to Implementation - 2010.06

Resilience

What is your backup strategy?

What is your recovery strategy?

How long does it take?

Have you actually tested it end to end?

Take the surveyhttp://ronaldbradford.com/blog/checked-your-mysql-recovery-process-recently-2010-02-15/

Productivity ROI• Company viability

Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Configuration

Design

Improving MySQL Productivity from Design to Implementation - 2010.06

Configuration Best Practices

Always use transactions

SQL_MODE

Data Integrity

MySQL Idiosyncrasies that BITE Talk

http://ronaldbradford.com/blog/mysql-idiosyncrasies-that-bite-2010-06-28/

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

Design

Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL User Security

Default is woeful

Minimum

$ mysql_secure_installation

Recommended

Operating System

Permissions & Privileges

Improving MySQL Productivity from Design to Implementation - 2010.06

Operating System Security

Defaults are not secure

Never run as 'root' user

Separate Data/Binary Logs/Logs/Configuration/Backups

Individual directory permissions

Productivity ROI• Minimize security risk• Better auditability

Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Installation Best Practice

Best Practice

/mysql /etc /data /binlog /log /mysql-version

/etc/my.cnf/etc/profile.d/mysql.sh/etc/init.d/mysqld

Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Installation Security

Software installed by root$ chown -R root:root /mysql$ chown -R mysql:mysql /mysql/{data,log,binlog/etc}$ chmod 700 /mysql/{data,binlog}$ chmod 750 /mysql/{etc,log}

Improving MySQL Productivity from Design to Implementation - 2010.06

Application User Security

CREATE USER appuser@localhost IDENTIFIED BY 'sakila';GRANT SELECT,INSERT,UPDATE,DELETE ON schema.* TO appuser@localhost;

CREATE USER superman@'%';GRANT ALL ON *.* TO superman@'%';

Best Practice

Normal Practice

See MySQL Idiosyncrasies that BITE session for more information

Improving MySQL Productivity from Design to Implementation - 2010.06

Application User Security Best Practices

Application User (Read/Write Access)

INSERT, UPDATE, DELETE, SELECT

Application Viewer (Read Only Access)

SELECT

Application DBA (Schema Access Only)

CREATE, DROP, CREATE ROUTINE, SELECT, INSERT, UPDATE, DELETE

Productivity ROI• Track Data Security• Separation of responsibilities

Improving MySQL Productivity from Design to Implementation - 2010.06

Why not use GRANT ALL

GRANT ALL ON *.* TO user@’%’

*.* gives you access to all tables in all schemas

@’%’ give you access from any external location

ALL gives youALTER, ALTER ROUTINE, CREATE, CREATE ROUTINE, CREATE TEMPORARY TABLES, CREATE USER, CREATE VIEW, DELETE, DROP, EVENT, EXECUTE, FILE, INDEX, INSERT, LOCK TABLES, PROCESS, REFERENCES, RELOAD, REPLICATION CLIENT, REPLICATION SLAVE, SELECT, SHOW DATABASES, SHOW VIEW, SHUTDOWN, SUPER, TRIGGER, UPDATE, USAGE

See MySQL Idiosyncrasies that BITE session for more information

Improving MySQL Productivity from Design to Implementation - 2010.06

Why SUPER is bad?

SUPER

Bypasses read_only

Bypasses init_connect

Can Disable binary logging

Change configuration dynamically

No reserved connection

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Improving MySQL Productivity from Design to Implementation - 2010.06

Development Best Practices

Version control

General query log

SQL Commenting

SQL Formatting

Future proofing

Embrace CLI

Caching

Improving MySQL Productivity from Design to Implementation - 2010.06

Version Control

Use for all development

Especially single user development

Use for schema management

Building block of automated build/release

Productivity ROI• Central repository of code• Track code changes by developer• Create reproducable releases

Improving MySQL Productivity from Design to Implementation - 2010.06

General Query Log

Turn on for all development environments

Logs all SQL Statements

Aggregate and Email results to developer

See Capturing, Analyzing and Optimizing SQL Presentation

Productivity ROI• Developers are seeing SQL in operation• Enables access to SQL to analyze

Improving MySQL Productivity from Design to Implementation - 2010.06

Leveraging General Query Log

For single user development environment

In SQL Session

In Application

Run Function/Process

In SQL Session

mysql> SELECT 'Function X Start';

mysql> SELECT 'Function X End';

Improving MySQL Productivity from Design to Implementation - 2010.06

Slow Query Log

Turn on for all development environments

Logs slow SQL Statements ( > 1 second)

Aggregate and Email results to developer

See Capturing, Analyzing and Optimizing SQL Presentation

Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Commenting

Identify queries by code function

Identify OLTP / Batch / Cache queries

SELECT /* XXX123 */ col1, ...

UPDATE /* YYY999 */ ...

SELECT /* Batch */...Productivity ROI

• DBA/SA can identify code function and purpose quickly

Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Commenting (2)

26 Query SELECT /* 5m cache */ ..... 26 Query SELECT /* ViewPost */ t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT /* batch */ meta_key, meta_value FROM wp_usermeta WHERE user_id = 2

Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Formatting

Create single line queries

Don't embed newlines

Enables per line analysis by CLI tools

Productivity ROI• DBA/SA can use simple CLI tools including grep,awk,cut etc for SQL analysis

Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Formatting (2)

26 Query SELECT FOUND_ROWS() 26 Query SELECT post_id,start,end,allday,rpt,IF(end>='2010-06-04 00:00:00',1,0) AS active FROM wp_ec3_schedule WHERE post_id IN (2849,2842,2836,2824,2812,2680,2770,2784) ORDER BY start

26 Query SELECT * FROM wp_users WHERE user_login = 'ronald' 26 Query SELECT t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag') AND tr.object_id IN (2849, 2842, 2836, 2824, 2812, 2680, 2813, 2800, 2770, 2784) ORDER BY t.name ASC 26 Query SELECT meta_key, meta_value FROM wp_usermeta WHERE user_id = 2

Improving MySQL Productivity from Design to Implementation - 2010.06

SQL Analysis

Bulk trending analysis

Volume of SQL statements

Query Execution Plan (QEP)

Online v Batch/Cache SQL via commenting

Productivity ROI• Identify bottlenecks ASAP without load• Iterative Design feedback

Improving MySQL Productivity from Design to Implementation - 2010.06

EXPLAIN Basics

EXPLAIN SELECT ...

No key used

key length

Extra - Using Filesort, Using Temporary

Improving MySQL Productivity from Design to Implementation - 2010.06

Future proofing your SQL

Specify INSERT columns

INSERT INTO table(a,b,c) VALUES(...)

SELECT * is generally bad

What columns are actually used in codeTEXT/BLOB can cause extra disk I/ONew columns can change performance

Productivity ROI• Reduce likelihood of runtime errors when structural changes to objects

Improving MySQL Productivity from Design to Implementation - 2010.06

Avoid Fancy Constructs

DELAYED

IGNORE

LOW PRIORITY

REPLACE

Productivity ROI• Changes ACID and statement precedence• May have additional performance overhead

Improving MySQL Productivity from Design to Implementation - 2010.06

Using Deterministic Functions

Use '2010-06-21' instead of CURDATE()

Same for NOW()

Leverage Query Cache (if enabled)

Productivity ROI• Leverages database caching when enabled• Allows testing via parameterization

Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors

Remove Duplicate Code

Productivity ROI• Less code to maintain• Remove chance of human errors

Improving MySQL Productivity from Design to Implementation - 2010.06

Common SQL Errors

Remove redundant SQL

Use general query log

You may be surprised!

Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Repeating Queries

5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist` WHERE (ArtistID = 196 )5 Query SELECT * FROM `artist` WHERE (ArtistID = 2188 )5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`5 Query SELECT * FROM `artist`

The WRONG way

Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Row at a time (RAT) Processing

SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes'SELECT option_value FROM wp_options WHERE option_name = 'aiosp_title_format' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_show_only_even' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_num_months' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_day_length' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_hide_event_box' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_advanced' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_navigation' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'ec3_disable_popups' LIMIT 1SELECT option_value FROM wp_options WHERE option_name = 'sidebars_widgets' LIMIT 1

Improving MySQL Productivity from Design to Implementation - 2010.06

RAT v CAT Processing

SELECT * FROM activities_theme WHERE theme_parent_id=0SELECT * FROM activities_theme WHERE theme_parent_id=1SELECT * FROM activities_theme WHERE theme_parent_id=2SELECT * FROM activities_theme WHERE theme_parent_id=11SELECT * FROM activities_theme WHERE theme_parent_id=16

SELECT * FROM activities_theme WHERE theme_parent_id in (0,1,2,11,16)

Row (RAT) Processing

Chunk (CAT) Processing

Improving MySQL Productivity from Design to Implementation - 2010.06

Common Coding Errors - Boundary Conditions

The following SQL executed 6,000 times in 5 minute analysis period

0 is an invalid pages_id

SELECT pages_id, pages_livestats_code, pages_title, pages_parent, pages_exhibid, pages_theme, pages_accession_num FROM pages WHERE pages_id = 0

The WRONG way

Improving MySQL Productivity from Design to Implementation - 2010.06

In a highly tuned system the greatest time in a

query is network overhead

Improving MySQL Productivity from Design to Implementation - 2010.06

About Database Connections

Development

Improving MySQL Productivity from Design to Implementation - 2010.06

Defining your database connection

$ cd public_html$ grep "mysql*_connect" * */* */*/* db-disp.php:$cid = mysql_connect("localhost", "museum", "******") or die ('I cannot connect to the database because: ' . mysql_error());test.php:$cid = mysql_connect("localhost", "museum", "******");PMCollection/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass");PMCollection/connection_live.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass");PMCollection/connection_local.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass");PMEcards/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass");core/connection.php: $dbcnx = mysql_connect("$sqlhost", "$sqluser", "$sqlpass");discussion_admin/db_fns.php: $cid = mysql_connect("localhost", "museum", "******");discussion_admin/header.php:// $cid = mysql_connect("localhost", "museum", "******");discussion_admin/inc_title.php: //$cid = mysql_connect("localhost", "museum", "******");discussion_admin/stats.php: //$cid = mysql_connect("localhost", "museum",

The WRONG way

Improving MySQL Productivity from Design to Implementation - 2010.06

Database connection example

class database {

function getConnection($type, $message) { $cp = $this->getConnectionParameter($type); if (empty($cp)) { $message = new message ("fatal", "Unable to determine '$type' ... return; } try { $con = mysqli_connect($cp->host,$cp->user,$cp->passwd,$cp->database); if (!$con) { $message = new message ("fatal", "Unable to obtain a '$type' ... return; } mysqli_query($con, "SET NAMES 'utf8'"); } catch (Exception $e) { $message = new message ("fatal", "Unable to obtain a '$type' ... debug($e->getMessage()); } return $con; }

The RIGHT way

Improving MySQL Productivity from Design to Implementation - 2010.06

Database connection parameters example

$ cat config/database.inc.sample

<? require_once "classes/core/connectionparameter.inc";

$connections = array();

// New connections "description", "host", "user", "passwd", "database", "settings"

$connections["write"] = new connectionparameter( "Write DB Connection", "localhost", "writeuser", "******", "db"); $connections["read"] = new connectionparameter( "Read DB Connection", "localhost", "readuser", "******", "db");?>

Improving MySQL Productivity from Design to Implementation - 2010.06

Database parameterization and version control

Parameters in separate file

config/database.inc.sample

NOTE: .sample in Version Control

Does not override deployment

Productivity ROI• Automated code deployment to multiple environments

Improving MySQL Productivity from Design to Implementation - 2010.06

Horror Stories - Connection Parameters

Changing Connector/J Settings (1)

Have to reboot server

i.e. no way to flush connection pool

How not to code connection parameters

Improving MySQL Productivity from Design to Implementation - 2010.06

Horror Stories - Connection Parameters

Changing Connector/J Settings (2)

Rebuild and redeploy jar

How not to code connection parameters

Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections

Open and close database connections only when necessary

Productivity ROI• Reduce unnecessary database load• Increases page serve volume• Increases true DB throughput

Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections Initialization

$ cat header.php ... $con = getConnection(); ...

if($this_user->user_row["status"]!='ws' && in_array($this_page->getValue(),$page)){ header("Location: /wholesale/permission.php"); exit(); } ... if () { header("Location: abc.php"); exit(); } ... if () { header("Location: xyz.php"); exit(); } ...

The WRONG way

Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections Deconstruction

$ cat footer.php

... $con->close();

The WRONG way

Improving MySQL Productivity from Design to Implementation - 2010.06

Database Connections - Optimal Usage

// do all possible pre-processing first

...

// Then get DB connection $con = database->getConnection("read");

// Close your connection ASAP

// Always future proof objects method calls if ($con != NULL) { $con->close(); }

The RIGHT way

Improving MySQL Productivity from Design to Implementation - 2010.06

Instrumentation

Development

Improving MySQL Productivity from Design to Implementation - 2010.06

Application Instrumentation

Creating one primary abstract DB call

Enable logging of ALL SQL statements

Enable embedded HTML output

Total Execution Time/Count

Individual SQL Execution Time/SQL

Productivity ROI• Enable runtime analysis via browser• No additional tools needed to gather

Copyright 2007 MySQL Inc MySQL - The Best Online Database for modern applicationsPerformance Analysis – Step by Step

117

Application Toolset

Joomla CMS

Copyright 2007 MySQL Inc MySQL - The Best Online Database for modern applicationsPerformance Analysis – Step by Step

118

Cold Fusion

Improving MySQL Productivity from Design to Implementation - 2010.06

Application Instrumentation Benefits

End to End Timing

Component Timing

(i.e. a series of SQL)

Observe as desired

Intelligent Activation

e.g. Page Load exceeds x ms

Improving MySQL Productivity from Design to Implementation - 2010.06

Quick Tips

Development

Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression

Via Database

COMPRESS() UNCOMPRESS()

Via Application

Spread CPU Load to many servers

Database Structure Change

TEXT --> BLOB Productivity ROI• Shorter Backup/Recovery times• Greater volume of data for disk usage

Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression - Example

CREATE TABLE `Mkt` ( `MktID` varchar(10) NOT NULL, `Description` text, PRIMARY KEY (`MktID`)) ENGINE=MyISAM DEFAULT CHARSET=latin1

select min(length(Description)) as min_len, avg(length(Description)) as avg_len, max(length(Description)) as max_lenfrom Mkt;

+---------+---------+---------+| min_len | avg_len | max_len |+---------+---------+---------+| 20 |1122.920 | 7999 | +---------+---------+---------+

Improving MySQL Productivity from Design to Implementation - 2010.06

Using Compression Example (2)

SELECT length(Description) as len_before, length(compress(Description)) as len_afterFROM Mkt WHERE MktID = '42';+------------+-----------+| len_before | len_after |+------------+-----------+| 7810 | 2025 |+------------+-----------+

...WHERE MktID LIKE '6_';+------------+-----------+| len_before | len_after |+------------+-----------+| 11 | 20 || 53 | 53 || 113 | 100 |

Best 74% saving

No benefit for short data

Improving MySQL Productivity from Design to Implementation - 2010.06

Using Text Compression (2)

Prime Candidates

Large Text Fields

Repeating data (e.g. xml elements)

Not Indexed

Not Searched

Improving MySQL Productivity from Design to Implementation - 2010.06

Caching 101

Use caching whenever possible

Caching options

Query results

Objects

HTML component content

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Te

stin

g

Improving MySQL Productivity from Design to Implementation - 2010.06

Testing is not about what works; testing is about breaking your

software

Improving MySQL Productivity from Design to Implementation - 2010.06

Writing testable code

Small code functions

Use Refactoring

Improving MySQL Productivity from Design to Implementation - 2010.06

Query Analysis

Database Size / data volumeVersion

QueryQEPResult set sizeRows affected/retrievedTable structure

Improving MySQL Productivity from Design to Implementation - 2010.06

Function Testability

MVC

API

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Te

stin

g

Instrumentation

Improving MySQL Productivity from Design to Implementation - 2010.06

If you don't have monitoring in place, make it

your top priority

Improving MySQL Productivity from Design to Implementation - 2010.06

Instrumentation Objectives

System Dashboard

Monitoring

Alerting

Public System Status

Improving MySQL Productivity from Design to Implementation - 2010.06

System Dashboard

What is the state of NOW!

sampling 1,5,10 seconds

One page listing most important numbers

Red/Yellow/Green

Private/Public View

Productivity ROI• One starting point for all staff. No manual "let me see" needed.

Improving MySQL Productivity from Design to Implementation - 2010.06

Title

Screen print

DashboardExample

Improving MySQL Productivity from Design to Implementation - 2010.06

Monitoring

Records history over time

Sampling (e.g. 5 minutes / 1 minute)

Graphical Analysis

Improving MySQL Productivity from Design to Implementation - 2010.06

Alerting

Identify key limits

Notification rules

24x7

Improving MySQL Productivity from Design to Implementation - 2010.06

Public Status

Keep your users informed

Planned outage

Unplanned outage

Host elsewhere

Use a Blog

Allows for user feedback

http://status.twitter.com/http://status.blogs.smugmug.com/

Improving MySQL Productivity from Design to Implementation - 2010.06

The Rule of Everything

Everything

Monitor

Measure

Graph

Automate

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Te

stin

g

ImplementationInstrumentation

Improving MySQL Productivity from Design to Implementation - 2010.06

Implementation Objectives

Instrumentation Implementation

No Downtime!

Pre-emptive Load Management

Higher Availability

Automated Deployment

Improving MySQL Productivity from Design to Implementation - 2010.06

Development Team:

We need a maintenance window for software

upgrades and new releases.

Improving MySQL Productivity from Design to Implementation - 2010.06

Management Team:

No Downtime

Improving MySQL Productivity from Design to Implementation - 2010.06

Development Team:

But we need this to fix problems and improve

performance.

Improving MySQL Productivity from Design to Implementation - 2010.06

Management Team:

No Downtime

Improving MySQL Productivity from Design to Implementation - 2010.06

What is your definition of no

downtime?

Improving MySQL Productivity from Design to Implementation - 2010.06

What is your requirements for no downtime?

For example:Serve PagesServe Ads

NOTE:No need to add/change contentSearch dataCharge for ad impressionsSell Merchandise

Improving MySQL Productivity from Design to Implementation - 2010.06

MySQL Higher Availability - First Step

MySQL pair

Master / Fail Over Master

Only as good as the weakest link

Slave SQL_THREAD

Slave Disk I/OProductivity ROI

• Actively tests primary DR strategy

Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management

Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (2)

What is the Twitter "Failed Whale"

Based on page Load time/Volume

Reject early

http://en.oreilly.com/velocity2009/public/schedule/detail/7479http://www.slideshare.net/netik/billions-of-hits-scaling-twitter

Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (3)

Combined with "Failed Whale"

Darkmode

Disable intensive tasks (e.g. Name Search)

Improving MySQL Productivity from Design to Implementation - 2010.06

Pre-emptive Load Management (4)

Data Messaging Queue

Per table/chunk

Write Access

Read Access

No Access

Proactively disable/limit application access

Improving MySQL Productivity from Design to Implementation - 2010.06

Data Messaging Queue Example

Disable Write access to user data

Application changes via notification

Disable option/Notification message

New user registration

Edit profile

Read Access to login/lookup etc

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Te

stin

g

Implementation

Support

Instrumentation

Improving MySQL Productivity from Design to Implementation - 2010.06

Don't change a setting without

evidence to prove/disprove the

Number 1 problem!

Improving MySQL Productivity from Design to Implementation - 2010.06

Where is the bottleneck?

Is the database the problem?

Front End Performance

Improving MySQL Productivity from Design to Implementation - 2010.06

Perception

The website will always be too slow

Identify the true components

End to End time

Database may be only a small portion

Improving MySQL Productivity from Design to Implementation - 2010.06

Front End Tips

• Know your total website load time http://getfirebug.com/

• How much time is actually database related?

• Reduce HTML page size - 15% improvement

• Remove full URL’s, inline css styles

• Reduce/combine css & js files

Improving MySQL Productivity from Design to Implementation - 2010.06

Front End Tips

• Split static content to different ServerName

• Spread static content over multiple ServerNames (e.g. 3)

• Sprites - Combining lightweight images - http://spriteme.org/

• Cookie-less domain name for static content

Improving MySQL Productivity from Design to Implementation - 2010.06

Continual Improvement

Lower MTTR (mean time to recovery)

Lower MTTD (Mean time to detect)

Improving MySQL Productivity from Design to Implementation - 2010.06

Conclusion

Improving MySQL Productivity from Design to Implementation - 2010.06

Security

DesignDevelopment

Te

stin

g

ImplementationInstrumentation

Database Connections

Datatypes

Schema ManagementSh

arding

AvailabilityScalabilityResilience

Version control

SQL Standards Caching

Dashboard - Monitoring - Alerting

NO DOWNTIME

AutomationUser Experience

Break your

system

Support

User Management

Improving MySQL Productivity from Design to Implementation - 2010.06

RonaldBradford.com

MySQL4OracleDBA.com

top related