cloud, cache, and configs

Cloud, Cache, and ConfigsWordCamp NYC 2012

Saturday, June 9, 12

Scott TaylorLead PHP Developer, eMusic

@wonderboymusicwww.scotty-t.com

#projectmanagementsoftware


http://www.scotty-t.com

http://www.scotty-t.com

eMusic

MultisiteRegionalized ContentRegionalized Caching

Shared Content across SitesTons of Custom Post Types

Post FormatsTons of Web Services


eMusic Architecture

• 12-24 Amazon EC2 instances (CentOS)

• 4 MySQL (1 Write, 3 Read)

• 4 Memcached (~28GB RAM)

• Amazon S3 / Cloudfront CDN

• PHP 5.3 / MySQL 5.5

• PECL: APC, Memcached, HTTP

• Batcache


APC = duh

• Essential for PHP

• Opcode cache

• apc.shm_size = 64M or higher


Logs

• Never allow any PHP-related notices, errors, warnings, exceptions, etc

• Check access logs regularly for 404s, 500s etc

• make a constant for toggling debug logging


Production Data

• Always pull down, never push up

• We only push imported legacy content up

• output buffering

• code filtering

• more to come on this....


ob_start( $callback )

ob_start();

echo ‘Daryl’;

$daryl = ob_get_clean();

Output Buffering


ob_start( function ( $data ) {

return str_replace( $urls_array, EMUSIC_CURRENT_HOST, $data );

} );


Configs

• Mandatory machine configs

• HyperDB config

• Overridable sunrise.php


Machine Config• DB credentials local to that machine

• Amazon S3 bucket

• Web Service Endpoints

• Memcached Servers

if ( file_exists( ‘/wp-config/config.php’ ) ) { require_once( ‘/wp-config/config.php’ );} else {

die( ‘You must have a local config!’ );}


Memcached

• Memcached PHP Extension (not Memcache)

• Can be used locally (127.0.0.1)

• Memcached Redux supports wp_cache_get_/set_multi( )

• Johnny Cache


Batcache

• Full-page caching

• Can be configured

• You can partition cache by unique values

• Loads before plugins - any code you need has to be duped or added early (sunrise.php)


class batcache {

// This is the base configuration. You can edit these variables or move them into your wp-config.php file. var $max_age = 300;

// Expire batcache items aged this many seconds (zero to disable batcache)

var $remote = 0;

// Zero disables sending buffers to remote datacenters (req/sec is never sent)

var $times = 5;

// Only batcache a page after it is accessed this many times... (two or more)

var $seconds = 120;

// ...in this many seconds (zero to ignore this and use batcache immediately)

var $group = 'batcache';

// Name of memcached group. You can simulate a cache flush by changing this.

var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY );

// If you conditionally serve different content, put the variable values here.

var $headers = array( 'X-nananana' => 'Batcache' );

. . . . }


Sunrise• Used to alter multisite context

• sets $current_blog and $current_site

• filters all URL functions to resolve all URLs to your current domain

• registers custom locations for media

• filters Admin URLs


switch_to_blog( $blog_id )

• All dynamic functions need to account for this

• Shared content needs to resolve proper URLs

• Different sites have different media locations


$current_blog = new stdClass();$current_blog->site_id = 1;$current_blog->archived = 0;$current_blog->mature = 0;$current_blog->spam = 0;$current_blog->deleted = 0;$current_blog->lang_id = 0;$current_blog->public = 1;$current_blog->registered = '2011-02-20 03:38:22';$current_blog->last_updated = $_SERVER['REQUEST_TIME'];$current_blog->domain = EMUSIC_CURRENT_HOST;

function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) { if ( $blog_id === $prev_blog_id ) return;

global $current_blog, $emusic_paths; $current_blog->blog_id = $blog_id; $current_blog->path = $emusic_paths[$blog_id];}

emusic_switch_to_blog( $the_id );

add_action( 'switch_blog', 'emusic_switch_to_blog', 10, 2 );

$blog_id = $the_id;$site_id = 1;

$current_site = new stdClass();$current_site->blog_id = $the_id;$current_site->id = 1;$current_site->domain = EMUSIC_CURRENT_HOST;$current_site->site_name = 'eMusic';$current_site->path = $the_path;

Fix switch_to_blog()


add_filter( 'pre_option_upload_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$id}/files";

return $_SERVER['DOCUMENT_ROOT'] . '/' . EMUSIC_UPLOADS;} );

add_filter( 'pre_option_upload_url_path', function () { $id = get_current_blog_id(); if ( 1 < $id ) return 'http://' . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files";

return 'http://' . EMUSIC_CURRENT_HOST . '/' . EMUSIC_UPLOADS;} );

add_filter( 'pre_option_siteurl', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra;} );

add_filter( 'pre_option_home', function () { global $current_blog; $extra = rtrim( $current_blog->path, '/' ); return 'http://' . EMUSIC_CURRENT_HOST . $extra;} );

Filter URLs


Plugins

• Filter active network plugins (don’t rely on database being correct)

• Filter each site’s plugins (if you have a manageable number)

• Use classes, not a bunch of functions

• Extend before you copy / paste


require_once( 'site-configs/global.php' );

if ( $the_id > 1 ) { define( 'UPLOADBLOGSDIR', 0 ); define( 'UPLOADS', 0 ); define( 'BLOGUPLOADDIR', $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$the_id}/files" );

switch ( $the_id ) { case 2: require_once( 'site-configs/bbpress.php' ); break;

case 3: require_once( 'site-configs/dots.php' ); break;

case 5: require_once( 'site-configs/support.php' ); break; }

add_filter( 'pre_option_template', function () { return 'dark'; } );} else { require_once( 'site-configs/emusic.php' );}

Site Configs


add_filter( 'pre_site_option_active_sitewide_plugins', function () { return array( 'batcache/batcache.php' => 1, 'akismet/akismet.php' => 1, 'avatar/avatar.php' => 1, 'bundle/bundle.php' => 1, 'cloud/cloud.php' => 1, 'download/download.php' => 1, 'emusic-notifications/emusic-notifications.php' => 1, 'emusic-ratings/emusic-ratings.php' => 1, 'emusic-xml-rpc/emusic-xml-rpc.php' => 1, 'johnny-cache/johnny-cache.php' => 1, 'like-buttons/like-buttons.php' => 1, 'members/members.php' => 1, //'minify/minify.php' => 1, 'apc-admin/apc-admin.php' => 1 //'debug-bar/debug-bar.php' => 1 );} );

Global Plugins


Site Plugins

add_filter( 'pre_option_active_plugins', function () { return array( 'artist-images/artist-images.php', 'catalog-comments/catalog-comments.php', 'emusic-post-types/emusic-post-types.php', 'discography.php', 'emusic-radio/emusic-radio.php', 'gravityforms/gravityforms.php', 'super-ghetto/super-ghetto.php' //,'theme-check/theme-check.php' );} );


class MyPlugin { function init() { add_action( ‘init’, array( $this, ‘register’ ) ); } function register() {}}

$my_plugin = new MyPlugin();$my_plugin->init();

class MyPlugin { function init() { add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) ); } function register() {}}

MyPlugin::init();


class FeaturePack extends eMusicPostTypes implements PostType { ..........}

Share code when possible


MySQL

• Use 5.5, better handlng of weird multibyte strings

• Use InnoDB, not MyISAM, pretty much in all cases

• HyperDB handles scaling for you

• Benchmark queries, don’t be afraid to roll your own SQL, tables, use $wpdb


HyperDB

• Can be empty locally

• Inherits wp-config DB defaults

• contains functions for replication lag detection (when used with mk-heartbeat)


Themes

• Theme setup is a class

• Extend before you repeat

• Classes are better than prefixing function names


class Theme_17Dots extends Regionalization { function __construct() { global $dots_regions_tax_map, $dots_regions_map; $this->regions_map = $dots_regions_map; $this->regions_tax_map = $dots_regions_tax_map; parent::__construct(); } function init() { add_action( 'init', array( $this, 'register' ) ); add_action( 'after_setup_theme', array( $this, 'setup' ) ); add_action( 'add_meta_boxes_post', array( $this, 'boxes' ) ); add_action( 'save_post', array( $this, 'save' ), 10, 2 ); add_filter( 'embed_oembed_html', '_feature_youtube_add_wmode' ); } . . . . . . . }

Theme Config in functions.php


class Regionalization { var $regions_map; function __construct() { add_filter( 'manage_posts_columns', array( $this, 'manage_columns' ) ); add_action( 'manage_posts_custom_column', array( $this, 'manage_custom_column' ), 10, 2 ); add_filter( 'posts_clauses', array( $this, 'clauses' ), 10, 2 ); add_filter( 'manage_edit-post_sortable_columns',array( $this, 'sortables' ) ); add_filter( 'pre_get_posts', array( $this, 'pre_posts' ) ); } . . . . . . }

Use base classes


pre_get_posts function regionalize( $query ) { global $regions_map; if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) {

$types = get_post_types( array( 'publicly_queryable' => true, '_builtin' => false ) ); $tax_region = array( 'taxonomy' => 'region', 'field' => 'term_id', 'terms' => array( $regions_map[ 'ALL' ], $regions_map[ THE_REGION ] ), 'operator' => 'IN' );

if ( is_home() ) { $query->set( 'posts_per_page', 10 ); } else if ( is_search() && get_option( 'editorial_search_enabled' ) ) { $ctx = get_query_var( 'search_context' ); $query->set( 's', stripslashes( urldecode( get_query_var( 's' ) ) ) );

if ( in_array( $ctx, array( 'features', 'features-books' ) ) ) { $query->set( 'posts_per_page', 48 ); } else { $query->set( 'posts_per_page', 12 ); }

if ( in_array( $ctx, array( 'books', 'features-books' ) ) ) { foreach ( $types as $type ) if ( false === strpos( $type, 'book' ) ) unset( $types[$type] ); }

if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; } else if ( is_tag() ) { if ( empty( $query->posts ) && $query->is_paged() ) $query->is_paged = false; }

//error_log( 'REGIONALIZING' );

if ( !is_post_type_archive() ) $query->set( 'post_type', array_keys( $types ) );

$query->set( 'tax_query', array( $tax_region ) ); }

return $query; }


Assets

• Use remote storage

• Use a CDN

• Replace hosts using output buffer

• Cloud - pieces of W3 Total Cache


Minify

• JS / CSS concatenation speed up your front-end loading / perceived loading

• Minify is automagic

• HTML5 CSS properties are automatically inflated

• Admin tool to cache-bust URLs

• Auto-locking while files are generated


Web Services• cURL PHP extension

• curl and curl_multi()

• Memcached is essential

• hooks into parse_request to load data along with WordPress, allows us to cause WP to 404 and bail early when required data response fails

• Parallelization with curl_multi()


class eMusicRequest { var $request; var $sub_request; var $path; var $page; function load( $page = '' ) { if ( !empty( $page ) ) $this->page = $page; $file = $this->path . $this->page . '.php'; if ( file_exists( $file ) ) require_once( $file ); } function parse() { $requests = array( $this->request, $this->sub_request ); foreach ( $requests as $request ) { if ( !empty( $request ) ) { $keys = array_keys( get_class_vars( get_class( $request ) ) ); foreach ( $keys as $var ) { if ( !empty( $request->$var ) ) { $GLOBALS[$var] = $request->$var; } } } } }}


class DarkRequest extends eMusicRequest { var $genre; var $post_type; function init( $request ) { global $_GENRES; $vars =& $request->query_vars;

......... }

}


switch ( get_current_blog_id() ) {case 1: $_dark_request = new DarkRequest(); add_action( 'parse_request', array( $_dark_request, 'init' ) ); break;case 4: $_my_emusic_request = new MyEMusicRequest(); add_action( 'parse_request', array( $_my_emusic_request, 'init' ) ); break;}


<?phpclass HomeRequest extends RequestMap { var $recommendations; function __construct() { parent::__construct(); if ( !get_option( 'recs_enabled' ) ) return; if ( is_user_logged_in() && 'US' === THE_REGION ) { $user = wp_get_current_user(); $user_id = isset( $_GET['user_id'] ) ? $_GET['user_id'] : $user->ID; $params = array( 'userId' => $user_id, 'return' => true ); $this->add( get_user_recommendations( $params ), array( $this, 'parse_recommendations' ) ); $this->send(); } } function parse_recommendations( $data ) { if ( empty( $data['recommendations'] ) ) return; $data = $data['recommendations']; if ( !empty( $data ) && isset( $data[key($data)]['items'] ) && !empty( $data[key($data)]['items'] ) ) { $this->recommendations = array_slice( $data[key($data)]['items'], 0, 18 ); shuffle( $this->recommendations ); foreach ( $this->recommendations as &$rec ) { $rec = array( 'work_id' => $rec['catalogId'] ); } } }}


<?php class RequestMap extends API { private $requests; private $responses; private $ttl; private $useCache = true; protected $error = false;

public function __construct() { $this->flush(); }

public function is_error() { return $this->error; } public function flush() { $this->requests = array(); } public function getTtl() { if ( empty( $this->ttl ) ) { $this->ttl = CACHE::API_CACHE_TTL; } return $this->ttl; } public function add( $url, $callback, $vars = array() ) { $params = new stdClass(); $params->url = $url; $params->callback = $callback; $params->params = (array) $vars; $this->requests[] = $params; } private function exec( $item, $response ) { $params = array_merge( array( $response ), $item->params ); call_user_func_array( $item->callback, $params ); } public function send() { if ( !empty( $this->requests ) ) { $this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache ); if ( is_array( $this->responses ) ) { foreach ( $this->responses as $i => $response ) { if ( !empty( $this->requests[$i] ) ) { $this->exec( $this->requests[$i], self::parse_response( $response ) ); } } } } }}


class API {

public static function batch( $urls, $ttl = '', $usecache = true ) { $response = array(); ob_start(); if ( empty( $ttl ) ) { $ttl = CACHE::API_CACHE_TTL; } if ( is_array( $urls ) ) { if ( $usecache ) { foreach ( $urls as $index => $url ) { $in = Cache::get( Cache::API, $url );

if ( $in ) { $response[$index] = $in; unset( $urls[$index] ); } } }

$keys = array_keys( $urls ); } $calls = self::multi_request( $urls );

. . . . . . . .


if ( is_array( $calls ) && count( $calls ) > 0 ) { $calls = array_combine( $keys, array_values( $calls ) );

foreach ( $calls as $index => $c ) { if ( $c ) { $response[$index] = self::parse_response( $c ); if ( isset( $response[$index]['status']['code'] ) && $response[$index]['status']['code'] < 400 ) { if ( isset( $response[$index]['results'] ) && !empty( $response[$index]['results'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } else { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else if ( !isset( $response[$index]['status']['code'] ) ) { Cache::put( Cache::API, $urls[$index], $response[$index], $ttl ); } } else { $response[$index] = null; } } } else if ( !empty( $calls ) && isset( $urls[0] ) ) { $data = self::parse_response( $calls ); if ( isset( $data['status']['code'] ) && $data['status']['code'] < 400 ) { if ( isset( $data['results'] ) && !empty( $data['results'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } else { Cache::put( Cache::API, $urls[0], $data, $ttl ); } } else if ( !isset( $data['status']['code'] ) ) { Cache::put( Cache::API, $urls[0], $data, $ttl ); } $response[] = $data; } ob_end_clean(); return $response;


Because we used classes, everything is

abstracted


Custom Authentication

• WP stores slashed passwords

• wp_authentication arguments are slashed

• Inherit your base system’s rules

• DO NOT sanitize email / password (WordPress does by default)

• User tables in this case are really a transient cache


Questions / complaints?


cloud, cache, and configs

Technology