enterprise-level ee - vector media group...1. run docker containers from pre-built images 2. git...
TRANSCRIPT
Enterprise-level EE:Uptime, Speed, and Scale
Reaching beyond EEtools and techniques to service
enterprise clients
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
vectormediagroup.com/blog
@VectorNYC
@aBenjaminSmithbenjaminsmith.com
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
Caching:Lowest-hanging fruit
on the tree of optimization
Caching» Tag caching caching
» Template caching
» Query caching
» HTTP proxy/accelerator
EE without reverse-proxy cache
» Standard LAMP setup
EE with Varnish Cache
» Sits between users and EE
» Passes through requests to EE
EE with Varnish Cache
» Sits between users and EE
» Passes through requests to EE
» Serves from memory if cached
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
Load Balancingexample.com served by multiple
web servers
No Load Balancing
Varnish Load Balancing
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
Multiple Environments» Local/development
» Staging
» QA
» Production
The old way...1. Checkout new branch from master
2. <do something amazing with code>
3. Add new fields to your local EE instance
4. Merge into staging branch
5. Deploy to staging server
6. Add new fields to your staging EE instance
7. Email the client for them to review!
Boo that» How to stage multiple WIP features at once?
» Field ID conflicts
» Which features are currently staged?
» Stale data on Staging/QA servers
DockerDocker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run
» consistent and reproducible
» isolated software
» lightweight
The new way...vmg stage ProjectA feature-new-modal
1. Deploys feature to feature-new-modal.projecta.vmg.com
2. DB is latest snapshot from live site (< 1 day old)
3. Runs tests
4. Send notifcations to HipChat
How is this possible?» Docker images built nightly from live site data
» Wildcard DNS
» Docker volume persistance
» Environment variable configuration (phpdotenv)
$db['default']['database'] = getenv('DB_NAME');$db['default']['username'] = getenv('DB_USER');$db['default']['hostname'] = getenv('DB_HOST');$db['default']['password'] = getenv('DB_PASS');$db['default']['db_debug'] = getenv('DB_DEBUG');
Build the database!
» Nightly script runs uploads MySQL dump to Amazon S3
» Docker run command pulls in latest DB nightly and builds MySQL container image
» Image is accessible from same machine as Docker LAMP stack
./run.sh build-db clientName
What we have now» Docker image with runnable custom MySQL server and latest
DB snapshot
» Can be "run" in milliseconds
Stage a feature!» Project name: "clientName"
» Git branch: "feature-new-modal"
./run.sh stage clientName feature-new-modal
Stage a feature!
1. run Docker containers from pre-built images
2. git clone
3. composer install --no-dev
4. configure Apache to set environment variables
5. dynamic .htaccess config
6. set file/dir permissions
7. let proxy server know we're here!
Proxy server» Listens on port 80/443
» Docker web servers, listening on random port
» new Docker images tell proxy server the port assignment
github.com/jwilder/nginx-proxy
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
Basic MySQL optimizations» MySQL schema tweaks
» indices
» column types
» MySQL 5.6+ (Percona)
» InnoDB
» Adding new channel fields?
MySQL doesn't like to
ALTER
ALTERing tables leads tolocking and downtime
pt-online-schema-changealters a table’s structure without blocking reads or writes
pt-online-schema-change
1. create an empty copy of the table to alter
2. ALTER the new table
3. copy rows from the original table into the new table
4. moves away the original table and replaces it with the new one
5. drops the original table
build it!
./ee-schema-change eefield add
./ee-schema-change eefield add
1. Run the command!
» Adds 2 new columns exp_channel_data
» Adds 1 row to exp_channel_fields
2. Configure the field as needed in the ExpressionEngine CP
1. Intro
2. In-memory Caching
3. Load Balancing
4. Multi-environment setup with Docker
5. Scaling MySQL Tables
6. Monitoring / Logging
EE as a componentin a larger application
» ExpressionEngine CMS/framework
» MySQL database
» Cron jobs (backup scripts)
» Elasticsearch cluster
» Worker queues
» Cache invalidation
» Data integrations/importing (CRM, ERP, API)
The old way...Scanning log files is not enough
» Which server do they live on?
» How to search? Grep/tail?
» Standardized format?
The new way...
The new way...Decentralized logging
for log shipping
for storage/access/high-availability
for visualzation / reporting
THANK YOU.— Ben