deploying and monitoring rails

Deploying and Monitoring

Ruby on Rails A practical guide

Mathias Meyer and Jonathan Weiss, 25.05.2009

Peritor GmbH

2

Who are we?

Jonathan Weiss

• Consultant for Peritor GmbH in Berlin

• Specialized in Rails, Scaling, Deployment, and Code Review

• Webistrano - Rails deployment tool

• FreeBSD Rubygems and Ruby on Rails maintainer

http://www.peritor.com

http://blog.innerewut.de

3

Who are we?

Mathias Meyer

• Consultant for Peritor GmbH in Berlin

• Specialized in Rails, Performance/Database Tuning, Deployment, and Refactoring

• Maintainer of Macistrano – Webistrano-Client for Mac OS X (and soon the iPhone), run_later and acts_as_solr

http://www.paperplanes.de

4

Agenda

Infrastructure Deployment Monitoring Q & A Practical Session

5

Infrastructure Infrastructure

6

Simple Rails Setup

One Rails instance handles all requests

Rails is single-threaded: There is only one concurrent request

7

Rails Setup

8

Rails Setup

9

Typical Rails Setup

• A load-balancer distributes the incoming requests

• Some load-balancers will deliver static requests themselves

• Several Rails instances handle all requests

• Number of concurrent requests equals number of Rails instances

10

Rails Setup Options

11

Deployment Questions

Mongrel?

Apache?

Ebb?

FastCGI?

mod_rails?

Thin?

Nginx?

Lighttpd?

Pound? mod_proxy_balancer?

HA-Proxy?

Phusion Passenger?

Load-balancer? Proxy?

Rails Application Server?

Swiftiply? Reverse Proxy?

Pen?

12

What we are going to cover today

Rails Application Server

• FastCGI

• Mongrel

• mod_rails / Phussion Passenger

• JRuby + Glassfish & Co.

Proxy/Web Server

• Apache2

• Nginx

• Lighttpd

• HA-Proxy

13

FastCGI

14

FastCGI

• Protocol to communicate with long-running CGI applications

• Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd

• Proxy local and remote FastCGI instances

• Oldest way of deploying Rails

• Deprecated and unstable

• Hard to debug (FastCGI protocol)

15

FastCGI

16

FastCGI

18

Mongrel

19

Mongrel

• Developed by Zed Shaw as an alternative to FastCGI

• Complete HTTP-Server that can load arbitrary Ruby-servlets

• Built-in Rails support

20

Mongrel

21

Mongrel

Apache 2.2 – mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound

22

Mongrel Cluster

Utility to manage several Mongrel instances

23

Mongrel Cluster

Control Mongrels

24

Mongrel and Apache 2.2

Define Mongrel Cluster in Apache

25

Simple Mongrel and Apache 2.2

Redirect all traffic to the Mongel cluster

26

A more complex example

• Redirect dynamic requests

• Serve static content

• Support cached pages

• Support maintenance page

• Enable client-side caching of images, stylesheets, and JavaScript

• Compress output if supported

27

Mongrel

• Very robust

• Strict HTTP parser

• Easy to debug (HTTP!)

• Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer

• Can be a bit difficult to setup (mongrel_cluster, ports, Apache)

• Not so easy on mass/virtual hosting

28

mod_rails

29

mod_rails a.k.a Phusion Passenger

• Module for Apache 2.2 (and Nginx)

• Allows Apache to control Rails instances

• Apache starts and stops application instances depending on the application load

• Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.)

• Only manages Rails on one host - no remote instances

• Combine with HTTP-Proxy / balancing solution

30

Install Phusion Passenger

Install Apache module

Load and activate in Apache

31

Install Phusion Passenger

Install Passenger with nginx (installs custom nginx)

Configure nginx

32

Customized Phusion Passenger

Control Rails instance number

33

Control Phusion Passenger

Restart after deployment:

34

Phusion Passenger

One machine

35

Phusion Passenger

One machine Multiple machines

36

Phusion Passenger

• Ready for production

• Makes setup easier – on the single machine level

• Multiple servers still require load balancer

• Suitable for mass-hosting

• upcoming standard way of deploying Rails

37

JRuby

38

• Ruby Runtime on the Java Virtual Machine

• Implemented in Java and Ruby

• Compiles Ruby into Java-bytecode

• Integrates with Java code and libraries

• Java’s promises of native threads and JIT

• Allows for Ruby/Rails applications to be packaged as WAR files

• WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, Jetty, …

39

JRuby on Rails

40

JRuby on Glassfish

One machine

41

JRuby on Glassfish

One machine Multiple machines

42

Setup JRuby on Glassfish

1. Download JRuby and Glassfish

2. From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.html

43

Warble Configuration

Define min/max Rails runtimes

44

Glassfish for development

Shrunk-down version of Glassfish v3 - packaged as a gem

Can run any Rack-based application

Installation:

• $ jruby -S gem install glassfish

• $ glassfish

• There is no Step 3

45

Rails Setup

46

Proxy Options

47

Proxy Requirements

• Hide cluster backend from the user

• Load-balancer backend instances

• Recognize down hosts

• Fair scheduler

• (Deliver static content)

48

Apache 2.2

• Apache 2.2 introduced mod_proxy_balancer

• mod_proxy_balancer can speak to multiple backends and balance requests

• Apache can acts as a pure proxy or can also serve static files

49

Apache 2.2




50

Apache 2.2




51

Apache 2.2

Pro

• Stable, robust, and mature

• Many people know how to work with Apache

• Integrates well with other modules (SVN, DAV, Auth, …)

Con

• Apache can be complicated to configure

• The stock Apache is quite resource-hungry compared to pure proxy solutions

52

Nginx – From Russia with love

• Nginx - popular Russian webserver with good proxy support

• Can load-balance multiple backends and deliver static content

• Quite popular with Mongrel as the Rails backend

• Recent Passenger support

53

Nginx Configuration

Simple proxy example

Get complete version here:

http://brainspl.at/nginx.conf.txt

54

Nginx

Pro

• Stable, robust, and fast

• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files

• Simpler configuration file

• Can directly talk to memcached - SSI

Con

• More documentation would be nice

• No equivalent for many Apache modules

55

Lighttpd

• Lightweight and fast webserver

• Balancing proxy support

• Good FastCGI support

• Used to be popular – until Mongrel came around

56

Lighttpd Configuration


57

Lighttpd

Pro

• Fast and lightweight

• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files

• Simpler configuration file

Con

• Unstable for some people

• Slow development cycle

• More documentation would be nice

• Configuration file can be too simple (virtual host aliasing)

• No equivalent for many Apache modules

• (No Passenger support)

58

HA-Proxy

• HAProxy – reliable, high performance TCP/HTTP load balancer

• Proxying and content inspection

• No content serving, just a proxy

• Mature proxy module (fair scheduler)

• ACL support

See also similar Pound and Pen

59

HAProxy


60

HAProxy

Pro

• Mature, stable, robust, and fast

• TCP and HTTP balancing

Con

• Few Rails examples

• Usually not needed in a Rails setup

61

Recommended Setups

62

Small Site

Recommendation

Apache 2.2 with mod_rails / Phusion Passenger

63

Medium Site

Recommendation

• Apache/Nginx as the frontend proxy

• Passenger/mod_rails as the backend

• Deliver static files with Apache

64

Large Rails Setup

Recommendation

• Redundant load-balancer

• Redundant proxy / web

• Passenger/mod_rails

65

Heavy Static Files

Recommendation

• Deliver static assets through separate web server farm

• Passenger/mod_rails

66

Java Shop

Recommendation

• Deliver a Rails-WAR file and you are done

• Integrate with existing Java landscape and infrastructure

68

Application Server Handler

69


70


71

Rack

72

Rack Handler

73

Rack

74

Rack Middleware

75

Rack Middleware

• Authentication (HTTP, OpenID)

• Sessions

• Routing

• Caching

• Logging

• Debugging

76

Support

Frameworks

• Rails

• Merb

• Camping

• Sinatra

• Mack

• Maveric

• Halcyon

• Ramaze

• Rum

• Vintage

• Waves

Web Server

• CGI

• FastCGI

• Mongrel

• Phusion Passenger

• JRuby Servlets

• WEBrick

• Ebb

• Thin

• LiteSpeed

• Swiftcore

• Unicorn

77

Remarks

78

Ruby Enterprise Edition

• Copy-On-Write patches to Ruby 1.8

• Saves memory when spawning several Rails instances

• Used by Phusion Passenger if available

79

Thin, Ebb, Evented Mongrel & Co.

• Alternatives to Mongrel

• Claim to be faster, lighter, and what have you

• Rendering “Hello World” is usually not your bottleneck

Stick with stable and robust Mongrel, or better yet, with Passenger

80

Infrastructure Deployment

82

Deployment Options

83

Deployment Options

84

Deployment Options

85

Deployment Options

86

What does Capistrano do?

87

Capistrano Deployment Cycle

88

Requirements

89

What doesn’t Capistrano do?

• Plan your initial server setup

• Configure basic services

90

Basic Ingredients

• The cap command

• Variables

• Roles

• Tasks

• Namespaces

91

Basic Ingredients - cap

Your one-stop deployment shop

92

Basic Ingredients - Variables

• Configure basic project information

• Override Capistrano’s default assumptions

• Once set, variables are available globally

• Defined using the set method

93

Basic Ingredients - Roles

• Define types of servers

• Default roles

• :www

• :app

• :db

• All can point to the same server

• But all three must be defined

• At least one database server needs to be primary

94

Basic Ingredients - Roles

Define custom roles as you please

Can be reused when defining tasks

95

Basic Ingredients - Tasks

• Define an atomic set of actions for Capistrano

• Can be called from the command line

• Or other tasks

96

Basic Ingredients - Tasks

To find all the tasks available in your project, use

97

Basic Ingredients - Namespaces

Group tasks together logically

Namespaces and tasks are separated with “:”

98

Get Your Capistrano On

99


Capfile, the place to include more recipes

100


config/deploy.rb, application specific configuration

101

Capistrano’s Defaults

• Your SCM is Subversion

• Deployment directory is /u/apps/#{application_name}

• User for SCM and SSH is the currently logged-in user

• Commands are run with sudo

102


103


• Capistrano expects a directory structure

• Can be created with cap deploy:setup

104

The Deployment Lifecycle

105


Check the prerequisites:

106


Set up your application for the first time

107


The initial deployment

1. Checks the revision from the local machine

2. Checks out the code on the remote machines

3. Sets a link called current pointing to the lates release

4. Runs the migrations

5. Fires up application servers

108


109


Subsequent deployments

1. Checks the revision from the local machine

2. Checks out the code on the remote machines

3. Updates current link

4. Restarts application servers

110


111

Common Capistrano Tasks

Deploy and run migrations

Run only the migrations

Restart application servers

Rollback to the previous release

112

Have Your Shell, and Eat It Too

113

Invoking any Command

114

Deployment Strategies

115


Direct checkout (from scratch) on the servers

116


Keep a cached copy of the current SCM head

117


Check out locally and transfer

118

Give it a little Spin

Capistrano expects a script called spin in script/process

For Passenger

119

Customizing Capistrano

120

Write your own Tasks

121

Write your own Tasks

122

Namespace your Tasks

123

Callbacks

Execute a task before another runs

Execute a task after another has finished

Callbacks are run in the order they’re defined

124

Useful Variables

125

Transactions

126

Transactions

127

Transactions

128

Transactions

129

The Rest

• Gem dependencies

• Support for deploying through gateway servers

• Parallel execution on multiple servers

• Server setup with deprec gem

130

Vlad the Deployer

131

Vlad the Deployer

• Simple, Rake-based deployment solution

• Comes with defaults for most popular SCM, app and web servers

132

Vlad the Deployer

Load

Configure

133

Vlad the Deployer

Deployment

134

Vlad the Deployer

• Simple, sometimes too simple

• No callback mechanism, instead the Rake syntax must be used

• Very non-verbose output, using --trace recommended

135

Chef

• System Integration Framework

• Written in Ruby – inspired by Puppet

• Manage multiple nodes

• Configuration

• Packages

• Custom scripts and commands

136

Architecture

137

Recipes

Install and upgrade packages

138

Recipes

Create and update remote files

139

Recipes

Create and update remote files

140

chef-deploy

Enable

Use in recipe

141

Chef and chef-deploy

• In heavy development: sometimes unstable and hard to debug

• Be prepared to get your hands dirty

• Better suited for configuration than deployment

• Community recipes at http://github.com/opscode/cookbooks

142

One Click Deploy

143

Webistrano

144

Webistrano

• Web-UI to Capistrano

• Manages projects and their stages

• Alerting and Accounting

• Scriptable and extendable

• BSD License

http://labs.peritor.com/webistrano

145

Macistrano

• Mac-GUI to Webistrano

• Fire and monitor deployments from your desktop

http://github.com/mattmatt/macistrano

146

Infrastructure Practical Session

147

Practical Capistrano

148

EC2 Instances

• Users

• root / railswaycon

• deploy / railswaycon

• MySQL

• root / ‘’

• Apache Configuration

• /etc/apache2/sites-enabled/default

• Rails

• /srv/www

149

Git Repository

• git://github.com/mattmatt/railscasts.git

• Setup the servers

• Deploy!

150

Tasks

• Make memcached restart with every deploy

• (Usually not a good idea, but why not?)

151

Tasks

• Make Apache recognize maintenance.html and display it if exists

152

Maintenance Page

• Create symbolic link to /etc/apache2/mods-available/rewrite.load in /etc/apache2/mods-enabled

• Update virtual host configuration with rewrite rules

• Run cap deploy:web:disable • Hook it into the deployment process

153

Tasks

• Deploy a Rack-based application

• Install Sinatra locally and on the server

• git://github.com/mattmatt/railswaycon.git

• Create deployment configuration

• Create Apache configuration

154

Infrastructure Monitoring

155

The two questions of monitoring

156

1. Is everything still running?

157

2. What are the trends?

158

Monit

• Process-level monitoring

• Checks PID-files, ports, and permissions

• Reacts by executing a script and/or alerting

• Has a web GUI for administering nodes

159

Monit

MySQL

160

Monit

Apache

161

Monit

Mongrel

162

God

• Ruby-based configuration

• Easily extendible

• Can work with event- and poll-based conditions

• Comes with less recipes than Monit

163

God

Mongrel

164

Munin

• Host-level monitoring

• Master periodically asks nodes for local data

• Checks system resources and records historical data

• Allows to recognize trends and make predictions

• Alerting support

165

Munin

166

Munin

167

Munin

168

Exception Monitoring

• Monitor errors raised by your application

• Usually notifying you by email

• Good old exception_notification Plugin

• Exceptions as a Service

169

Hoptoad

170

Exceptional

171

New Relic

• Rails Performance monitoring

• Live view into your application

172

Other Tools

• Nagios

• Big Brother

• New Relic RPM

• FiveRuns

• JMX

173

Infrastructure Advanced Deployment

174

Search

175

Search

Full text search

Can become very slow on big data sets

176

Full Text Search Engine

Separate Service

• Creates full text index

• Application queries search daemon

• Index update through application or database

Possible Engines

• Ferret

• Sphinx

• Solr

• Lucene

• …

177

Search Slave

Database replication slave

• Has complete dataset

• Migrates slow search queries from master

• Can use different database table engine

178

Database Index

PostgreSQL Tsearch2

• Core since 8.3

• Allows to create full text index on multiple columns or arbitrary SQL expressions

MySQL MyISAM FULLTEXT index

• Only works with MySQL <= 5.0 and MyISAM tables

• Full text index on multiple columns

179

What to use?

Different characteristics

• Real-time updates and stale data

• Lost updates

• Performance

• Document content and format

• Complexity

180

Background Processing

181

Problem

Long running tasks

• Resizing uploaded images

• Mailing

• Computing an expensive operation

• Accessing slow back-ends

When running inside request-response-cycle

• Blocks user

• Blocks Rails instance

• Hard to monitor and debug

182

Solution

Asynchronous processing in the background

Message/Queue Scheduler

183

Background Processing

184

Options

Options for message bus:

• Database

• Amazon SQS

• Drb

• Memcache

• ActiveMQ

• …

Options for background process:

• (Ruby) Daemon

• Cron job with script/runner

• Forked process

• Delayed Job / BJ / (Backgroundrb)

• run_later

• ….

185

Database/Ruby daemon example

186

Cloud Infrastructure

187

Cloud Infrastructure

Servers come and go

• You do not know your servers before deploying

• Restarting is the same as introducing a new machine

You can’t hardcode IPs

database.yml

188

Solution #1

Query and manually adjust

• Servers do not change that often

• New nodes probably need manual intervention

• Use AWS ElasticIPs to ease the pain

Set servers dynamically AWS Elastic IP

189

Solution #2

Use a central directory service

• A central place to manage your running instances

• Instances query the directory and react

190

Solution #2

Use a central directory service

• A central place to manage your running instances

• Instances query the directory and react

191

Central Directory

Different Implementations

• File on S3

• SimpleDB

• A complete service, capable of monitoring and controlling your instances

192

Q & A

193

193

Peritor GmbH

Teutonenstraße 16 14129 Berlin

Telefon: +49 (0)30 69 20 09 84 0 Telefax: +49 (0)30 69 20 09 84 9

Internet: www.peritor.com E-Mail: [email protected]

Peritor GmbH - Alle Rechte vorbehalten

deploying and monitoring rails

Technology