plone in the cloud - an on-demand cms hosted on amazon ec2
DESCRIPTION
The rise of utility computing platforms such as Amazon EC2 has made it more feasible to build turnkey hosted solutions on top of open source software. Learn how we built PondCMS, a turnkey CMS deployed to Amazon's EC2. This presentation discusses the advantages of hosting Plone sites in the elastic computing cloud and some of the challenges we faced. Watch the video from this talk at http://pycon.blip.tv/file/1949202/TRANSCRIPT
![Page 1: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/1.jpg)
Plone in the CloudA CMS hosted on Amazon EC2
Nate Aune (Jazkarta.com)PyCon 2009Chicago, IL
1Tuesday, April 7, 2009
![Page 2: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/2.jpg)
• President of Jazkarta (based in Boston)
• 3 Full-time and 10 freelance developers
• Python-based open source web applications
• Members of core Plone framework team
• Member of the Plone Foundation Board
2Tuesday, April 7, 2009
![Page 3: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/3.jpg)
In the next 30 min...
• Amazon Web Services (AWS)
• AWS with Python
• Plone with AWS
• Tools for launching & remote deployment
• Challenges of using AWS
• Monitoring and maintaining EC2 instances
3Tuesday, April 7, 2009
![Page 4: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/4.jpg)
What is Amazon Web Services?
4Tuesday, April 7, 2009
![Page 5: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/5.jpg)
What is Amazon Web Services?
• Suite of elastic IT infrastructure services
• Requisition computing power and storage
• Run any application you want on any OS
5Tuesday, April 7, 2009
![Page 6: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/6.jpg)
Benefits of AWS
• Metered pricing (per hour / per GB)
• no large up-front capital investments
• only pay for what you use
• Scaleable hosting architecture
• Redundancy for high availability
6Tuesday, April 7, 2009
![Page 7: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/7.jpg)
AWS Architecture
• EC2 = Elastic Computing Cloud
• S3 = Simple Storage Solution
• EBS = Elastic Block Storage
• Elastic IP = Static IP addresses
7Tuesday, April 7, 2009
![Page 8: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/8.jpg)
Elastic Computing Cloud (EC2)amazon.com/ec2
• Use an existing AMI (Amazon Machine Image) to launch a new instance
• Or create your own AMIs (even paid ones)
• Attach EBS volumes to your EC2 instance
8Tuesday, April 7, 2009
![Page 9: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/9.jpg)
Elastic Block Storage (EBS)
9Tuesday, April 7, 2009
![Page 10: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/10.jpg)
So how does Python play into all this?
10Tuesday, April 7, 2009
![Page 11: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/11.jpg)
Enter Boto11Tuesday, April 7, 2009
![Page 12: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/12.jpg)
What is Boto?
• Python package that provides interfaces to Amazon Web Services
• Thanks to author Mitch Garnaat!
• Boto supports:
• S3 via the REST API
• EC2 via the Query API
• and more...
12Tuesday, April 7, 2009
![Page 13: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/13.jpg)
Install Boto & Bototools$ sudo easy_install boto
$ svn co http://bototools.googlecode.com/svn/trunk/ bototools
$ cd bototools
$ sudo python setup.py install
$ sudo python ./src/bototools/postinstall.py -p /usr/local/bin
• This will put these commands in /usr/local/bin:
• ec2start, ec2stop, ec2ip, ec2ls, ec2img
• s3ls, s3rm, s3cp
13Tuesday, April 7, 2009
![Page 14: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/14.jpg)
Using Bototools
• Set these environment variables:
• AWS_ACCESS_KEY
• AWS_SECRET_ACCESS_KEY
• EC2_SSH_KEY
• Can also put in ~/.boto or /etc/boto.cfg
14Tuesday, April 7, 2009
![Page 15: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/15.jpg)
Find an AMI
• We will use the Ubuntu 8.10 Intrepid (32-bit) from Alestic.com - ami-7cfd1a15
• More can be found at:http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=171
15Tuesday, April 7, 2009
![Page 16: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/16.jpg)
Launch an instance$ ec2start -a ami-7cfd1a15 -k gsg-keypair
Reservation:r-0f26a666
[Instance:i-8e3154e7]
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 pending None 2009-03-25T01:49:24.000Z
...wait...
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 running None 2009-03-25T01:49:24.000Z ec2-174-129-238-8.compute-1.amazonaws.com
16Tuesday, April 7, 2009
![Page 17: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/17.jpg)
Associate a static IP$ ec2ip -g
Address:174.129.238.8
$ ec2ip -l
174.129.238.8 ->
$ ec2ip -a 174.129.238.8 i-8e3154e7
True
$ ec2ip -l
174.129.238.8 -> i-8e3154e7
• Note: once you assign a static IP, you won't be able to access the machine using the original IP.
17Tuesday, April 7, 2009
![Page 18: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/18.jpg)
Log in
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 running None 2009-03-25T01:49:24.000Z ec2-174-129-238-8.compute-1.amazonaws.com
$ ssh -i ~/id_rsa-gsg-keypair [email protected]
Linux domU-12-31-39-00-E5-14 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686
root@domU-12-31-39-00-E5-14:~#
• Time to play!
18Tuesday, April 7, 2009
![Page 19: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/19.jpg)
Stop the instance
• Be sure to stop the instance or you will be charged!
$ ec2stop i-8e3154e7
Stopping ['i-8e3154e7']
$ ec2ls
Reservation r-0f26a666
i-8e3154e7 terminated None 2009-03-25T01:49:24.000Z
19Tuesday, April 7, 2009
![Page 20: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/20.jpg)
Use the AWS Console
20Tuesday, April 7, 2009
![Page 21: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/21.jpg)
Or use ElasticFox
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609&categoryID=88
• Firefox extension to start/stop EC2 instances, assign IPs, associate EBS volumes and create keypairs.
21Tuesday, April 7, 2009
![Page 22: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/22.jpg)
Case studyPlone on EC2
22Tuesday, April 7, 2009
![Page 23: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/23.jpg)
What is Plone?
• Open source content management system (CMS)
• Built on top of Zope
• Written in Python
• Large international community
• Used by Oxfam, NASA, Novell, Akamai, FSF, etc.
• more listed at http://plone.net/sites
23Tuesday, April 7, 2009
![Page 24: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/24.jpg)
Why Plone on EC2?
• Too few Zope hosting providers
• Avoid renting rackspace in co-lo
• Flexible and scalable architecture
• Complete control of servers
+
24Tuesday, April 7, 2009
![Page 25: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/25.jpg)
What is PondCMS?
• Hosted and supported version of Plone
• Eliminate the complexities of hosting
• Turnkey - automated and repeatable
• Bundled with certified add-ons
PondCMS = Plone On Demand
www.jazkarta.com/pondcms
25Tuesday, April 7, 2009
![Page 26: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/26.jpg)
PondCMS Technology stack
!"#$%&!'()Virtual Servers
*#"+,-./+Operating System
*#"+,-./+Web App Server
*#"+,-./+PondCMS + add-ons
0%1&2
HTTP proxy
Caching proxy
Load Balancer
26Tuesday, April 7, 2009
![Page 27: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/27.jpg)
One server deploymentHTTP
Apache
(HTTP)
Varnish
(Caching)
Pound
(Load balancing)
Zope Client #1
1
Zope Client #1
2
Zope Client #n
n
Zeo Server
Zeo Backup
27Tuesday, April 7, 2009
![Page 28: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/28.jpg)
Add Zeo clients as demand increases
28Tuesday, April 7, 2009
![Page 29: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/29.jpg)
HTTP
public IP: 72.44.50.237
Apache
Varnish
Pound
zope-1.web.private:8222
Zope client #1
1
zope-2.web.private:8222
Zope client #2
2
zope-n.web.private:8222
Zope client #n
n
Data.fs Backup
Apache logs
Data.fs
Frontend server
zeo-1.web.private:8221
Zeo server #1
Elastic Block Storage
Elastic Block Storage
Application Servers
Database server
Website visitor
Simple Storage Solution (S3)
Data
files
Multi server deployment
29Tuesday, April 7, 2009
![Page 30: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/30.jpg)
Goal: automate everything
Scripts for repeatable deployment
1. Bootstrap the server
2. Install dependencies
3. Install PondCMS (includes Plone + add-ons)
4. Configure backups
5. Create the Plone site and Apache vhosts
30Tuesday, April 7, 2009
![Page 31: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/31.jpg)
1. Bootstrap server
1. Create the EC2 instance
2. Create an EBS volume
3. Associate volume with EC2 instance
4. Acquire IP address
5. Associate IP with EC2 instance
31Tuesday, April 7, 2009
![Page 32: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/32.jpg)
2. Install dependencies1. Install packages with apt-get
• zope, python, varnish, apache2, pound, postfix
2. Initialize EBS volume and mount as ext3 fs
3. Store Apache log files on EBS volume
4. Register server with Landscape
5. Set up Pound to start automatically
6. Set up Pound to point to Zeo clients
32Tuesday, April 7, 2009
![Page 33: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/33.jpg)
3. Install PondCMS
1. Make a 'zope' user
2. Create a virtualenv
3. Make Zope data dirs on EBS volume
4. Download PondCMS tarball release (incl. Plone)
5. Install PondCMS into virtualenv
33Tuesday, April 7, 2009
![Page 34: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/34.jpg)
4. Configure backups
• Configure logfile rotation
• Setup backup scripts (using repozo)
• Setup sync of backups to S3 (using Duplicity)
• Create cron jobs for backups (daily, weekly)
34Tuesday, April 7, 2009
![Page 35: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/35.jpg)
5. Create Plone site & vhosts
1. Use Fabric (remote deployment tool) to:
• Make Plone site incl. CacheFu setup
• Make vhost for accessing site on port 80
2. Create Apache vhosts for:
• supervisor.domainname.com (start/stop Zope)
• admin.domainname.com (ZMI access)
35Tuesday, April 7, 2009
![Page 36: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/36.jpg)
What is Fabric?
36Tuesday, April 7, 2009
![Page 37: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/37.jpg)
What is Fabric?http://www.nongnu.org/fab/
• Simple remote deployment tool
• Similar to Capistrano (Ruby on Rails)
• Deploy applications to multiple similar hosts
37Tuesday, April 7, 2009
![Page 38: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/38.jpg)
@require('host', 'site_id')
@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename')
def createplonesite(host=None, user=None, key=None, site_id=None,
public_host=None):
"""Create a new plone site """
if not site_id:
site_id = host
installzcml(host=host, user=user, key=key, package='pondcms.basicapp')
restart(host=host, user=user, key=key)
put('createplonesite.py', '/tmp/createplonesite.py')
s = SERVICE+'/bin/utility run /tmp/createplonesite.py -a pondcms.basicapp -s '+site_id
if public_host:
s += ' --site-url='+public_host
run(s)
Setup site Fabric script
38Tuesday, April 7, 2009
![Page 39: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/39.jpg)
@require('host', 'vhost_name')
@fabricset(host='fab_hosts', user='fab_user', key='fab_key_filename')
def setupvhost(host=None, user=None, key=None, vhost_name=None):
"""Setup an apache2 vhost"""
site = VHOST_TEMPL % {'vhost_name': vhost_name}
tmpfile = '/tmp/'+get('fab_timestamp')+'_vhost'
f = open(tmpfile, 'w')
f.write(site)
f.close()
targetf = vhost_name+'.conf'
target = '/etc/apache2/sites-available/' + targetf
put(tmpfile, target)
run('a2ensite vhost_name')
run('/etc/init.d/apache2 reload')
Setup vhost Fabric script
39Tuesday, April 7, 2009
![Page 40: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/40.jpg)
Running Fabric scripts$ cd fabric
$ source bin/active
(fabric)$ cd fabs/pondcms
(fabric)$ fab createplonesite:host=test.pondcms.net,user=root,key=~/id_rsa-gsg-keypair,site_id=test.pondcms.net,public_host=http://test.pondcms.net
...
(fabric)$ fab setupvhost:host=test.pondcms.net,user=root,key=~/id_rsa-gsg-keypair,vhost_name=test.pondcms.net
40Tuesday, April 7, 2009
![Page 41: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/41.jpg)
Challenges• Persistent data must be stored on EBS
• Static IPs
• Mail sent from EC2 instances flagged as spam
• Monitoring and maintenance of VMs
• Landscape
• ZenOss
• Nagios
• Hyperic41Tuesday, April 7, 2009
![Page 42: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/42.jpg)
42Tuesday, April 7, 2009
![Page 43: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/43.jpg)
AWS Status history
43Tuesday, April 7, 2009
![Page 44: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/44.jpg)
Landscapemonitoring / updates
• http://landscape.canonical.com
44Tuesday, April 7, 2009
![Page 45: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/45.jpg)
Landscape update pkgs
45Tuesday, April 7, 2009
![Page 46: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/46.jpg)
Thank you!Questions?
Follow me on Twitter: natea
Or email me at:natea (at) jazkarta (dot) com
46Tuesday, April 7, 2009
![Page 47: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/47.jpg)
More info
• Amazon EC2: http://amazon.com/ec2
• Boto: http://code.google.com/p/boto
• Bototools: http://code.google.com/p/bototools/
• Fabric: http://www.nongnu.org/fabric
• Fabulatr: http://code.google.com/p/fabulatr/
• Plone: http://plone.org
• PondCMS: http://jazkarta.com/pondcms
47Tuesday, April 7, 2009
![Page 48: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/48.jpg)
Other resources• Ubuntu on EC2:
http://groups.google.com/group/ec2ubuntu
• Ubuntu AMIs: http://www.alestic.com
• Beta program: http://www.ubuntu.com/ec2
• Landscape: http://landscape.canonical.com
• vm-builder-ec2: http://packages.ubuntu.com/intrepid-updates/utils/python-vm-builder-ec2
• Enomaly's Elastic Computing Platform:http://src.enomaly.com/
48Tuesday, April 7, 2009
![Page 49: Plone in the Cloud - an on-demand CMS hosted on Amazon EC2](https://reader034.vdocuments.us/reader034/viewer/2022052315/55512f05b4c905f1528b4f00/html5/thumbnails/49.jpg)
Photo credits
• http://www.flickr.com/photos/docman/66137352/
• http://www.flickr.com/photos/thomashawk/126534906/
• http://www.flickr.com/photos/moeview/498544386/
• http://www.flickr.com/photos/shellysblogger/2464969383/
• http://www.flickr.com/photos/mhw/56386662/
• http://www.flickr.com/photos/mrtopf/1541527579/
49Tuesday, April 7, 2009