appscale @ la.rb

The AppScale ProjectPresented by Chris Bunch

(on behalf of the AppScale team)April 14, 2011 @ LA.rb

Overview

• Google App Engine

• AppScale - now with 50% Ruby!

• Neptune - A Ruby DSL for the cloud

Google App Engine

• A web framework introduced in 2008

• Python and Java supported

• Offers a Platform-as-a-Service: Use Google’s APIs to achieve scale

• Upload your app to Google

Quotas

Data Model

• Not relational - semi-structured schema

• Compared to models in Rails

• Exposes a get / put / delete / query interface

Storing Data

• Datastore API - Persistent storage

• Memcache API - Transient storage

• User can set expiration times

• Blobstore API - Store large files

• need to enable billing to use it

Be Social!

• Mail API - Send and receive e-mail

• XMPP API - Send and receive IMs

• Channel API - Creating persistent connections via XMPP

• Use for chat rooms, games, etc.

Background Tasks

• Cron API - Access a URL periodically

• Descriptive language: “every 5 minutes”, “every 1st Sun of Jan, Mar, Dec”, etc.

• Uses a separate cron.yaml file

• TaskQueue API - Within your app, fire off tasks to be done later

Dealing with Users

• Users API: Uses Google Accounts

• Don’t write that ‘forgot password’ page ever again!

• Authorization: via app.yaml:

• anyone, must login, or admin only

When Services Fail

• Originally: failures throw exceptions

• Just catch them all!

• Capabilities API: Check if a service is available

• Datastore, Memcache, and so on

Deploying Your App

• Develop locally on SDK

• Stub implementations of most APIs

• Then deploy to Google

How to Scale

• Limitations on the programming model:

• No filesystem interaction

• 30 second limit per web request

• Language libraries must be on whitelist

• Sandboxed execution

Enter AppScale

• App Engine is easy to use

• but we really want to tinker with the internals!

• Need an open platform to experiment on

• test API implementations

• add new APIs

Enter AppScale

• Lots of NoSQL DBs out there

• Hard to compare DBs

• Configuration and deployment can be complex

• Need one-button deployment

Deploying Your App

• run-instances: Start AppScale

• describe-instances: View cloud metadata

• upload-app: Deploy an App Engine app

• remove-app: Un-deploy an App Engine app

• terminate-instances: Stop AppScale

Deployment Models

• Cloud deployment: Amazon EC2 or Eucalyptus (the open source implementation of the EC2 APIs)

• Just specify how many machines you need

• Non-cloud deployment via Xen or KVM

AppController

• The brains of the outfit

• Runs on every node

• Handles configuration and deployment of all services (including other AppControllers)

• Written in Ruby

Load balancer

• Routes users to their app via nginx

• haproxy makes sure app servers are live

• Can’t assume the user has DNS:

• Thus we wrote the AppLoadBalancer

• Rails app that routes users to apps

• Performs authentication as well

App Server

• We modified the App Engine SDK

• Easier for Python (source included)

• Harder for Java (had to decompile)

• Removed non-scalable API implementations

• Goal: Use open source whenever possible

A Common Feature Request

Database Options

• Open source / open APIs / proprietary

• Master / slave v. peer-to-peer

• Differences in query languages

• Data model (key/val, semi-structured)

• In-memory or persistent

• Data consistency model

• Interfaces - REST / Thrift / libraries

Neptune

• Need a simple way to run compute-intensive jobs

• We have the code from the ‘net

• We have the resources - the cloud

• But the average user does not have the know how

• Our solution: create a domain specific language for configuring cloud apps

• Based on Ruby

Syntax

• It’s as easy as:

neptune :type => “mpi”,

:code => “MpiNQueens”,

:nodes_to_use => 8,

:output => “/mpi/output-1.txt”

Neptune Supports:

• Message Passing Interface (MPI)

• MapReduce

• Unified Parallel C (UPC)

• X10

• Erlang

Extensibility

• Experts can add support for other computational jobs

• Biochemists can run simulations via DFSP and dwSSA

• Embarassingly parallel Monte Carlo simulations

Wrapping It Up

• Thanks to the AppScale team, especially:

• Co-lead Navraj Chohan and advisor Chandra Krintz

• Check us out on the web:

• http://appscale.cs.ucsb.edu

• http://neptune-lang.org