google app engine

Google Confidential and Proprietary 1

Google App Engine

Feb – March, 2010Patrick ChanezonDeveloper AdvocateGoogle Developer Relationschanezon@google.com

Monday, March 8, 2010

Agenda

-App Engine introduction-Why build it?-App Engine tour-What's different?-Wrap up-Questions

Isha Upanishad

4. The spirit is unmoving, one, swifter than the mind. The senses do not reach It as It is ever ahead of them. Though Itself standing still It outstrips those who run. In It the all-pervading air supports the activities of beings

Vayu computing?

Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg

Vayu Computing?

7−10 Then they said to Vayu (Air): "O Vayu! Find out who this great Spirit is." "Yes," he said, and hastened to It. Brahman asked him: "Who are you?" He replied "I am known as Vayu; I am also called Matarisva." Brahman said: "What power is in you, who are so well known?" Vayu replied: "I can carry off all−whatever there is on earth." Brahman put a straw before him and said: "Carry this." He rushed toward it with all his ardour but could not move it. Then he returned from the Spirit and said to the gods: "I could not find out who this Spirit is,"

Kena Upanishad

PlacePostage H

IaaSPaaSSaaS

What is cloud computing?

IaaS value proposition…

Google App Engine

“We wear pagers so you don’t have to”

Google App Engine

-Easy to build-Easy to maintain-Easy to scale

By the numbers

- Built 100K apps- Maintained by 250K developers- Scaled to 250M pageviews daily

semi-transparent collage of apps

gigya Socialize

gigya Socialize - traffic

Gigya Socialize - traffic

App Engine

Time.com

Socialwok

Social networking at scale

>40M Users

PubSubHubbubhttp://code.google.com/p/pubsubhubbub/

The future is distributed: There will be big hubs, many small hubs, and tons of publishers and subscribers.

Publishers, subscribers, and hubs may play multiple roles.

Hell's Kitchen on facebook

Handling peak traffic: mental_floss

Elastic / redundant capacity

Short term events: earthhour.org

http://www.allforgood.org/

http://www.snapabug.com/

エコポイント

Japan: Q&A for our Future

http://www.google.com/tipjar

Chillingo CrystalGaming meets Social

Guerilla Bob

Zombie Dash Angry Birds LITE underground Meltdown

Mission Deep Sea Speed ForgeExtreme

Ravensword:The Fallen King

Angry Birds

Why build it?

Wrench

It's just too difficult

Hosting means hidden costs

• Idle capacity• Software patches & upgrades• License fees• Lots of maintenance• Traffic & utilization forecasting• Upgrades

Cloud development in a box

• SDK & “The Cloud”• Hardware• Networking• Operating system• Application runtime

o Java, Python• Static file serving• Services• Fault tolerance• Load balancing

Distributed Cloud ComputingReliability and scalability

Know your Distributed Memes

-Divide & conquer -Sharding-Tolerate / expect failure- Idempotency-Just scale it up

Rocket scienceMonolithic Computing

Hot spare“Just in case”

Distributed Meme:Expect, tolerate and design for failure

MTFB: O(Years)

MTFB: O(Months)

MTFB: O(Years)

MTFB: O(Months)

MTFB: O(Years)

MTFB: O(Months)

MTFB: O(Years)

MTFB: O(Minutes/Seconds)

Monolithic computing

stop watchsmall

stop watchlarge

Distributed web hosting platform

XRetry

Vertical / horizontal scalability

Vertical scalability

Horizontal scalability

Tolerate, plan for and expect small failures

Easy to deploy & scale

http://www.yourdomain.com/

Thinking about scalability

....the tools, platform and design don't matter too much

Just a few users....

Thinking about scalability

....you must design for scalability

Lots and lots of users...

But the problem is...

http://digg.com/http://slashdot.org/

The real problem is...

When it happens.

Can you afford to puteverything else on hold?

Google App Engine

Leveraging Google'splatform to better serveyour customers

-Great for web apps• Request based, data backed

-Parallel processing-Scales automatically-Available globally-Configuration free-Built-in DoS protections

Distributed datastore

-Arbitrary horizontal scaling

-Arbitrary horizontal scaling-Parallel processing

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance-No deadlocks

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance-No deadlocks-No global schema

Distributed memcache

Memcache

Distributed Meme: Divide & ConquerSpecialized services

BlobstoreImages

Mail XMPP Task Queue

Memcache Datastore URL Fetch

User Service

Distributed Meme: Sharding

Single Sharded

Counter CounterCounter Counter

Single Sharded

Counter CounterCounter CounterCounter

Single Sharded

Counter CounterCounter CounterCounter CounterCounter

Language runtimes

JVM languages

-Scala-JRuby (Ruby)-Groovy-Quercus (PHP)-Rhino (JavaScript)-Jython (Python)

Ensuring portability

Complete Java development stack

Google Plugin for Eclipse

Google Web Toolkit (GWT)

Firefox

SafariChrome

GWT Debugging in the browser

Quotes to remember

“The fastest code is thatwhich does not run.”

Joel WebberGWT co-creator

Google's scalable serving architecture

Google Apps + your apps

Your custom applicationsOur Google Apps

Google Apps integration

http://appid.appspot.com/

http://yourapp.yourdomain.com/

2. Log in to your Google Apps domain as an admin

3. Accept terms and specify web address

4. The new feature is available in your control panel

5. Additional configuration is available for each feature

Google Apps + App Engine

Federate your on-premise data

Secure Data Connector (SDC)

Secure Data Connector

and 50+ more...

Using Secure Data Connector

Installation- Determine access rules- Configure and install SDC

Getting ready to serve- SDC opens SSL tunnel

Serving- User request sent to App Engine- User authenticated- App makes request through tunnel- SDC performs access checks- Results returned

Database export / import

Cron support

- clean-up DB- prepare reports - batch mail

- perform calcs- call mom

<?xml version="1.0" encoding="UTF-8"?><cronentries> <cron> <url>/recache</url> <description>Repopulate the cache every 2 minutes</description> <schedule>every 2 minutes</schedule> </cron> <cron> <url>/weeklyreport</url> <description>Mail out a weekly report</description> <schedule>every monday of month 08:30</schedule> <timezone>America/New_York</timezone> </cron></cronentries>

("every"|ordinal) (days) "of" (monthspec) (time)

every 5 minutesevery 12 hours2nd,third mon,wed,thu of march 17:00every monday of month 09:001st monday of sep,oct,nov 17:00

Cron support

Distributed Meme: Divide & ConquerTask Queues

Brett SlatkinMay, 2009

Offline Processing onApp Engine: A Look Ahead

Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Task 7

Tasks Queues - Motivation

• Why do background processing? oDo work continuously without user requestso Incrementally process data, compute resultsoSmooth out load patterns, lower user latency

• Simple idea in general:1.Describe the work you want to do now2.Save the description somewhere3.Have something else execute the work later

• Work executed in the order received (best-effort FIFO)• If execution fails, work will be retried until successful

Task Queue Benefits

• BenefitsoAsynchronous

Why do work now when we can do it later?o Low-latency (for users)

Tasks are light-weight; ~3x faster than DatastoreoReliable

Once written, a task will eventually complete oScalable

Storage of new tasks has no contentionParallelizable with multiple workers

• Many features can extend this basic concept

How do traditional task queues work?

How do traditional task queues work? (2)

• Polling has problems:oWorker sits in a loop polling the front of the queue

Not event driven; wasted work oWorkers stay resident when there's no work to do

Wastes machine resourcesoFixed number of workers

Admins must manually add more workers to keep up or queue will grow without bounds

• Limited optimization possible

oMany systems fake a polling interface with something event-driven under the hood

o Long-lived, hanging connections

How does our Task Queue API work? (2)

How does our Task Queue API work? (3)

Your application's health

App Engine's health history

creative commons licensed photograph from cote

Scalable serving architecture

Distributed frontends

-Edge Caching

-Edge Caching-Priority routing

-Edge Caching-Priority routing-Geographically close

connection endpoint91

Life of a request

UserApp

EngineFrontend

App'sRuntime

Life of a request

UserApp

EngineFrontend

App'sRuntime

Scales dynamically / automatically

UserApp

EngineFrontend

App'sRuntime

UserUser

Scales dynamically / automatically

App'sRuntime

UserApp

EngineFrontend

App'sRuntime

UserUser

Multiple versions

UserApp

EngineFrontend

Version 1.1

Version 2.1

Datastore

http://2.1.fredsa.appspot.com/

http://fredsa.appspot.com/

http://labs.google.com/papers/bigtable.html97

Bigtable :A distributed, sharded, sorted array

Row key Row data

Shard 1

Shard 2 . .

Shard n

Bigtable :A distributed, sharded, sorted array

Row key Row data

Datastore design

-Distributed-Bigtable + entity groups-ACID transactions-Optimistic concurrency-Entities + indexes-Protobuf encoded entities

Datastore properties

-Core value types-List properties-Text & binary blobs-Reference

What's different?

Datastore - what's new

-Distributed-Scales to 'internet scale'-No deadlocks-Predictable query performance

Datastore - what's different

-No inner/outer/natural joins-Dense index scans-Per entity metadata-Soft schema-No more DDL

Datastore - what's familiar

- Indexes-ACID transactions-GQLSELECT * FROM StoryWHERE title = 'App Engine Launch'AND author = :current_userAND rating >= 10ORDER BY rating, created DESC

The new ACID

AssociativeCommutativeIdempotentDistributed

AtomicConsistentIsolatedDurable

Entities

Entity groups

Amortizing costs

Customer Order

Count(COUNT)

SUM(Order Amount)Transaction

CreateUpdateDelete

An evolving platform

Apr 2008 Python launchMay 2008 Memcache, Images APIJul 2008 Logs exportAug 2008 Batch write/deleteOct 2008 HTTPS supportDec 2008 Status dashboard, quota detailsFeb 2009 Billing, larger filesApr 2009 Java launch, DB import, cron support, SDCMay 2009 Key-only queriesJun 2009 Task queuesAug 2009 Kindless queriesSep 2009 XMPPOct 2009 Incoming emailDec 2009 BlobstoreFeb 2010 Datastore cursors, Async Urlfetch

23 months in review

- Support for mapping operations across datasets

- Alerting system for exceptions in your application

- Datastore dump and restore facility112

App Engine Roadmap

Wrap up

Always free to get started

~5M pageviews/month• 6.5 CPU hrs/day• 1 GB storage• 650K URL Fetch calls/day• 2,000 recipients emailed• 1 GB/day bandwidth• 100,000 tasks enqueued• 650K XMPP messages/day

Purchase additional resources *

* free monthly quota of ~5 million page views still in full effect115

Thank you

Read morehttp://code.google.com/appengine/

Contact infoPatrick ChanezonDeveloper Advocatechanezon@google.comhttp://twitter.com/chanezon

Questions?

ThanksTo Alon Levi, Fred Sauer, Brett Slatkin and others for their slides

google app engine - devfest india 2010

Technology

google devfest 2009 argentina - intro to appengine

google devfest mn - windows azure notification hubs

building robots using android and arduino google devfest

google devfest 2009 argentina - keynote

app engineday 1-devfest yde 2013

google devfest 2009 argentina - google and the social web

google cloud platform. google app engine

% steps google app

developing touchactive with gae and gwt - google devfest...

google devfest 2012 presentation

implemetning google analytics - 2011-09-24 google devfest...

proyecto - google app

ruby on google app engine: upgrade to google app "turbo"...

google devfest 2009 argentina - building google wave robots

google app inventor

google cloud challenge - php - devfest gdg-cairo

presentación android tv - google devfest lleida

google cloud for developers - devfest manila

devfest chiang mai - implementing google analytics -...