google app engine - devfest india 2010

Google Confidential and Proprietary 1

Google App Engine

Feb – March, 2010Patrick ChanezonDeveloper AdvocateGoogle Developer [email protected]

Monday, March 8, 2010

mailto:[email protected]


Agenda

-App Engine introduction-Why build it?-App Engine tour-What's different?-Wrap up-Questions

2


Isha Upanishad

3

4. The spirit is unmoving, one, swifter than the mind. The senses do not reach It as It is ever ahead of them. Though Itself standing still It outstrips those who run. In It the all-pervading air supports the activities of beings


Vayu computing?

4

Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg


http://commons.wikimedia.org/wiki/File:Vayu.jpg

http://commons.wikimedia.org/wiki/File:Vayu.jpg

Vayu Computing?

5

7−10 Then they said to Vayu (Air): "O Vayu! Find out who this great Spirit is." "Yes," he said, and hastened to It. Brahman asked him: "Who are you?" He replied "I am known as Vayu; I am also called Matarisva." Brahman said: "What power is in you, who are so well known?" Vayu replied: "I can carry off all−whatever there is on earth." Brahman put a straw before him and said: "Carry this." He rushed toward it with all his ardour but could not move it. Then he returned from the Spirit and said to the gods: "I could not find out who this Spirit is,"

Kena Upanishad


PlacePostage H

ere

6

IaaSPaaSSaaS

What is cloud computing?


IaaS value proposition…

7


Google App Engine

8

“We wear pagers so you don’t have to”

pager


Google App Engine

-Easy to build-Easy to maintain-Easy to scale

9


By the numbers

- Built 100K apps- Maintained by 250K developers- Scaled to 250M pageviews daily

10

semi-transparent collage of apps


gigya Socialize

11


gigya Socialize - traffic

12


Gigya Socialize - traffic

13


App Engine

14


App Engine

15


Time.com

16


Socialwok

17


Social networking at scale

18


Social networking at scale

18

>40M Users


PubSubHubbubhttp://code.google.com/p/pubsubhubbub/

19

The future is distributed: There will be big hubs, many small hubs, and tons of publishers and subscribers.

Publishers, subscribers, and hubs may play multiple roles.


http://code.google.com/p/pubsubhubbub/

http://code.google.com/p/pubsubhubbub/

Hell's Kitchen on facebook

20


Handling peak traffic: mental_floss

21


Elastic / redundant capacity

22


Short term events: earthhour.org

23


http://www.allforgood.org/

24


http://www.allforgood.org

http://www.allforgood.org

http://www.snapabug.com/

25


http://www.snapabug.com

http://www.snapabug.com

エコポイント

26


Japan: Q&A for our Future

27


http://www.google.com/tipjar

28




Chillingo CrystalGaming meets Social

Cogs

Guerilla Bob

Zombie Dash Angry Birds LITE underground Meltdown

Mission Deep Sea Speed ForgeExtreme

Ravensword:The Fallen King

Angry Birds


Why build it?

30

Wrench


It's just too difficult

31


Hosting means hidden costs

• Idle capacity• Software patches & upgrades• License fees• Lots of maintenance• Traffic & utilization forecasting• Upgrades

32


Cloud development in a box

33

• SDK & “The Cloud”• Hardware• Networking• Operating system• Application runtime

o Java, Python• Static file serving• Services• Fault tolerance• Load balancing


Distributed Cloud ComputingReliability and scalability


Know your Distributed Memes

-Divide & conquer -Sharding-Tolerate / expect failure- Idempotency-Just scale it up

35


Rocket scienceMonolithic Computing

36


Rocket scienceMonolithic Computing

36

Hot spare“Just in case”


Distributed Meme:Expect, tolerate and design for failure

37

MTFB: O(Years)



37

MTFB: O(Months)

MTFB: O(Years)



37

MTFB: O(Months)

MTFB: O(Years)

MTFB: O(Minutes/Seconds)


Monolithic computing

38

stop watchsmall

stop watchlarge


Distributed web hosting platform

39



39

X



39

XRetry


Vertical / horizontal scalability

40


Vertical scalability

41


Vertical scalability

41

Horizontal scalability


Tolerate, plan for and expect small failures


Easy to deploy & scale

43

http://www.yourdomain.com/

1

2


http://yourapp.bestbuy.com




Thinking about scalability

....the tools, platform and design don't matter too much

Just a few users....

44


Thinking about scalability

....you must design for scalability

Lots and lots of users...

45


But the problem is...

When?

http://digg.com/http://slashdot.org/

46


The real problem is...

When it happens.

Can you afford to puteverything else on hold?

47


Google App Engine

Leveraging Google'splatform to better serveyour customers

48



49



50

-Great for web apps• Request based, data backed

-Parallel processing-Scales automatically-Available globally-Configuration free-Built-in DoS protections


Distributed datastore

51



52



52

-Arbitrary horizontal scaling



52

-Arbitrary horizontal scaling-Parallel processing



52

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'



52

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance



52

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance-No deadlocks



52

-Arbitrary horizontal scaling-Parallel processing-Scales to 'Internet scale'-Predictable query performance-No deadlocks-No global schema


Distributed memcache

53

Memcache


Distributed Meme: Divide & ConquerSpecialized services

54

BlobstoreImages

Mail XMPP Task Queue

Memcache Datastore URL Fetch

User Service


Distributed Meme: Sharding

Single Sharded

55

Counter CounterCounter Counter



Single Sharded

55

Counter CounterCounter CounterCounter



Single Sharded

55

Counter CounterCounter CounterCounter CounterCounter


Language runtimes

Duke, the Java mascotCopyright © Sun Microsystems Inc., all rights reserved.

56


JVM languages

-Scala-JRuby (Ruby)-Groovy-Quercus (PHP)-Rhino (JavaScript)-Jython (Python)

57


Ensuring portability

58


Complete Java development stack

59


Google Plugin for Eclipse

60


Google Plugin for Eclipse

61


Google Web Toolkit (GWT)

62


Google Web Toolkit (GWT)

IE

Firefox

SafariChrome

Opera

63


GWT Debugging in the browser

64


Quotes to remember

“The fastest code is thatwhich does not run.”

Joel WebberGWT co-creator

65


Google's scalable serving architecture

Google Apps + your apps

Your custom applicationsOur Google Apps

66


Google Apps integration

http://appid.appspot.com/

http://yourapp.yourdomain.com/

67


http://appid.appspot.com

http://appid.appspot.com

http://yourapp.yourdomain.com

http://yourapp.yourdomain.com

2. Log in to your Google Apps domain as an admin

68


3. Accept terms and specify web address

69


4. The new feature is available in your control panel

70


5. Additional configuration is available for each feature

71


Google Apps + App Engine

72


Federate your on-premise data

73


Secure Data Connector (SDC)

74


Secure Data Connector

75


Secure Data Connector

and 50+ more...

76


Using Secure Data Connector

77



Installation- Determine access rules- Configure and install SDC

77




Getting ready to serve- SDC opens SSL tunnel

77




Getting ready to serve- SDC opens SSL tunnel

Serving- User request sent to App Engine- User authenticated- App makes request through tunnel- SDC performs access checks- Results returned

77


Database export / import

78


Cron support

- clean-up DB- prepare reports - batch mail

- perform calcs- call mom

79


<?xml version="1.0" encoding="UTF-8"?><cronentries> <cron> <url>/recache</url> <description>Repopulate the cache every 2 minutes</description> <schedule>every 2 minutes</schedule> </cron> <cron> <url>/weeklyreport</url> <description>Mail out a weekly report</description> <schedule>every monday of month 08:30</schedule> <timezone>America/New_York</timezone> </cron></cronentries>

("every"|ordinal) (days) "of" (monthspec) (time)

every 5 minutesevery 12 hours2nd,third mon,wed,thu of march 17:00every monday of month 09:001st monday of sep,oct,nov 17:00

Cron support

80


Distributed Meme: Divide & ConquerTask Queues

81

Brett SlatkinMay, 2009

Offline Processing onApp Engine: A Look Ahead

Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Task 7


Tasks Queues - Motivation

• Why do background processing? oDo work continuously without user requestso Incrementally process data, compute resultsoSmooth out load patterns, lower user latency

• Simple idea in general:1.Describe the work you want to do now2.Save the description somewhere3.Have something else execute the work later

• Work executed in the order received (best-effort FIFO)• If execution fails, work will be retried until successful



Task Queue Benefits

• BenefitsoAsynchronous

Why do work now when we can do it later?o Low-latency (for users)

Tasks are light-weight; ~3x faster than DatastoreoReliable

Once written, a task will eventually complete oScalable

Storage of new tasks has no contentionParallelizable with multiple workers

• Many features can extend this basic concept



How do traditional task queues work?


How do traditional task queues work? (2)

• Polling has problems:oWorker sits in a loop polling the front of the queue

Not event driven; wasted work oWorkers stay resident when there's no work to do

Wastes machine resourcesoFixed number of workers

Admins must manually add more workers to keep up or queue will grow without bounds

• Limited optimization possible

oMany systems fake a polling interface with something event-driven under the hood

o Long-lived, hanging connections


How does our Task Queue API work? (2)


How does our Task Queue API work? (3)


Your application's health

88


App Engine's health history

89


creative commons licensed photograph from cote

Scalable serving architecture

90


Distributed frontends

91



-Edge Caching

91



-Edge Caching-Priority routing

91



-Edge Caching-Priority routing-Geographically close

connection endpoint91


Life of a request

UserApp

EngineFrontend

App'sRuntime

92


Life of a request

UserApp

EngineFrontend

App'sRuntime

93


Scales dynamically / automatically

UserApp

EngineFrontend

App'sRuntime

User

UserUser

User

94


Scales dynamically / automatically

App'sRuntime

App'sRuntime

UserApp

EngineFrontend

App'sRuntime

User

UserUser

User

95


Multiple versions

UserApp

EngineFrontend

Version 1.1

Version 2.1

Datastore

http://2.1.fredsa.appspot.com/

http://fredsa.appspot.com/

96



http://labs.google.com/papers/bigtable.html97


Bigtable :A distributed, sharded, sorted array

Row key Row data

98


Shard 1

Shard 2 . .

Shard n

Bigtable :A distributed, sharded, sorted array

Row key Row data

99


Datastore design

-Distributed-Bigtable + entity groups-ACID transactions-Optimistic concurrency-Entities + indexes-Protobuf encoded entities

100


Datastore properties

-Core value types-List properties-Text & binary blobs-Reference

101


What's different?

102


Datastore - what's new

-Distributed-Scales to 'internet scale'-No deadlocks-Predictable query performance

103


Datastore - what's different

-No inner/outer/natural joins-Dense index scans-Per entity metadata-Soft schema-No more DDL

104


Datastore - what's familiar

- Indexes-ACID transactions-GQLSELECT * FROM StoryWHERE title = 'App Engine Launch'AND author = :current_userAND rating >= 10ORDER BY rating, created DESC

105


The new ACID

AssociativeCommutativeIdempotentDistributed

AtomicConsistentIsolatedDurable

106


Entities

107


Entity groups

108


Amortizing costs

Customer Order

Count(COUNT)

SUM(Order Amount)Transaction

CreateUpdateDelete

109


An evolving platform

110


Apr 2008 Python launchMay 2008 Memcache, Images APIJul 2008 Logs exportAug 2008 Batch write/deleteOct 2008 HTTPS supportDec 2008 Status dashboard, quota detailsFeb 2009 Billing, larger filesApr 2009 Java launch, DB import, cron support, SDCMay 2009 Key-only queriesJun 2009 Task queuesAug 2009 Kindless queriesSep 2009 XMPPOct 2009 Incoming emailDec 2009 BlobstoreFeb 2010 Datastore cursors, Async Urlfetch

23 months in review

111


- Support for mapping operations across datasets

- Alerting system for exceptions in your application

- Datastore dump and restore facility112

App Engine Roadmap


Wrap up

113


Always free to get started

~5M pageviews/month• 6.5 CPU hrs/day• 1 GB storage• 650K URL Fetch calls/day• 2,000 recipients emailed• 1 GB/day bandwidth• 100,000 tasks enqueued• 650K XMPP messages/day

114


Purchase additional resources *

* free monthly quota of ~5 million page views still in full effect115


Thank you

Read morehttp://code.google.com/appengine/

Contact infoPatrick ChanezonDeveloper [email protected]://twitter.com/chanezon

Questions?

116


http://code.google.com/appengine/

http://code.google.com/appengine/



http://twitter.com/fredsa

http://twitter.com/fredsa

ThanksTo Alon Levi, Fred Sauer, Brett Slatkin and others for their slides


google app engine - devfest india 2010

Technology