google app engine memcache ae-09-session
TRANSCRIPT
Google App Engine MemCache
ae-09-session
www.appenginelearn.com
Storage in App Engine•There are two ways to store information in the App
Engine
•The datastore is permanent storage - it is slower - it is stored permanently and backed up - when you put data in the datastore it is there until you delete it
•The memcache is temporary storage - it is much faster because it uses the memory of the application servers and does not need to store data on any disks
What is a Cache?
http://en.wikipedia.org/wiki/Cache
“In computer science, a cache is a collection of data duplicating original values stored elsewhere or computed
earlier, where the original data is expensive to fetch (owing to longer
access time) or to compute, compared to the cost of reading the cache. In other words, a cache is a temporary
storage area where frequently accessed data can be stored for rapid
access.”
From the French cacher, meaning "to
hide"
Comparison• Memcache
• Fast
• Uses memory
• Limited size
• Not backed up
• May go away if you run out of memory
• Datastore
• Slower
• Uses disk
• Unlimited size
• Backed up
• Does not go away until you delete it
ApplicationInstances
DDaattaa
SSttoorree
....
MMeemmoorryy
CCaacchhee
The memory cache uses memory on the Google Servers your
application is running on. When
data is added to the cache on one
application instance, it is immediately
transmitted to the other caches so that all instances have
the same view of the memory cache.
Why use Memcache?• It is *fast* -- making it possible for your
application to scale to large numbers of users
•Usually we use it to hold information that was costly to retrieve or compute but does not change a lot
•So we retrieve or compute it once and keep it in memcache for a few minutes or so and use the local copy for a while
•After a while we throw out the local copy of the data and retrieve or compute it again
What is Memcache?
• It is a giant shared dictionary-like thing
•Every element in the cache has a key
•The values can be strings, dictionaries, lists, or other Python objects.
•You can create, read, update, and delete entries in the memory cache.
Memcache Features
•set(key, value, time=0)
•get(key)
•replace(key, value, time=0)
•delete(key)
http://code.google.com/appengine/docs/memcache/clientclass.html
The time is in seconds. The value can be any Python
object. The key should be a
string.
from google.appengine.api import memcache
x = memcache.get("1234")if x is None: print "Nothing found in key 1234"else: print "Found key 1234" print x
x = memcache.get("7890")if x is None: print "Nothing found in key 7890"else: print "Found key 7890" print x
y = { 'a': 'hello', 'b': 'world' }memcache.add("7890", y, 3600)
x = memcache.get("7890")if x is None: print "Nothing found in key 7890"else: print "Found key 7890" print x
z = { 'l': 'more', 'n': 'stuff' }memcache.replace("7890", y, 3600)print "7890 replaced"
Nothing found in key 1234Nothing found in key 7890Found key 7890{'a': 'hello', 'b': 'world'}7890 replaced
Nothing found in key 1234Found key 7890{'a': 'hello', 'b': 'world'}Found key 7890{'a': 'hello', 'b': 'world'}7890 replaced
from google.appengine.api import memcache
x = memcache.get("7890")if x is None: print "Nothing found in key 7890"else: print "Found key 7890" print x
First run Second run
Developer Console
•The Application Engine provides a developer console in each application which allows you to look through the memcache or data store, and execute code on the application engine
•http://localhost:8080/_ah/admin
• If this is running on the Google Cloud - your Google Account must be marked as an “application admin”
You can view the memory cache and look up elements by key.
You get a sense of how large the cache is and how much it is being used. You can even add new data into the
cache.
Using Memcache to Store Sessions
Understanding sessions.pyae-09-session
Memcache for Sessions :)
•Memcache is fast
•Memcache is good at storing small things
•We use the session on nearly every request (i.e. is the user logged in?)
•Sessions don’t need to last forever
•Seems almost perfect ...
Memcache for Sessions :(
•The rule that the memcache data can go away when the servers start to run out of memory is a non-starter
•Users would see themselves logged out randomly
•The servers would start to thrash as sessions were lost and immediately recreated
• It works for small, simple, development applications
Real Session Code
•Real session code uses a combination of memcache and the datastore to allow your application to scale to higher levels without thrashing
•http://code.google.com/p/gaeutilities/
Understanding - sessions.py
•Since the Google Application Engine does not provide a session capability, we need to add one - extending out application
•Download from
•http://www.appenginelearn.com/downloads/util.zip
•Install in your application in the directory util to make it available in your application
Using the Session Library
from util.sessions import Session
class LogoutHandler(webapp.RequestHandler): def get(self): self.session = Session() self.session.delete('username') doRender(self, 'index.htm')
Find the file sessions.py in the
folder util and import the class
named Session from that file.
def post(self): self.session = Session() .... if pw == "secret": self.session['username'] = acct doRender(self,"index.htm",{ } ) else: self.session.delete('username') doRender(self,"login.htm",{'error' : 'Incorrect password'} )
In LoginHandler()
Most of the work is done in the
constructor of the Session object.
Once we have retrieved the Session() we just use it like a dictionary.
Overall Flow
•Look for a session id cookie on the incoming HTTP request
• If there is a cookie set, retrieve the session
• If there is no cookie on incoming request or no matching session in the memcache, make a new session, put it in memcache, and set the cookie
class Session(object): def __init__(self): self.sid = None self.key = None self.session = None string_cookie = os.environ.get('HTTP_COOKIE', '') self.cookie = Cookie.SimpleCookie() self.cookie.load(string_cookie)
# Check for existing Session if self.cookie.get(COOKIE_NAME): self.sid = self.cookie[COOKIE_NAME].value self.key = "session-" + self.sid self.session = memcache.get(self.key) if self.session is None: logging.info("Invalidating session "+self.sid) self.sid = None self.key = None
Retrieve the cookie from the
request.
Look up session in memcache
using the cookie. If we don’e find a session make a
new one.
# Make a new session and set the cookie if self.session is None: self.sid = str(random.random())[5:]+str(random.random())[5:] self.key = "session-" + self.sid logging.info("Creating session "+self.key); self.session = dict() memcache.add(self.key, self.session, 3600)
self.cookie[COOKIE_NAME] = self.sid self.cookie[COOKIE_NAME]['path'] = DEFAULT_COOKIE_PATH # Send the Cookie header to the browser print self.cookie
If we get this far, and we still don’t have a session, make one. Make a big random string, an empty dictionary for the session data and put it into the memory cache using the session ID as
the key. Then set the cookie in the browser.
Cookie Header• In the HTTP
Request/Response Cycle, Cookies are set in the headers of the response before the response data.
•So we must establish the session before any data is sent to the browser.
HTTP/1.1 200 OKContent-type: text/htmlSet-Cookie: sessid=123
<head> .. </head><body><h1>Welcome ....
Set-Cookie: appengine-simple-session-sid=921288672590409739; Path=/
A Little Test
•Start with a fresh browser, and either start with a fresh application of clear the memcache using the Developer Console
•Watch the application log
•Start your browser and go to http://localhost:8080
•A session will be started
We are not logged in but we now have a cookie and a session for the
user. The session exists but is *empty*.
We can use the developers console
to look up data stored in the
memory cache. We look up the session using the key which was in the log and there is indeed a
dictionary which is empty in the
memory cache.“Pickled” means “converted into a format suitable
for sending across a network”. a.k.a “Serialization”
If we then log in and again check the session
key in them memory cache, we see that the username is now set in the session indicating that we have logged in
successfully.
More in sessions.py
•Most of the work is in the constructor __init__()
•The rest of the methods in the class are to make the class function like a dictionary
# Private method to update the cache on modification def _update_cache(self): memcache.replace(self.key, self.session, 3600)
# del session[keyname] def __delitem__(self, keyname): if keyname in self.session: del self.session[keyname] self._update_cache() return raise KeyError(str(keyname))
# x = session[keyname] def __getitem__(self, keyname): if keyname in self.session: return self.session[keyname] raise KeyError(str(keyname))
Python transforms certain language elements like
session[keyname]into
__getitem__(session,keyname)
Summary•Memcache is an easy-to-use high performance
memory-bases storage area that is well-suited to store small amounts of data with short-to-medium life spans
•Wise use of memcache really helps in the scaling of your application
•We implement a simple Session() storage implementation using memcache as our backing storage as a developer exercise.