advanced python, part 1

© 2014 Zaar Hai tech.zarmory.com

Topics in Advanced PythonNo agenda – jumping in...

http://tech.zarmory.com/

2 © 2014 Zaar Hai tech.zarmory.com

Decorators

Decorator is a function that accepts a function and returns another function

from functools import wrapsdef deco(f): @wraps(f) def wrapper(*args, **kwargs): print "started at %s" % time.time() rv = f(*args, **kwargs) print "finished at %s" % time.time() return wrapper

@decodef sleeper(): "Sleeper function" time.sleep(2)

wraps takes care about wrapped aux data. Without it:>>> print sleeper.__name__wrapper

>>> sleeper()started at 1384245972.01finished at 1384245974.01>>> print sleeper.__name__sleeper

Important when reading online help of someone else's code



Decorator usage examples

To return cached valuesTo implement retriesTo guard methods/functions with a lockTo validate input/output of the functionTo limit function execution by a certain timeout

Decorators greatly help with code reuse andMake code explicitly readable



Decorator is just a function

You can call decorators “inline” in your code

@decodef sleeper(): passsleeper()

deco(sleeper)()

Similarly, decorators themselves can accept parameters

@deco1@deco2def sleeper(): passsleeper()

deco1(deco2((sleeper))()

@deco(5)def sleeper(): passsleeper()

deco(5)(sleeper)()

Note: The invocation order is a bit differentNote:@deco(5) is executed when the code is imported



Decorator with parameter - example

$ cat mydeco.pydef deco(p):

print "got %s" % pdef real_deco(f):

@wraps(f)def wrapper(*args, **kwargs):

print "decorating"return f(*args, **kwargs)

return wrapperreturn real_deco

@deco(5)def hello(): pass

>>> import mydeco # outer function runs during importgot 5>>> mydeco.hello() # decorator runs when function executeddecorating>>>



Decorator can be a class

Some times its useful to implement decorators as a classclass deco(object): def __init__(self, p): self.p = p print "got %s" % p

def __call__(self, f): @wraps(f) def wrapper(*args, **kwargs): print "decorating" return f(*args, **kwargs) return wrapper

__call__ is a special method that is invoked when you try calling an object as if it was a function



How does @property decorator work?

@property – one of the most common decorators in Pythonclass A(object): @property def a(self): return "a"

>>> A().a'a'

But how does it work?



Meet descriptors

Descriptor is a protocol for accessing object attributesclass A(object): def __init__(self): self.a = 1

>>> f = A(); f.a

When you access a, what actually happens isv = self.__dict__["a"]if hasattr(v, '__get__'):

return v.__get__(self)else:

return v

I.e. when attribute defines __get__ (or __set__) methods, they are called to produce (or set) actual attribute value



Back to the @property

@property implements both decorator and descriptor semantics like this:class Property(object): def __init__(self, getter): self.getter = getter

def __get__(self, obj, csl): return self.getter(obj)

class A(object): @Property def a(self): return "a"

>>> o = A(); o.a'a'>>> type(o.a)<type 'str'>



Descriptors in action

One-time caching of method's return values

Reusable – follows DRY principleMore on this here and here

class BigMath(object): def get_big_data(self): if hasattr(self, "_rv"): return self._rv self._rv = self._get_data() return self._rv def _get_data(self): print "working hard...." return 42

class OneTime(object): def __init__(self, func): self.func = func

def __get__(self, obj, cls): to_augment = obj or cls rv = self.func(to_augment) pname = self.func.__name__ setattr(to_augment, pname, rv) return rv

class BigMath(object): @OneTime def big_data(self): print "working hard...." return 42

Nice, but leads to a lot of copy/paste code


http://tech.zarmory.com/2013/08/onetime-evaluation-for-python-object.html

http://tech.zarmory.com/2013/07/delaying-attribute-initialization-for.html


Multiple inheritance“LEGO” goes on



M-I in action - MixIns

“MixIn” is a programming concept about creating aux class that enriches functionality of the main class

class SteeringWheel(object): ...

class HornMixIn(object): """Mix-in class to enable horn functionality""" def horn(self, *args, **kwargs): print "move over!"

class CruiseControlMixIn(object):def cc_set(self): …def cc_cancel(self): ...def cc_restore(self):...

class Mazda2SteeringWheel(CruiseControlMixIn, HornMixIn, SteeringWheel): """Sometimes there is nothing to configure here at all"""



M-I in action – MixIns (continued)

Here is another exampleclass Synchronized(object): def __init__(self, *args, **kwargs): self._sync_lock = threading.Lock() super(Synchronized, self).__init__(*args, **kwargs)

@staticmethod def synchronized(func): @wraps(func) def wrapper(self, *args, **kwargs): with self._sync_lock: return func(self, *args, **kwargs) return wrapper

class Writer(Syncrhonized, Db):@Synchronized.synchronizeddef write(self, o):

self.db.write(o)

As we see, for a lot of real-live examples M-I can be used straight-forward and its behavior in Python “makes sense”



“New style” classes

Not exactly “new” - since Python 2.2Always inherit from objectReally, Always inherit from objectUse super to call ancestor methods

class Foo(object):def bar(self): pass

class Bar(Foo):def bar(self):

return super(Bar, self).bar()

class Foo:def bar(self): pass

class Bar(Foo):def bar(self):

return Foo.bar(self)

Following the above technique will make your and others' life much easier



Multiple inheritance and MRO

What will be output of running C()?“A B C”?“B A C”?

How do we arrive from super(A..) to the method of B?

class A(object): def __init__(self): super(A, self).__init__() print "A"

class B(object): def __init__(self): super(B, self).__init__() print "B"

class C(A,B): def __init__(self): super(C, self).__init__() print "C"



Multiple inheritance and MRO

MRO – method resolution orderPython utilizes C3 algorithm

>>> C.__mro__(<class 'my2.C'>, <class 'my2.A'>, <class 'my2.B'>, <type 'object'>)

And the answer to the previous question is “B A C”For most real-life example I've seen, C3 logic “makes sense”Without inheriting from object and using super, one had to do MRO by himself in the code.

Bottom line – super() does not return ancestor, but the next member of MRO chain


http://www.python.org/download/releases/2.3/mro/


MRO pitfalls

Where super(E, self).__init__() will end up?

class A(object): passclass B(object): passclass C(A,B): passclass D(B,A): passclass E(C,D): pass



MRO pitfalls

Where super(E, self).__init__() will end up?

class A(object): passclass B(object): passclass C(A,B): passclass D(B,A): passclass E(C,D): pass

>>> class E(C,D): pass... Traceback (most recent call last): File "<stdin>", line 1, in <module>TypeError: Error when calling the metaclass bases Cannot create a consistent method resolutionorder (MRO) for bases B, A

There are cases when C3 would (and should) fail to prevent a naive programmer from creating a mess



type function revised

All of you are familiar with type:>>> class A(object): pass>>> a = A()>>> type(a)<class '__main__.A'>

BTW, don't use type(a) == A, but isinstance(a, A)

What type(A) will print?



type function revised

>>> type(A)<type 'type'>

In Python, object model hierarchy has two levels:Objects are instances of their ClassClasses are instances of their Type – AKA metaclass

All classes by default are instances of type metaclass, but we can define our own metaclasses of course:class MetaA(type): pass

class A(object):__metaclass__ = MetaA

>>> type(A)<class '__main__.MetaA'>

Metaclass can augment class creation. More on this later



Yet another face of type function

type can be used to create new classes on the flyThe syntax: type(name, bases, dict)

class Actor(object):def get(self, req):

return self.val

for c in [Volumes, Hosts]:HandlerClass = type('Handler', (Actor, c), {})HttpServer.addUrl(c.url, HandlerClass)

HttpServer.start()

class Conf(object): pass

class Volumes(Conf):url = '/volumes'val = 5

class Hosts(Conf):url = '/hosts'val = 4

Allows really decoupled design. Yet flexible and efficient.



Metaclass example – proper Enum

Magic? May be; but a simple one

class UserRoles(Enum): root = 10 user = 20 user__desc = "Regular users" Nobody = 30

>>> UserRoles.root10>>> UserRoles.root.name'root' # Wow, I have a name!>>> UserRoles.user.desc'Regular users' # And even description >>> UserRoles.root == 10 True # I look and quack just like the native type I was assigned to>>> role = UserRoles(10) # Cast me! (with the DB value for example)>>> role; role.name; role == 1010'root'True>>> role == UserRoles.rootTrue



Magic revealed

class EnumMeta(type): def __new__(mcs, name, base, d): cls = super(EnumMeta, mcs).__new__(mcs, name, base, d) for k, v in cls.__dict__.items():

pname = getattr(cls, k+"__name", k) pdesc = getattr(cls, k+"__desc", "")

n = type(v).__name__ prop = type(n, (type(v),), {"name" : pname, "desc" : pdesc})

p = prop(v) setattr(cls, k, p)

return cls class Enum(object): __metaclass__ = EnumMeta

Full story here


http://tech.zarmory.com/2013/08/python-enum-on-steroids.html


Thank you …


advanced python, part 1

Technology

zaar hai tech

rv self

print sleeper

kwargs print

sleeper function time

p print

decorating return f

time rv