jozef ve providing scalability for pirates, lizards and zombies at #doxlon

Providing scalability for Pirates, Lizards and Zombies.

August 2014

Amanda&Pla)orm&for&distributed&services

Who am I

Software developer at MPC !

Python since 2009 !

Loves services !

Obsessed by monitoring !

@JozefVanE on twitter

Who is MPC

We create Visual Effects !

»Advertising »Feature films

We create Visual Effects !

»Across 8 sites »Fully integrated cross

site pipeline

What are visual effects?

Visual effects involve the integration of live-action footage and generated imagery to create environments which look realistic, but would be dangerous, expensive, impractical, or simply impossible to capture on film.

(source wikipedia)

What are visual effects?

What are visual effects?

What are visual effects?

What are visual effects?

Production workflow.

Production considerations» Management » Assets, Production, Users

» Various sources » Used by: » In house tools » 3rd party applications »Multiple environments

» Users » Artists » Developers

MPC’s approach - Amanda» Service based architecture » Platform as a service » Multi protocol, transport and concurrency » Ecosystem to write a service for developers of any level » 1st generation 2007 » 2nd generation 2012-2013

!

» Godzilla » 450 years if rendered on a single machine

Some statistics.

!

» Godzilla » 450 years if rendered on a single machine » Or, 15 years after Henry 8th died

Some statistics.

!

» Godzilla » 650 TB of data

Some statistics.

!

» Godzilla » 650 TB of data » Or, 41000 mobile phones with 16GB drives.

Some statistics.

!

» Godzilla » 250.000 Amanda req/min.

Some statistics.

!

» Godzilla » 250.000 Amanda req/min. » Or, 4 times Germany VS Brazil tweets

Some statistics.

Amanda - building the platform

Amanda - building the platform

Amanda - building the platform

Services

class MakeMovieService(object):!!

@public! def greet_director(self, director_name):! return "hello %s" % (director_name,)!

Amanda - building the platform

We have a service, but lets make it do something useful

Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)!

Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)! def _get_shot(self, shot):! """ Get shot data! """! query = "<insert db query>"! return self._db.execute(query, (shot,))!!

Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)! def _get_shot(self, shot):! """ Get shot data! """! query = "<insert db query>"! return self._db.execute(query, (shot,))!! @public! def make_movie_magic(self, shotInfo):! """ Do some magic! """! shot = self._get_shot(shotInfo)! if not self._check_data_on_disk(shot["files"]):! self._log.error(‘files not found')! raise DataNotAvailable()! self._do_magic(shot["files"], self._config.production.jobsite())

Amanda - building the platform

Ok now it is doing something but that is one user, one service and not very useful at scale…

Amanda - building the platform

Amanda - building the platform

Service providers

services = ServiceProvider(! make_movies=MakeMovieService,! storage=StorageService! config=ConfigService! logging=LoggingService! __infrastructure__={! "_db": DbInfrastructure,! "_config": ConfigInfrastructure,! "_xmesh": XmeshInfrastructure,! "_log": LoggingInfrastructure! }!)!services.make_movies.make_movie_magic("shot1")!services.logging.change_logging_level("make_movies", "debug")!

Amanda - building the platform

Amanda - building the platform

Proxies

class Proxy(object):!!

def __getattr__(self, method):! def method_stand_in(*args, **kwargs):! return self._transport.publish(self._service, method, *args, **kwargs)! return method_stand_in!!

services = ServiceProvider(! make_movies=Proxy("make_movies", transport)!)!services.make_movies.do_movie_magic(“shot1”)!!

Amanda - building the platform

Amanda - building the platform

Ok, at this point we can scale a bit but what about concurrency/parallel processing?

Amanda - building the platform

Amanda - building the platform

Amanda - building the platform

Amanda - building the platform

This gives us all the building blocks to assemble a distributed system with the possibility to use the

best technologies for the given tasks

Amanda - assembling & maintaining the platform

Amanda - assembling & maintaining the platform

Setting up for a cross language pipeline.

Amanda - assembling & maintaining the platform

Amanda - assembling & maintaining the platform

Amanda - assembling & maintaining the platform

Setting up for a cross language, fault tolerant

pipeline.

Amanda - assembling & maintaining the platform

Setting up for cross language, fault tolerant,

monitorable pipeline.

Amanda - assembling & maintaining the platform

request: userAdmin, getUserIdByUserName, [u'jozef'], {} request id: 1393351037.307669-userAdmin-staffdirectory-3500, by jozef from staffdirectory exception: <type 'NoneType'>, None handled by: amanda-host-1_28368-GeventConcurrencyPool_worker_37 stats: sent on Tue Feb 25 17:57:17 2014 received on Tue Feb 25 17:57:17 2014 frontend->backend delivery took 0.003s` internal queuing took 0.0000015s request execution took 0.005s total time: 0.011s

Amanda - assembling & maintaining the platforminternal queuing: 0.0000015s

total time: 0.011s

frontend->backend: 0.003s

request execution: 0.005s

Amanda - assembling & maintaining the platform» Maintenance. » Salt » Extended with an Amanda module. » Salt client wrapped inside a service so we can access it from anywhere not

just the master node. » Predefined, repeatable configuration.

Amanda» Adaptable » Extendable » Configurable » Best of breeds

Amanda» Looking ahead » Containerisation » Auto scaling » Open sourcing possibilities

In Production

Montréal

We’re Now Hiring! !

Opportunities across all our Studios and across all disciplines !

Check out www.moving-picture.com for vacancies !

Or email recruitment@moving-picture.com

Amanda

Thank you, any questions?

Jozef van Eenbergen - twitter: @JozefVanE