jozef ve providing scalability for pirates, lizards and zombies at #doxlon
DESCRIPTION
Video: https://www.youtube.com/watch?v=OGjaXtauO-c Jozef will talk about building a new generation of distributed services framework called Amanda at MPC. MPC is a leading visual FX company that works with extremely high volumes of data and a vast array of computing power. This talk gives a behind the scenes (sorry) look at how art assets get processed at scale. And like all good talks it includes plenty of metrics!TRANSCRIPT
Providing scalability for Pirates, Lizards and Zombies.
August 2014
Amanda&Pla)orm&for&distributed&services
Who am I
Software developer at MPC !
Python since 2009 !
Loves services !
Obsessed by monitoring !
@JozefVanE on twitter
Who is MPC
We create Visual Effects !
»Advertising »Feature films
We create Visual Effects !
»Across 8 sites »Fully integrated cross
site pipeline
What are visual effects?
Visual effects involve the integration of live-action footage and generated imagery to create environments which look realistic, but would be dangerous, expensive, impractical, or simply impossible to capture on film.
(source wikipedia)
What are visual effects?
What are visual effects?
What are visual effects?
What are visual effects?
Production workflow.
Production considerations» Management » Assets, Production, Users
» Various sources » Used by: » In house tools » 3rd party applications »Multiple environments
» Users » Artists » Developers
MPC’s approach - Amanda» Service based architecture » Platform as a service » Multi protocol, transport and concurrency » Ecosystem to write a service for developers of any level » 1st generation 2007 » 2nd generation 2012-2013
!
» Godzilla » 450 years if rendered on a single machine
Some statistics.
!
» Godzilla » 450 years if rendered on a single machine » Or, 15 years after Henry 8th died
Some statistics.
!
» Godzilla » 650 TB of data
Some statistics.
!
» Godzilla » 650 TB of data » Or, 41000 mobile phones with 16GB drives.
Some statistics.
!
» Godzilla » 250.000 Amanda req/min.
Some statistics.
!
» Godzilla » 250.000 Amanda req/min. » Or, 4 times Germany VS Brazil tweets
Some statistics.
Amanda - building the platform
Amanda - building the platform
Amanda - building the platform
Services
class MakeMovieService(object):!!
@public! def greet_director(self, director_name):! return "hello %s" % (director_name,)!
Amanda - building the platform
We have a service, but lets make it do something useful
Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)!
Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)! def _get_shot(self, shot):! """ Get shot data! """! query = "<insert db query>"! return self._db.execute(query, (shot,))!!
Amanda - building the platformInfrastructuresclass MakeMovieService(object):! storage = Dependency(“storage")! def _check_data_on_disk(self, files):! """ Check data is available! """! return self.storage.check_exists(files)! def _get_shot(self, shot):! """ Get shot data! """! query = "<insert db query>"! return self._db.execute(query, (shot,))!! @public! def make_movie_magic(self, shotInfo):! """ Do some magic! """! shot = self._get_shot(shotInfo)! if not self._check_data_on_disk(shot["files"]):! self._log.error(‘files not found')! raise DataNotAvailable()! self._do_magic(shot["files"], self._config.production.jobsite())
Amanda - building the platform
Ok now it is doing something but that is one user, one service and not very useful at scale…
Amanda - building the platform
Amanda - building the platform
Service providers
services = ServiceProvider(! make_movies=MakeMovieService,! storage=StorageService! config=ConfigService! logging=LoggingService! __infrastructure__={! "_db": DbInfrastructure,! "_config": ConfigInfrastructure,! "_xmesh": XmeshInfrastructure,! "_log": LoggingInfrastructure! }!)!services.make_movies.make_movie_magic("shot1")!services.logging.change_logging_level("make_movies", "debug")!
Amanda - building the platform
Amanda - building the platform
Proxies
class Proxy(object):!!
def __getattr__(self, method):! def method_stand_in(*args, **kwargs):! return self._transport.publish(self._service, method, *args, **kwargs)! return method_stand_in!!
services = ServiceProvider(! make_movies=Proxy("make_movies", transport)!)!services.make_movies.do_movie_magic(“shot1”)!!
Amanda - building the platform
Amanda - building the platform
Ok, at this point we can scale a bit but what about concurrency/parallel processing?
Amanda - building the platform
Amanda - building the platform
Amanda - building the platform
Amanda - building the platform
This gives us all the building blocks to assemble a distributed system with the possibility to use the
best technologies for the given tasks
Amanda - assembling & maintaining the platform
Amanda - assembling & maintaining the platform
Setting up for a cross language pipeline.
Amanda - assembling & maintaining the platform
Amanda - assembling & maintaining the platform
Amanda - assembling & maintaining the platform
Setting up for a cross language, fault tolerant
pipeline.
Amanda - assembling & maintaining the platform
Setting up for cross language, fault tolerant,
monitorable pipeline.
Amanda - assembling & maintaining the platform
request: userAdmin, getUserIdByUserName, [u'jozef'], {} request id: 1393351037.307669-userAdmin-staffdirectory-3500, by jozef from staffdirectory exception: <type 'NoneType'>, None handled by: amanda-host-1_28368-GeventConcurrencyPool_worker_37 stats: sent on Tue Feb 25 17:57:17 2014 received on Tue Feb 25 17:57:17 2014 frontend->backend delivery took 0.003s` internal queuing took 0.0000015s request execution took 0.005s total time: 0.011s
Amanda - assembling & maintaining the platforminternal queuing: 0.0000015s
total time: 0.011s
frontend->backend: 0.003s
request execution: 0.005s
Amanda - assembling & maintaining the platform» Maintenance. » Salt » Extended with an Amanda module. » Salt client wrapped inside a service so we can access it from anywhere not
just the master node. » Predefined, repeatable configuration.
Amanda» Adaptable » Extendable » Configurable » Best of breeds
Amanda» Looking ahead » Containerisation » Auto scaling » Open sourcing possibilities
In Production
Montréal
We’re Now Hiring! !
Opportunities across all our Studios and across all disciplines !
Check out www.moving-picture.com for vacancies !
Or email [email protected]
Amanda
Thank you, any questions?
Jozef van Eenbergen - twitter: @JozefVanE