Pop-Up Render Farms

Production in the Clouds and Beyond

Presented by Chris Bond

Pop-Up Render Farms

The logistics of rendering in a VFX environment can be incredibly complex; add

cloud-based cores into the mix and they become even more complex.

Different render applications, simulation tools, assets and dependencies are driven

by unique workflows tailored to the needs of the client; the artist; and the job at

hand.

The key to making cloud rendering successful, is to bring that incredibly scalable

resource to a facility's unique and local pipeline. Learn how studios can seamlessly

merge public cloud resources such as Google, Amazon, Azure, Rackspace and

others to their internal farms, adding thousands of cores as if they were inside the

studio, giving facilities the flexibility to work the way that suits them best.

Pop-Up Render Farms – Overview

Define „Rendering‟

What is and Why use a Render farm Management tool.

Clouds

Decisions you need to make

First, a little bit about me:

Began career in Commercials & Computer Graphics in early „90s

First Feature Film: Swordfish (1999) Bank Explosion (VFX Supervisor)

Production across 3 countries with 4 offices in 2001 at Frantic Films

Over 40 projects, last VFX credit – AVATAR (VFX Supervisor)

Formed Thinkbox Software November 2010

Let’s talk about Rendering

What is rendering?

Defined as (1) : to reproduce or represent by artistic or verbal

means : to depict (2)

In Media & Entertainment [M&E] industries, Rendering is the process

of having a computer do work to generate an image.

Often used to describe simulation, meshing, encoding, conversion

I prefer „Computing‟ to „Rendering‟

Render ‘Farms’

Computing can take time, often minutes, hours or days.

Generate 24, 30, 60 or more images for a second

5 seconds at 30fps, is 150 images (frames).

If each takes one hour to render – 150 hours or over 6 days

It is clear that you need to distribute that workload

Collectively these computers are known as a „farm‟ or „render farm‟

Render Farm Management

How do you distribute all that work across those computers?

Manual assignment is time consuming and prone to error.

Render Farm Management Systems distribute work.

Provide management and flexibility

Multiple artists, editors, departments and teams can share.

Advanced Rendering Systems

Sanity Checks, Dependency Systems, Load Balancing, Notification systems,

Idle checking, Scripting access, and a myriad of other features.

Render Farm Management

Example Screen:

JOBS

SLAVES GRAPHS!

JOB TASKS

Render Farm Management Continued

How complex is the workflow?

Consider the following: Image ingestion and processing – cleanup, rotoscopy, tracking, color correction

Data ingestion and processing – point clouds, camera data, shared mesh, model assets

Simulation – complex fluid simulation, animation rig/hair simulation, breaking, dynamics, skin and cloth

simulation, fire, smoke and particle generation

„Rendering‟ – generation of images using a variety of rendering tools, creating passes, iterating light,

texture, camera and color

Compositing – input of images, compositing work and color correction

Encoding – generation of video formats and codecs

Collectively, often referred to as the “PIPELINE”.

Pipelines can differ completely.

Even within the same company.

Virtual Machines vs Physical Machines

A short summary

Traditionally Physical Machines aka „bare metal‟

More are building „Virtual Machines.

Flexible platforms (OS)

Software deployments

Archival

Look for a Render Manager that supports both Physical and Virtual machines

directly.

Lets talk about ‘Private Clouds’

Studio Compute Resources:

Dedicated compute resource

Idle time

Combination of those.

Multiple physical locations

Studios, design firms were essentially rendering on a „private cloud‟

Private Cloud Benefits

The Benefits of Rendering Locally

Data exists.

Security.

Pipeline and workflow.

Multiple offices, you can share your internal, idle resources.

Cost

If used consistently

Lets talk about Public Clouds

What are public clouds?

Offsite from your facility.

owned by a 3rd party

rent per month, per hour or per minute.

Amazon, Google, Microsoft, Rackspace

No access to „bare metal‟

Control by Virtual Machines.

Public Clouds have most advantageously been used by developers to

scale their websites, host on-demand software applications and as a

frequent delivery model for business/enterprise applications.

Public Cloud Benefits

Considering Public Cloud for your Pipeline

A fairly elastic resource

Scale

On demand billing

No initial capex fees

No real build-out time

Maintenance/systems

Lowered costs for infrastructure

A Note about SAAS

Public Clouds and SAAS

SAAS [Software As A Service] model

business applications (CRM, ERP)

within a browser

mobile applications

Pros and Cons

Often simple to engage

little control of the pipeline

Lack of ownership

Premium cost

Multi-resident issues

Render Farms aka Compute 2.0

Lets get to the good stuff

Basics

Reasons to expand

Make your work better

Get results faster.

Most important resource: your talent

Artist/Designer Time

0

20

40

60

80

100

120

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100105110115120

AR

TIS

T

PR

OD

UC

TIV

ITY

JOB TURNAROUND TIME (MINUTES)

Artist/Designer Time

0

20

40

60

80

100

120

140

160

180

200

S M T W R F S S M T W R F S S

PR

OD

UC

TIV

ITY

DAY

Artist/Designer Time

0

20

40

60

80

100

120

140

160

180

200

S M T W R F S S M T W R F S S

PR

OD

UC

TIV

ITY

DAY

Pop-Up Cloud

A hybridized approach to render farms

Combine local compute resources, offsite private resources, and public

cloud resources into one logical pipeline:

REMOTE Office

VPN

LOCAL Office

Database

&

Repositor

y VPN

CLOUD Region (on demand)

Pop-Up Cloud

How does it work?

Software/ Machine configurations

Created

Stored

Budgets and Needs

Data is Synchronized

Instances are controlled by a „Load Balancer‟

Lets go over that in a bit more detail!

Pop-Up Cloud – Software/Machine Configurations

Duplicating your workflow on a Public Cloud

Review Tasks

Build a Virtual Machine Image

Some Render Management Systems provide a default installation tool.

Map your VM Image to the Cloud machines [instances]

Assign Group names

Render Management System controls the system.

Software & Machine Configurations

Pop-Up Cloud – Budget Assignment

The Public cloud requires some constraints

There is a cost.

A system should not spend money if the jobs cannot render [assets]

Provide a set of rules

Cost Estimation

Be Safe

Be very aware of the „Free Soda‟ effect

Pop-Up Cloud – Budget Assignment

Pop-Up Cloud – Data Synchronization

Determine the best approach for your task

Data

Project file

Assets (models, textures, files)

Results

A variety of approaches to project rendering in the cloud:

Synchronize that data early

Sync everything

Sync on demand

Pop-Up Cloud – Data Synchronization

Further to Synching On-Demand (or ‘just in time’)

Asset dependencies in Render Management Software ensure that a

job is not started in a Cloud Region [or a remote office for that matter]

without the assets being available.

Asset lists are generated and compared to determine an action.

Asset dependencies can trigger a „sync‟ request.

JOB: SYNC

Asset TRANSFERRING

ASSET

QUEUE

JOB:

Render SUBMITTED

LOCAL CLOUD

ASSET DEPENDENCY:

ASSET REQUIRED

LOCAL

Pop-Up Cloud- Load Balancing Turning Cloud Machines on and off as required

Enabling cloud machines by hand would be tedious

Saves on cost, disabling and enabling

Balance across regions and providers.

True cross-cloud workflows, redundancy.

Cloud Region West

VM instance

VM instance

VM instance

VM instance

VM instance

Cloud Region East

VM instance

VM instance

VM instance

VM instance

VM instance

Pop-Up Cloud – SECURITY

How important is Security to your project?

SAAS models might be less attractive.

Most secure is local rendering

Private Cloud use inside a company or across owned resources next

most secure.

Public Cloud can be secure with VPN or HTTPS control

Because you control the VM, you can better secure/harden

Ultimately you determine what you process off-site. Elements vs. final shots

Remove data once you are complete

Pop-Up Cloud Conclusions

Musings and Thoughts

Public Cloud compute can help you get work done faster.

Other Considerations:

Delay decisions as long as possible?

Emergency assist?

Redundancy to ensure up-time/responsiveness?

Do you want to build the workflow of the future?

Is your team remote?

Enable time and materials billing?

Will it provide actual costs of rendering time?