fmx2013: butterfly effect

Butterfly Effect:Workflow, Rendering & Shading

Renaldas ZiomaUnity Technologies

@__ReJ__

Monday, April 29, 13

This talk will cover workflows and techniques that we adopted while working on the Butterfly Effect.

•

Unite template2

What is it?Real-time animated shortCollaboration between Unity, Passion Pictures and NVIDIA


If you haven’t seen Butterfly Effect is realtime animated short, developed in collaboration between Unity, Passion Pictures and NVIDIA

•

Opportunity to Learn

• Passion Pictures• New world of RealTime• No prior experience with Unity

• Unity Technologies• CG Pipeline• Huge amounts of Data

3


Passion Pictures is a film and animation production company with broad experience in CG, stop-motion, documentary and traditional media. Passion Pictures have developed visual elements for animated rock band Gorillaz, cinematics for Rock Band video game and infamous (in UK) Meerkat advertisements.

Prior to this collaboration Passion Pictures had NO experience with Real Time rendering / Game technologies or any experience working with Unity tools.

We approached Passion Pictures in late Fall of 2011 looking for collaboration on a real-time short.Real-time was a completely new area for Passion Pictures thus presented a learning opportunity to adopt RealTime for internal pre-visualization and leverage experience for working with Interactive projects (studio is working on number of such projects since).

Passion Pictures: “We felt we had an opportunity to do something special that we hadn’t done before, get experience with the Unity real time engine and help drive Unity in a direction that could integrate it with our studio.”

Project Outline

• 30GB of source Assets• 5GB of runtime Data• 137 shots• 40 “scenes” in Editor

4


Project Organization• Team of ~10 - first month

• pre production• research / tools

• Team of ~30 - next 2 months• content• tools / engine modifications

• Team of 5 - last 3.5 months• polish• optimizations

5


Unite template6


Passion Pictures presented a number of short film ideas and once Dan Sumich’s “Butterfly Effect” pitch was chosen - work started.Storyboard was developed from original pitch. Next storyboard was put into Animatronic - animated sequence of images in sync with music too capture flow and timing.

Example of the storyboards for Scooter section in development.

Unite template7


Color Boards helped to capture mood and lighting.

It is worth noting how well Story and Color Boards resemble the final look of the short.

Previz• One Sequence• Goals

• Initial timings• Visualize Space

• Prime lenses • 10, 14, 18, 21, 24, 27, 35, 40, 50, 75, 100, 200

• Zoom lens • 18 to 150

8


First pre-visualization was done in traditional to Passion Pictures way - directly in Softimage XSI.Only standard camera settings based on physically cameras and lenses as used in Live shoots (Prime/Zoom lenses) were used to capture the filmic feel.

Pipeline

• Connect Passion pipeline with Unity Editor• Main Goal

Minimal changes to Artist workflow!

9


Very short production cycle (traditional to CG industry) required minimal changes to artist workflow.Both Passion pipeline (XSI scripts) and Unity Editor scripting capabilities proved to be flexible enough.

Passion Pictures: “After the creative challenge of finding the right director and script for the film, the two immediate challenges were making sure that we had the right tools in Unity for the job so that we could take our pipeline into Unity, and also making sure that we could still work in the way we normally do as a studio. If we break down our pipeline into animation, simulation, rendering and compositing then we wanted to be able to bring these as much as possible into Unity working with our existing content creation tools as normal.”

Passion Pipeline

• No Change for Artists• Use Existing Asset Pipeline

• Sequences SB, BU• Shots SB_01_00, SB_02_00• Assets [Character/Camera/Layout]

• Export Models and Publish Shots as normal• Add Unity support

10


Standard Passion Pictures asset pipeline could be mapped to Unity asset representation quite easily. Multiple shots were grouped into a single Unity scene - some scenes contained one shot, some multiple. Sequence of shots (SB - SkateBoard, BU - Bus) became a group of Unity scenes sharing same folder.

Standard Passion Picture pipeline (set of Softimage XSI scripts) was augmented with “publishing” shots to Unity project file structure and to incorporate SVN version control. SVN was a new tool for artists.

Unite template

Character Rig

• XSI Softimage Rig• Bones• Envelope• Face Shapes• Controls

• Unity Rig• Bones• Blend Shapes• Materials

11


Unite template

Facial Rig

• Bones• Head, neck, jaw, eyes

• Blend shapes • Face

• Mocap shoot• Face markers

• Cubic motion • Retarget facial animation onto XSI controls

12


Facial animations in Softimage XSI / Unity: combination of bones and blend shapes.Face markers were used for facial motion capture, retargeted on controls in Softimage XSI.

Unite template13


Mocap facial shoot on the right. Animation transferred onto Softimage XSI character rig on the left.

We worked with Centroid for the motion capture shoot. Passion Pictures provided them with a FK rig containing just the bones and skin for our Hero character. Centroid sent FBX files for our FK rig send back FBX files of the FK rig from Centroid.

Using an intermediate rig Passion Pictures transferred the FK animation back onto our IK character rig in Softimage XSI. By keeping the shoot rig live in the scene, we were able to either keep the motion capture, animate on top of it or replace it with new animation, depending what worked in the shot.

Unite template14


Facial animation controls in Softimage XSI after retargeting.

Non-linear time

• Treat Sequences as Live Shoot• Cover the action from different angles• Each sequence needs multiple cameras

15


Action packed Butterfly Effect storyboard pushed Passion Pictures to treat film almost as a Live shoot. In traditional Live shoots multiple cameras are used to cover action from different angles. During the montage footage might overlap or jump in time according to Directors whim.

Film was broken down into sequences which correspond to a continuous action (such as Skateboard sequence).Description about every camera was created in Final Cut and exported to custom XML format (based on Edit Decision List or EDL file format of Final Cut).

Unite template

Non-linear time• Shot Description

• Final Cut edit• Export EDL as XML file• Camera in/out points

• Timeline in Unity

• Sequence Preview in RV• Using EDL from Final Cut• Play whole preview sequence in RV• Instant Feedback for Animators

16


Action packed Butterfly Effect storyboard pushed Passion Pictures to treat film almost as a Live shoot. In traditional Live shoots multiple cameras are used to cover action from different angles. During the montage footage might overlap or jump in time according to Directors whim.

Film was broken down into sequences which correspond to a continuous action (such as Skateboard sequence).Description about every camera was created in Final Cut and exported to custom XML format (based on Edit Decision List or EDL file format of Final Cut).

Unite template

Timeline in Unity

17


Next this data could be read in Unity and used to control cameras, animated objects and physical simulations. Custom timeline viewer was developed in Unity using Editor scripting. Changes to the film edit, jumping back and forward in time and reordering of the shots became possible without need to rebake animations in Softimage XSI.

Unite template

Timeline in Unity

18


Next this data could be read in Unity and used to control cameras, animated objects and physical simulations. Custom timeline viewer was developed in Unity using Editor scripting. Changes to the film edit, jumping back and forward in time and reordering of the shots became possible without need to rebake animations in Softimage XSI.

Unite template

Non-linear Challenges

• Animation• Particles• Physics

• Rigid Body simulation

19


Unite template

Point Cache

• Per-vertex Deformations• Cloth• Hair

20


Two types of vertex animation: BlendShapes and Point Cache were introduced in Unity for Butterfly Effect.

Passion Pictures: “In our standard pipeline we create point caches for each animated model in the scene using .pc2 or alembic files. This creates a break in the pipeline so that when we’re rendering we don’t need all the animation rig in the scene. For software rendering the cache sizes and data throughput aren’t an issue but in a real time game engine we have to keep the enveloped characters and bake the animation on the bones per frame.”

21

Point Cache


Point Cache used for extreme face deformation

Unite template

Tessellation

• CatmullClark• Now: on CPU• Future: OpenSubdiv library

22


(Mostly due to multiple layers of resting cloth) CatmullClark tessellation was used to match DCC tools exactly. It is problematic to implement CatmullClark using DirectX11 hardware tessellation because of recursive nature of certain patches. Instead of approximation we went for exact CPU implementation for 1st level of subdivision.

Unite template

Hair

• Guide “strands” for modeling• GPU Tesselation and “Amplification”•

23


Guide hair strands provide excellent control for animators.

Unite template24

Hair


Final look of the hair.

25

MentalRay Architectural (MIA) Shaders• Familiar• Small set of parameters• Physically Based

• Micro facet• Energy conserving


We aimed to provide PassionPictures artists with familiar tools, hence decision to implement realtime counterparts for MentalRay architectural shaders.

Unite template26

1 parameter for Diffuse


Between Clay and Plastic.Screenshot from the Unity Editor.

Unite template27

2 parameters for Specular


Roughness and ReflectivityScreenshot from the Unity Editor.

Unite template28

2 parameters for Reflectivity


Reflectivity for depending on angle of incidence. Defined as pair of parameters - for ray hitting at Grazing angle (90) and for ray perpendicular to the surface.Screenshot from the Unity Editor.

Unite template

MIA Implementation

• Oren-Nayar• Diffuse

• Cook-Torrance• Specular• Geometric + Roughness + Fresnel term• Ward distribution for Roughness

• I think... ;)• Direct + Indirect illumination

29


Unite template

Unity Implementation

• Oren-Nayar• very expensive• OK approximations

• Cook-Torrance• Normalized Blinn-Phong instead of Ward• Roughness to Phong exponent (n):

• n = 2 / roughness2 - 2 • Tweaked to match MIA look:

• n = 2 / roughness4 - 2• Schlick approximation for Fresnel

• Only direct illumination at RunTime

30


Unite template31

MentalRay Unity


Realtime implementation matches offline MentalRay implementation quite well. Note some small differences in size of specular and very glossy surfaces.

Unite template32


Most of materials where setup by Passion Pictures in Softimage XSI and were transferred to Unity. Tweaks were necessary to achieve final look in Unity, but relative compatibility of the MIA and Unity shaders saved a bulk of the manual work for us.

Unite template33

Indirect Illumination (GI)

• Point Clouds for Dynamic Geometry• Spherical Harmonics for Diffuse Bounce• Cube Textures for Reflections


For indirect lighting we went for very practical approach - storing secondary bounces in the low-resolution textures / Spherical Harmonics probes.

Unite template34


• Textures for Static Geometry• Diffuse Bounce• AO


Unite template35


• Calculated offline with Autodesk Beast• Path Tracer + Final Gather

• IBL for Sky• Fake Caustics


Beast setup:. 500 Final Gather rays (1 bounce). Path Tracer (4 bounces). 1024 IBL sky raysFinal quality scene bake would take between 20 minutes and 8 hours depending on the size of the scene. Bake quality for all scenes in the short was carried on over-night on 4 machines.

Unite template36

Real Time Shadows


Unite template37

Baked Shadows• Of course baked only for static environments


38

Ambient Occlusion• HBAO - Screen Space


Horizon-Based Ambient Occlusion - developed by NVIDIA.

39

Ambient Occlusion• Too Dark & Dirty


40

Ambient Occlusion• Applied ONLY to Indirect Illumination• Colored to match Sky better


Screen Space Ambient Occlusion information is calculated at the start of the frame and passed into the shaders, where it is applied to (occludes / darkens) indirect lighting only.Ambient Occlusion is tinted towards the sky to achieve more plausible look.

Unite template

Skin

• Texture Space Diffusion• 3 Layers

41


Subsurface only scattering of the skin on the left side. Lighting in texture (unwrapped UV) space on the right.

Unite template

Back Scattering• Translucency Map• Adds intensity to sub-

dermal layer

42


HDR

43

2 Specular lobes


Unite template

HDR

44


Final look of the skin.

Unite template45

Depth Of Field

• Antialiasing aware• FXAA based

• With Splatted Bokeh Texture


46


High variation of depth causes Depth of Field to produce noticeable aliasing on the edges of geometry. We married Depth Of Field filter with additional FXAA based antialiasing technique driven by radius of Circle of Confusion to blur neighboring pixels.

Example of aliasing caused by discontinuities in depth buffer on the left side. Our approach on the right side.


Depth Of Field using “standard” blur - top image.Detecting bright pixels and “splattering” bokeh sprites on top - bottom image.

48

Thank you!Talented people, come join our team working on Showcases for Unity!http://unity3d.com/jobs


References

49

[Bavoil08] Image-Space Horizon-Based Ambient Occlusion. L. Bavoil, M. Sainz, 2008

[Catmull78] Recursively generated B-spline surfaces on arbitrary topological meshes. E. Catmull, J. Clark, 1978

[CookTorrance82] A Reflectance Graphics Model for Computer graphic. R. L. Cook, K. E. Torrance, 1982

[Hoffman10] Crafting Physically Motivated Shading Models for Game Development. N. Hoffman, 2010

[Lottes09] FXAA. T. Lottes, 2009

[Mental07] mental ray Architectural and Design Visualization Shader Library. mental images GmbH, 2007

http://www.mentalimages.com/fileadmin/user_upload/PDF/arch_and_design.pdf

[Mikkelsen09] Microfacet Based Bidirectional Reflectance Distribution Function. M. Mikkelsen, 2009

[Nießner12] Feature-Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces. M. Nießner, C. Loop, M. Meyer, T. Derose, 2012

[Olano10] LEAN Mapping. M. Olano, D. Baker, 2010

[Schüler11] The Blinn-Phong Normalization Zoo. C. Schüler, 2011

http://www.thetenthplanet.de/archives/255

[Zhukov98] An ambient light illumination model. S. Zhukov, A. Inoes, G. Kronin, 1998




fmx2013: butterfly effect

Technology