Integration Of CG & Live-Action For Cinematic Visual Effects

by

AmarnathDirector, Octopus Media School

INTRODUCTION

Visual Effects (VFX) have served an increasingly important role in film since the late 1980s as the computational power provided by enhanced hardware and software has rapidly increased

Numerous films have demonstrated the art of invisibly bridging the imaginary and the real with the aid of computer graphics

It is possible to produce any kind of visual feast imaginable with computer-based technologies - Photorealistic actors, Dinosaurs, Aliens, Transforming Robots – you name it

The key to achieving realism in much of visual effects is to successfully combine a variety of different elements - matte paintings, locations, live-action actors, real and digital sets, CG characters and objects - into a single shot that looks like it was all there at the same time.

An important, subtle, and frustratingly complex aspect is to match the lighting amongst these elements. Not only do the objects and environments need to be lit with the same sources of light, they need to properly reflect and shadow each other

For successful compositing it is crucial to seamlessly integrate real and CG components.

Such a seamless integration requires solving a set of extremely difficult problems.

Most of these problems are related to recreating the following real world information:

Camera positions and parameters Illumination information to recreate the same

lighting environment for both CG and real objects

Workflow for Integrating Live-action and CG

Shooting Matchmoving CG Integration Image Based Lighting Rendering CG Compositing Color Grading

Shooting

CG Integration

Match Moving

Image Based Lighting

Rendering

Compositing

Grading

Output

Shooting

When shooting a scene that will use CGI or 3D Animation interacting with live action there are many precautions to be taken to ensure that you get what you need to make the shot work

Recording Camera Settings By recording the position and focal length of

camera during the shoot it allows the CG Artist to match the virtual camera with the field camera making the perspective of shot look accurate

Photo Reference Photos of surroundings to use in reflections Photos of reference objects

Green Screen For situations where CG characters and real actors

crossover

Matchmoving

Matchmoving is one of the most crucial processes in visual effects for integration of CG elements into live action footage. Matchmoving enables insertion of CG elements into live action footage so that the position, scale, orientation and motion of the CG elements are relative to the shot footage.

The process of matchmoving involves generating a virtual camera by tracking the movements of a camera through a live-shot scene, so that the tracked virtual camera can be used inside 3-D programs to match CG elements to the shot footage.

Process of Matchmoving

On Set - camera information (camera height and angle,

focal length) set measurements survey data (distance between camera and objects,

distance between track points, etc.) Matchmoving Software

Feature Tracking Camera Solve Exporting the camera for 3d software

CGI Integration

Based on the camera solve information exported from the matchmoving software, CG elements (Set Extension, 3D Character, Building, Vehicles etc.,) are placed at the right position.

Scale and perspective of the CG elements are also matched to the live-action plate.

The CG elements are modeled, textured, and lit to match the scene lighting

Lighting

The CG elements are lit to match the scene lighting using lights in the 3D program and as well as through Image Based Lighting

Image based lighting generally creates softer shadows, so its required to use the lights in the software to create better and sharper shadows

Image Based Lighting

In the context of visual effects, the advent of image-based lighting (IBL) has made a significant contribution to providing real-world lighting and rendering CG objects

IBL has tremendously facilitated the process of lighting-setup in standard 3D softwares in terms of raising the believability & photorealism of CG and live-action integration

Process of Image Based Lighting

Images of light in the real world are obtained in various ways

Fish Eye Lens (gives a 180 degree field of view) Mirror Ball or Light Probe (Photographs of varied

exposures of a mirrored ball placed in a real scene for catching the reflected view on its surface)

Fish Eye Lens Light Probe

HDR (High Dynamic Range) Images are generated from the photographs taken from any of the above method

The HDR image is then projected as an environment map in the 3D Software and used as the source of illumination through the aid of a global-illumination rendering process to provide the lighting matched to the original environment for CG objects and scenes,

Hence this process is named Image Based Lighting

Rendering

Once the CG elements are complete with all the details to make it look seamless in the live-action plate, the scene is rendered as an image sequence with various passes (render passes)

The render passes allows the user to make further corrections of the output in compositing software without the need to re-render the scenes

Render Passes

The most common passes that are rendered are Beauty Pass or Diffuse Pass Specular Pass Reflection Pass Shadow Pass Depth Pass AO (Ambient Occlusion)

Compositing

Compositing is the process of taking images from variety of sources and combining them in such a way that they appear to be shot at the same time under the same lighting conditions with the same camera.

The image sequences of the render passes are imported into a compositing software.

The sequences are then tweaked further for properties such as reflectivity, specularity etc so that they match perfectly with the live footage