Cinematography in Augmented Reality

Richard Shemaka

1. Background on Camera and Cinematography2. Introduction to AR3. Camera in AR4. Two Approaches to Cinematography in AR

Exploring the notion of camera and presentation in augmented reality

Camera and Cinematography

• Ever since the advent of film, the notion of camera has become an import tool for media makers to convey their ideas

• The moving photographic image gave rise to a new language of emotion and meaning-making: cinematography

• Some larger considerations of cinematography include lighting, camera movement, composition (framing, blocking, etc), and special effects

Camera in Entertainment Media

• While traditional cinematography is concerned with the photographic image, we can derive important aspects with respect to any moving image

• The lessons learned from photography and cinematography have been applied to great affect in new media such as websites and video games

Uncharted 2: Among ThievesNaughty Dog - 2009

Augmented Reality

• An emerging media which blends reality with computer images

• First appeared in the early 1990’s• In recent years, advances in mobile

technologies have allowed for more wide-spread development

History of Camera in AR

• Wearable AR systems focused on first-person presentation of material

• Focus was more on the presentation of information– Ease-of-use and transparency

have been the goals of many AR camera systems

Camera in AR

• Most AR cameras rely on user controlled position and orientation

• This alone renders many of the traditional techniques of cinematography irrelevant– E.g. calculated camera movement

Leveraging Cinematography in AR

• As with most new forms of media, we can bring our knowledge of older media to bear when designing new experiences (remediation)

• Recently, video games have started to integrate cinematic camerawork with interactive controls

• Because most interactive AR applications rely on a user-controller camera, new techniques and approaches will be needed

Lighting

• Lighting is a powerful cinematic tool that could be easily adapted to AR

• Both scene lighting and interactive light-based mechanics could contribute to a scene’s presentation

Framing and Composition

• Framing and composition in film are generally accomplished with careful camera placement

• I am proposing two separate approaches to accomplishing these effects in a typical handheld augmented reality interface– Guiding the user’s gaze within the space– Adjusting the scene according to the user’s view

Two Separate Approaches

Focus of Attention• The user’s attention is

directed towards particular parts of the scene

• Borrows directly from film with techniques such as focus, action, framing, etc.

Camera-Aware Scenes• The scene recognizes and

adapts to the way it is being presented

• Props and characters change themselves to better ‘fit the shot’

Focus of Attention

• Loaded Scenes– In this approach (originally taken from

stage), the scene is constructed in such a way to lead the user towards a particular view

– Borrows heavily from composition• Action– Actors and events direct the user’s

attention towards something particular

Camera-Aware Scenes

• Actors’ positioning and pose– Having the actors in a shot change to better

present themselves• Shifting scene– The scene shapes itself based on the way it is

viewed

Implementation

• In order to explore these designs in practice, I plan to build a suite of AR prototypes which focus on cinematic presentation

• With the nature of prototyping, I will not be going into these with any expectation beyond leveraging the designs I am outlining for the project

Platform

• Qualcomm’s AR SDK– Natural feature tracking– API for some AR-specific mechanics

• Unity3D for Android– Game engine + IDE– Integrated with Qualcomm’s SDK