immersive actor feedback system

1 Research & Innovation

Immersive actor feedback system

• What sees the actor on set ?

( Answer! )

2 Research & Innovation

Immersive actor feedback system

• Viewpoint dependent projection– Virtual reality: CAVE system

• Integration into Chroma-keying environment– Retro-reflective cloth (BBC patent)

3 Research & Innovation

Immersive actor feedback system

• Retro-reflective cloth– “truematte” developed by BBC

Camera with LED ring

LED off

LED on

4 Research & Innovation

Immersive actor feedback system

Projector

3D model of Actor

3D model of virtual scene

ViewpointRenderer

Head position

Mask generation

5 Research & Innovation

Immersive actor feedback system• View dependent projection

The Ambassadors (1533), by Hans Holbein the Younger. National Gallery London

6 Research & Innovation

Immersive actor feedback

• Head tracking

Fast template matching filter on voxel data

7 Research & Innovation

Production Visualisation Tools

• real-time, low-quality

Monitor

Actor Feedback

Director’s View

3D model of real scene

3D model of virtual scene

On-set

Planning,On-set

8 Research & Innovation

Immersive actor feedback system

9 Research & Innovation

Bamzooki

• BBC Children Production

10 Research & Innovation

Free-viewpoint video for sports

• ‘Virtual Replay’ of football scenes– Allows viewer interactive

review

– Manually modelled offline

– Not realistic

11 Research & Innovation

Free-viewpoint video for Sport visualisation

• Piero

• Overcome limitations: use multi-camera system, iview project

12 Research & Innovation

Applying multi-camera techniques to sport scenarios

• Test shoot England-Wales in Old Trafford 9 Oct 2004

13 Research & Innovation

Where to put the cameras?

• Broadcast coverage cameras

• Pro:– Works without extra setup

• Cons:– Unreliable framing

14 Research & Innovation

Applying multi-camera techniques to sport scenarios

• Calibration with wand• Problems:

– Cameras knocked overnight– Cameras not rigidly mounted

15 Research & Innovation

Line-based calibration

• Colour-based edge detection

• Line-fitting• Calibration of camera

parameters (T,R,f) against dimensions of the pitch

• Runs at video frame rate

16 Research & Innovation

Processing - Segmentation

• Chroma-key against green

17 Research & Innovation

Processing – 3D Reconstruction

• 3D representations– Billboard (flat polygon, sprite)– Visual hull (Shape from silhouette)

18 Research & Innovation

Results

From 16 cameras, visual hull + graph-cut shape optimisation

(results from iview project, jointly with University of Surrey)

19 Research & Innovation

Results

20 Research & Innovation

Summary

• Computer vision techniques are important building blocks for the automated generation of visual effects

• For the integration of video and graphics several optical phenomena have to be harmonised. In descending order of importance:– Camera perspective, occlusions, shadows, ..

• Visual feedback systems integrated into the production pipeline are important to allow interaction with virtual objects

• Free-viewpoint video allows new insights– Goal: Stream 3D to the viewer at home

21 Research & Innovation

22 Research & Innovation

3D Modelling

• Graph-cut

• Details: A Bayesian Framework for Simultaneous Matting and 3D Reconstruction, J.-Y. Guillemaut, A. Hilton, J. Starck, J. Kilner, and O. Grau, 3DIM’07

23 Research & Innovation

Results

• Video from ORIGAMI demo production

24 Research & Innovation

Applications: Requirements

I

coarse rich detailed

3-D

II

III IV

2 1/2-D2-D

Applications:

a) virtual studiob) On-set Visualisationc) Pre-Visualisationd) Interactive e) Digital effects

Optical phenomenon:

1) camera perspective2) occlusions3) shadows4) reflections

1 2 3 4

I

II

III

IV

a

b

e

Shape classes:

d c

25 Research & Innovation

Introduction

• Conventional work flow in (feature) TV or film productions

PlanningPhase

On-setPhase

PostProduction

Animations /Virtual Set

Storyboard

FinalProgramme

expensive

26 Research & Innovation

Introduction

• ORIGAMI approach– Mixed-reality techniques for visualisation through the production

PlanningPhase

On-setPhase

PostProduction

Animations / Virtual Set

Storyboard

FinalProgramme