Maths & Technologies for GamesStereoscopic Rendering

This presentation uses some red/cyan anaglyph images

Lecture ContentsLecture Contents

1. Human Vision• Depth Perception• Binocular Vision• Convergence and Accommodation

2. Stereoscopy: Background3. Stereoscopic Rendering in Practice

• Parallax• Cameras: Parallel / Toe-In / Off-axis• Rendering / Combining Two Scenes• Anaglyph Display• Optimisations

4. Improving the Viewer Experience• Adjusting 3D Strength• Avoiding Causes of Discomfort

Depth Perception – 2DDepth Perception – 2D

• There are a number of depth cues in a 2D image/video:– Position and perspective

• Nearer objects are larger and tend to be lower down in our vision

– Known sizes of objects, or relative sizes of similar objects

– Visible detail / texture• Detail is visible with less acuity in the distance

– Motion parallax• Nearer objects appear to move faster

– Shadows and lighting

– Occlusion• Nearer objects hider further ones

– Atmospheric blurring (“distance fog”)

• None of these require two eyes– Only require monocular vision

2D Depth Cues2D Depth Cues

Depth Perception – Binocular VisionDepth Perception – Binocular Vision

• We gain additional cues from having two eyes:

– Convergence• We turn our eyes inwards to see

nearer objects

• Our eyes turn almost parallel to see distant objects

– Binocular Disparity• The image seen in each eye is

different

• Our brain resolves it into one image with depth

Binocular Depth CuesBinocular Depth Cues

• Although we can infer considerable depth information from a single image, binocular depth cues are powerful– Especially for short to medium distances

• Our instincts and reactions have evolved to rely strongly on our binocular vision– Most predators have forward-facing

binocular vision to focus on a target– Prey, on the other hand, often have eyes

facing in opposite directions, favouring field of view over depth perception

Binocular Depth CuesBinocular Depth Cues

Depth Perception – AccommodationDepth Perception – Accommodation

• In the real world we get one additional cue:– Accommodation

• Eye muscles adjust the shape of the lens in our eye• To focus the light coming a given distance• Stretch the lens flatter to focus on distant objects• Squash it to a rounder shape to focus on nearer objects

• Accommodation does not occur when viewing a flat screen– Accommodate to screen distance regardless of any apparent depth– This can be a cause of discomfort viewing stereoscopic images

• Our eye muscles are converging, but not accommodating - unnatural

Stereoscopy - BackgroundStereoscopy - Background

• Stereoscopy enhances the depth perception of an image or video by presenting a different image to each eye– Viewer gets extra depth cues of binocular disparity & convergence

• The simplest form is just to place two images side-by-side:

View the effect by going cross-eyed until the cars merge (easier close-up)

Stereoscopy – BackgroundStereoscopy – Background

• For a single user, it is possible to provide two sources feeding a truly separate image into left and right eye:– E.g. head-mounted dual displays, or historically, the “View-Master”

• More common to combine left and right images into a single image and require viewer to wear special glasses– Can be viewed by several people at once

• However, this presents the problem of how to combine / separate the images– Preferably without quality loss

• Also, people don’t like to wear glasses…

Stereoscopy via GlassesStereoscopy via Glasses

• Three main types of glasses used for stereoscopic viewing

• Anaglyph– Two images are overlaid using different

colour components

– Glasses contain two distinct coloured filters to separate out the two images

• Red/cyan, red/green, or other variants

• Pros/cons– No special display hardware required

– Glasses are cheap

– Colour reproduction is poor

– Crosstalk – filters not perfect, can see left image in right eye and vice versa

Stereoscopy via GlassesStereoscopy via Glasses

• Polarised Light– Polarise the two images differently at

source and combine for projection– Glasses contain polarised filters to

separate out the images again

• Pros/cons– Glasses are cheap– Colour reproduction is good– No crosstalk– Requires special projection equipment– Polarisation will reduce brightness of

the projected image

Stereoscopy via GlassesStereoscopy via Glasses

• Shutter Glasses– Project alternate frames at a high

frequency– Glasses blank out each eye in

synchrony with the tv / projector

• Pros/cons– Colour reproduction is good– Little crosstalk– Glasses expensive– Requires capable display

Stereoscopy via GlassesStereoscopy via Glasses

• One screen per eye– Screen close to each eye– Or even on the eye – contact lenses

• Pros/cons– High quality image– No crosstalk or colour problems– No out of screen problems– Expensive (compared to other options)– Can be uncomfortable– Can only be used by one person

(Not always expensive, lol)

Stereoscopic Rendering in PracticeStereoscopic Rendering in Practice

• There are three key steps to stereoscopic rendering:1. Set up two cameras, one for each eye2. Render scene with these to two render targets (a stereo pair)3. Combine the two rendered images into one image for display

• The first point is straightforward if done correctly– However, there are some pitfalls to avoid

• Incorrectly rendered stereoscopic 3D may not look immediately wrong, but may give an uncomfortable, sub-standard result

• The latter two points are also straightforward where performance is not a concern

– However, real-world applications need to consider optimisation– Rendering the scene twice is not cheap

Stereoscopic Rendering – ParallaxStereoscopic Rendering – Parallax

• The same object on the left and right rendered scene will typically appear in two slightly different locations

• The relative positioning determines whether we see the object as near or far. This is called parallax:

Stereoscopic Rendering - CamerasStereoscopic Rendering - Cameras

• We normally consider the viewer as a single camera• When using one camera per-eye, we might initially think to

simply offset our single camera to the left, and to the right– Creating two parallel facing cameras

– The distance between the cameras is the distance between the eyes

– Called the interocular distance

• This approach will produce a comfortable result, but reduces the scope of the 3D effect

– In particular, it is not possible to create negative parallax

– I.e. cannot make objects look nearer than the physical screen

Stereoscopic Rendering - CamerasStereoscopic Rendering - Cameras

• We can try to rotate the cameras inwards in the same way that the eyes turn inwards to focus on an object

– Must decide a central focal point– This is called the “toe-in” method– It allows negative parallax

• But this approach has problems– It introduces vertical parallax

• Vertical differences between left and right eye

– We cannot move our eyes vertically independently so not comfortable

– Also sharply limits the comfortable range of depths that can be used

• Or viewer will have to cross or diverge their eyes too much

Stereoscopic Rendering - CamerasStereoscopic Rendering - Cameras

• A better approach is to try and combine the above– Parallel camera axes, but inward facing

• We do this with “off-axis” cameras– Both cameras face the same direction

– But offset the rendered area away from the axis towards the centre

• This produces a comfortable viewing experience

• Allows for negative parallax– Out of screen effects

• Requires a special form for the camera projection matrix

Standard Perspective ProjectionStandard Perspective Projection

• The standard projection matrix uses the near and far clip distances and the camera FOV:

ratioaspect viewport the and viewof field vertical theis distance clipfar distance, clipnear where

0)/(00

1)/(00

00)2/tan(/10

000)2/tan(/1

afovzz

zzzz

zzz

fov

fova

fn

nffn

nff

P

– This version reworked to use the viewport aspect ratio

• We need to update this matrix to shift the centre of the rendered area away from the camera axis

Off-Axis Perspective ProjectionOff-Axis Perspective Projection

• This is the horizontal off-axis variant:

parallax) zero to(i.e. planescreen the todistance theis andcamera theofoffset horizontal theis where

0)/(00

1)/(0)2/tan(/)/(

00)2/tan(/10

000)2/tan(/1

s

x

nffn

nffsx

zoff

zzzz

zzzfovazoff

fov

fova

P

– Note that this appears different from versions you might find online. However, it is equivalent and simpler to work with

• offx the horizontal offset is just half the interocular distance– The cameras are offset opposite directions– Note that zs is not the same as the near and far clip distances

Rendering / Combining Two ScenesRendering / Combining Two Scenes

• So we set up two cameras, each with their own view and projection matrices

– Each view matrix positions the camera at one eye. They both face down the same viewing axis

– Each projection matrix is of the form given above

• Then render the scene normally through each camera– Into two render targets

• For 3D display hardware, these two render targets are processed by the display API

– E.g. Displayed as alternate frames for shutter glasses

• Alternatively we can combine them in a post-processing pass into one image for display in anaglyph form

Splitting Colours for AnaglyphSplitting Colours for Anaglyph

• The core idea of anaglyph is to store two images in the separate colour channels of a single image

• For example, the left eye image goes into the red channel and the right image into the blue (and green) channels

– [For red/cyan anaglyph, other colour variants operate in a similar way]

• This can be written in matrix form:

2

2

2

1

1

1

100

010

000

000

000

001

b

g

r

b

g

r

b

g

r

– Where r,g,b is the combined image, r1,g1,b1 the first image (left) and r2,g2,b2 the second image (right)

– This equation simply copies the red from the left eye image into the red of the output, and the green/blue from the right eye into the green/blue of the output

Splitting Colours for AnaglyphSplitting Colours for Anaglyph

• Using the colour combining formula from the last slide:

2

2

2

1

1

1

100

010

000

000

000

001

b

g

r

b

g

r

b

g

r

• We get a result like this:• Colour reproduction is poor

– Inevitable with anaglyph

• There is also retinal rivalry– The red in the flowers is much

stronger than the green/blue– The eyes get very different images– Very uncomfortable

Splitting Colours for AnaglyphSplitting Colours for Anaglyph

• Recognising that anaglyph is not suited to colour reproduction we can create greyscale anaglyph. We get a different formula:

• A calmer result, but no colour:• The formula converts each image to

grayscale before copying into the channels of the output

– The (0.299,0.587,0.114) coefficients are from the Rec.601 broadcast standard for the luminance of the red, green and blue primaries. It takes account of the fact that for our eyes, green is brighter than red, which is brighter than blue..

2

2

2

1

1

1

114.0587.0299.0

114.0587.0299.0

000

000

000

114.0587.0299.0

b

g

r

b

g

r

b

g

r

Splitting Colours for AnaglyphSplitting Colours for Anaglyph

• Half-colour anaglyph calms the red channel by using a greyscale image at its source, but copies the blue/green directly.

– Calmer, some colour reproduction

2

2

2

1

1

1

100

010

000

000

000

114.0587.0299.0

b

g

r

b

g

r

b

g

r

• Optimised anaglyph creates a fake red channel from the blue and green of the first image

– Even calmer image, brighter colour

3/2

2

2

2

1

1

1

:red of correction gammaby followed

100

010

000

000

000

3.07.00

r r

b

g

r

b

g

r

b

g

r

OptimisationsOptimisations

• Rendering two scenes will clearly be costly

• There are a number of optimisations that can be employed given that the two images will be very similar

– Screen space reprojection: render the second image from the first by offsetting elements depending on their depth. Use inpainting techniques to fill any gaps. Turns out to be similar to parallax mapping

– Render both images at once to a single wide image using the geometry shader to create the duplicate geometry (problems?)

– Render with different optimisations in depth slices – e.g. furthest scene elements same in both images, render that slice only once

– What else?

Viewer ExperienceViewer Experience

• There are a number of settings that can be changed to affect the viewer’s experience of stereoscopic material:

– Interocular distance– Distance to the screen plane

• Affecting how much of the game world comes out of the screen

– FOV in projection matrix

• Each of these may need to be adjusted based on the physical situation of the viewer:

– Size of their display and how far away they are from it– Their actual interocular distance!

• Practically speaking it is as well to set sensible defaults for a given environment (e.g. sofa at home, computer desk)

• Only adjust interocular distance for “3D strength” slider– Distance to screen plane should be game dependent

Causes of DiscomfortCauses of Discomfort

• Poorly thought-out 3D can be off-putting or even nauseating

• Problems to avoid:– Extreme negative parallax – things

coming too far out of the screen– Window violations with negative

parallax – out of screen objects going over the screen edges

– Text & UI should normally be on the screen plane (but see below)

– Extreme parallax differences in a focus area, e.g. screen plane text over a distant enemy unit making eyes struggle to depth adjust

GalleryGallery

GalleryGallery

GalleryGallery

GalleryGallery

GalleryGallery