Is 3D More Complicated than 2D …. Really?

Philip Corriveau Intel Corporation

Paulette Pantoja BluFocus Inc

Human Factors Steering Team

February 22, 2011

SMPTE Hollywood Chapter

3/4/2011 1

* Wikipedia ViewMaster Article

* Jeff Hecht. 3D TV and Movies: Exploring the Hangover Effect

* Disney Toy Story 3 Web Page

Evolution of 3D

2

Short Answer is Yes……

Picture this…..

3

• Meet the consumer demand for a better viewing experience whether

the consumer can not tell if the experience is real or an optical

illusion (Onural 2006)

• Consumer experience involves more than a display, it involves all

facets of the 3D system and how they interact with the consumer

• It also involves the expectations of the consumer as well as the

perceptions from the media

Goal

4

• Information gap between the media, the consumer and the current technology

• 3-D glasses coming soon from Oakley, Marchon and Xpand. Real life will

continue to be in 2-D, however (USA TODAY)

• Nintendo Warns on 3-D for Children. Young children shouldn't play three-

dimensional games. Looking at 3-D images for a long period of time could have

an adverse effect on the eyesight development (Wall Street Journal)

• The American Optometric Association (AOA), representing America’s family

eye doctors, says 3D in movies, TV and even 3D on Nintendo’s 3DS isn’t

necessarily bad for adults or children (AOA)

Why Is It Complicated?

5http://www.usatoday.com/tech/products/2010-11-08-3dglasses08_ST_N.htm

http://online.wsj.com/article_email/SB10001424052970204304204576051021329863968-lMyQjAxMTAxMDAwNDEwNDQyWj.html

• Eyes separated by a small difference

seeing two slightly different images

• Images are combined in the brain

creating the illusion of depth

• Other cues to depth include texture,

binocular disparity, perspective

• Human Factors study how these cues

affect perception and the user experience

The big picture: 3D Physical & Cognitive!!

6http://3duniversity.net/page.aspx?page=7

ST 5 Plans around - 3D STUDIES/RESEARCH

7

Overall Goals

• Phase 1: Gather Research

• Phase 2: Create 3D Test Material & Test

Plans

• Phase 3: Conduct Research and Studies

on the effects of 3D

• Phase 4: Release Studies

8

Phase 1 – Gather 3D Research

9

• Successfully gathered research on past and current

3D studies

• Concluded that further studies need to be conducted

because previous/current studies:– are not done with “Real” 3D Content. Current 3D Content is purely

technical and lab driven with more scientific results.

– are not done in typical “at home” testing environments using a

matrix of typical viewer equipment/hardware

– do not have a wide enough range of testing scenarios that closely

mimic typical user viewing habits

Phase 2 – Create Research

10

Conduct Studies

Use 3D@Home relationships with

Educational Institutions, Committees,

Groups, etc. to conduct 3D Human

Factors studies using “Real 3D

Content” and “3D Test Plan/Matrix”

Phase 3

Create “Real 3D

Content

Create “3D Test

Plan/Matrix”

Phase 2 – Create Research (cont.)

11

Create “Real 3D Content”"Bad" 3D Material

developed with variations using "tricks", ghosting, extreme divergence, multiple points of reference, abrupt

depth changes, depth conflict, reverse 3D, etc.

"Good" 3D Material developed using preferred techniques that do not have

the issues noted above

"Good" 3D Material mixed with "Bad" 3D Material

3D Material that goes to 2D for short breaks and returns to 3D Create “3D Test

Plan/Matrix”

Conduct Studies

* Subcommittee Needed

Subcommittee To-Do’s•Create “Real 3D Content” types neededDetermine/Oversee:•Who will create content•Who will encode content•Who will author content

Phase 3

12

Create “Real 3D Content”

Create “3D Test Plan/Matrix”

A matrix for how to test the “Real 3D” content must be created.

• *CONTENT: Bad 3D Content, Good 3D Content, Mixture, Animation, Live Action, Graphics, etc.

• *VIEWER: Age, Sex, Prescription Glasses, Medical conditions, viewer calibration, etc

• *HARDWARE: Display type, etc.

• *ENVIRONMENT: Distance from display, Angle of display, lighting conditions, viewer positioning

• *DURATION: Viewing time, viewing intervals, etc.

Conduct Studies

* Subcommittee Needed

Subcommittee To-Do’s•Test plans/Matrix Creation•Work with other ST committees •Work with current group relationships

Phase 2 – Create Research (cont.)

Phase 3

13

Conduct Studies

Use 3D@Home relationships with Educational

Institutions, Committees, Groups, etc. to

conduct 3D Human Factors studies using “Real

3D Content” and

“3D Test Plan/Matrix”

Subcommittee To-Do’s•Develop ST5 3D Study Program for groups to participate in

•Process, Submission Technique, Database for collection, etc.

•Identify ST5 relationships across variety of areas to conduct ST5 3D Studies:

•Educational•Scientific•Etc

Phase 3 – Conduct Studies

Create “Real 3D Content Create “3D Test Plan/Matrix”

ST 5 Overall Summary

Phase 1 - Research

Phase 2 - Create

Phase 3 – Conduct

•Completed yet ongoing for possible future collaborations and for continued knowledge

•Create a “3D Test Plan/Matrix subcommittee

•Create “Real 3D Content” subcommittee

•Create “Conduct Research” subcommittee

14

Create Real 3D Test Content and 3D Test Plans in order to provide and work with Institutions to conduct 3D Human Factor Studies which we will later share with the

Industry

Release research conducted to the Industry in order to assist in the creation of good 3D to ensure its longevity and overall proper usage for entertainment, education, etc. purposes

Using the output of all 3 subcommittee’s, conduct research

Phase 4 – Release

What Are Impressions….

15

User experience: Impressions of 3D

• Online study from CEA

and Entertainment

Technology Center of

1,914 adults between

December 9-21, 2009

• Overall, negative factors

associated with 3D

decreased after

experiencing 3D

16http://www.etcenter.org/etc-activities/projects/consumer-3d-experience-project/3d-research/

User Experience: 3D Satisfaction

• 85% were satisfied with the overall experience, four percent

were neutral to the quality but still satisfied with the experience

17http://www.etcenter.org/etc-activities/projects/consumer-3d-experience-project/3d-research/

Physiological Aspects of Real life..

18

• Accommodation: Ability the eye to focus on an object

• Vergence: Ability of the eyes to move in opposite directions to form a

single image

Vergence-Accommodation

19

Changing viewing distance

• Accommodation

http://www.sapdesignguild.org/editions/edition9/images/accomodation.png

Changing viewing distance

• Vergence

http://www.sapdesignguild.org/editions/edition9/images/accomodation.png

HF Implications by 3D Display

22

Technology Examples Associated advantages Associated

Disadvantages

Passive glasses, 1 panel Anaglyph, Micro polarizer,

Xpol, Active Retarder

Inexpensive glasses Vision Fatigue, Vertical

resolution is reduced

Passive glasses, 2 panels StereoMirror, Variable

Polarization

Flicker-free, no fatigue Expensive, bulky

Active Shutter Glasses Checkerboard Frame-

sequential

Low price, unlimited 3D

viewing angle

Shutter glasses,

brightness is reduced.

Can be expensive

Autostereo Parallax Barrier,

Lenticular

Do not require glasses Horizontal Resolution

Reduced. Low Brightness

Human Factors and Stereo Studies

23

Health Concern Outcomes Studies Limitations

Visual Fatigue /

Visual Discomfort

Subjects with moderate binocular

status show meaningful changes

Lambooij 2010 Content, Low exposure

Motion Sickness Strongest symptoms after stereoscopic

game. Movie in HMD was relaxing

Hakkinen 2002 No comparison between

stereo and non-stereo

games, HMD

Postural Sway Sway effect for stereoscopic game not

for movie

Hakkinen 2002 HMD, standing and

sitting

High-Level Cues Incorrect high-level cues such as

motion parallax suggest mind strain

Patterson 2010 No study conducted

Visual Fatigue /

Visual Discomfort

Binocular Viewer in children had few

additional

adverse effects compared to HDTV

Kozulin 2009 HMD

• Visual Fatigue

– Objective measure of decrease in performance of

the human vision system

• Visual Discomfort

– Subjective measure

• Visual Discomfort measurements should

provide an indication of the objectively

measurable visual fatigue

• Disparity

• Vergence-Accommodation

Human Factors and Stereo

24Lambooij 2009, Patterson 2009

• There are no studies on long term exposure

• Studies usually involve computer generated

content and immersive environments

• People who already have bad vision present

symptoms (Hakkinen 2002)

• There is no strong evidence for or against

stereoscopic applications causing permanent

damage to the visual system (Lambooij 2009)

General notes of current research

25

• A holistic concept that extends beyond the

quality of the sensory information and

involves presence, immersion (Chertoff 2010,

Strohmeier 2009)

User experience

26

Dimension Description

Sensory Sensory Input and perception of the stimuli. Hardware and software

Cognitive Mental Engagement with the experience. Task engagement

Affective Emotional state of the user.

Active Personal connection the user feels to the experience.

Relational Social aspects of the experience.

• Content– Longer length

– Action movies, live events, advertisement, games

– Real-unreal content does not necessarily reflect

good to bad image quality

• Unsuitable content– Soap Operas

– Documentaries

– Talk Shows

– News

User Experience: expectations(Strohmeier 2008)

27

• Requirements– Presence and engagement

– High viewing comfort

– Compatibility

– Costs

– Auto displays

– Viewing freedom

– Switchable 2D/3D

– Low cross-talk

User Expectations

28

Measuring User Experience: Subjective (Meesters 2004,

Lambooij 2009)

• Exploratory Studies

• Psychophysical Scaling

– Performance Oriented

– Appreciation oriented

• Questionnaires

29

Name Measures Type

Double-stimulus-continuous-

quality-scale method

(DSCQS)

Overall image quality for short video

sequences

Subjective

Single-Stimulus Continuous-

quality evaluation (SSQE)

Continuous time-varying quality judgments of

longer video sequences

Subjective

Measuring User Experience: Objective

Measurements (Lambooij 2009)

• Direct Measurements

– Pre- and post- tests to measure

visual fatigue

– Record vergence and

accommodation

• Indirect Measurements

– Prism, visual charts

• Brain Measurements

30

Future Research Directions

• User experience for 3D@Home involving more than

visual perception

– Immersion

• Studies that involve long exposure

• Develop methods to measure for user experience

– From Image Quality

– Scalability

31

Research Findings

32

• Home Theatre Study - Yang S,

Cooper S, Corriveau P, Doherty R,

Sheedy J

• Intel & Vision Performance Institute -

Pacific University College of

Optometry

• To compare symptoms between 2D

and 3D viewing of the same movie

Current studies on Human Factors

33

• Two hundreds and five adults (44%

female, averaged 36.6 years of age)

• Wore habitual optical correction

• Randomly assigned to 2D or 3D viewing

groups.

• Samsung 55” HD3D LCD TV

• 17 item questionnaire before and after

viewing

Procedures

34

Results

Figure 1. The 95%

confidence intervals of the

natural log odds ratio for

changes in visual and physical

symptoms (Red = 2D, n=103;

Blue = 3D, n=100). Bars

placed entirely to the right of

zero (0) indicate symptoms

that significantly increased

from baseline during or after

viewing. Bars completely left

of zero indicate symptoms

that significantly decreased

from baseline.

Results

Figure 2. The 95%

confidence intervals of the

natural log odds ratio for

changes in visual quality and

motion symptoms (Red = 2D,

n=103; Blue = 3D, n=100).

Illustrated as in Figure 1.

36

Current Studies on Human Factors

• Vergence, Accommodation, and Visual Symptoms in 3D Viewing -Shun-nan Yang

• Intel & Vision Performance Institute -Pacific University College of Optometry

37

Testing Setup

Variance in Vergent Response

Variance in Accommodative Response

Conclusions

• Current research is focused on the visual system

and the visual experience

• Lack of studies on long exposure and 3D home

content

• Lack of studies on the user experience

• Need for metrics that tie the image quality to the

user experience

41

The Ecosystem

42

The big picture: S3D System

43

S3D System

Capturing S3D

Coding Delivery Display

User

What we think happens NOW

The big picture: S3D System

44

S3D System

Capturing S3D

Coding Delivery Display

User

The Reality (the Consumer)

UserUser User

• Acquiring the real-world scene and the

corresponding 3D cues digitally while

keeping the original visual appearance

(Onural 2006, Su 2010, Meesters 2004)– Traditional 2D video camera

– Depth Camera

– Two or more cameras

• Artifacts:– Cardboard effect

– Keystone

S3D System: Capturing S3D

45

Capturing S3D

• Objects appear flat, like a scenery in a

theater (Meesters 2004, Reyes 2010)

• Caused by image acquisition parameters

or coding

• No depth perception for the consumer

Artifact: Cardboard Effect

46*Traveling Log: http://www.colouredbuttons.com/2009/06/matchbox-monday-4-rocky-

monster.html

• Trapezoidal picture shape in opposite directions

• Due to a converging camera configuration where

the cameras are positioned at an angle toward

each other

• More noticeable in the corners

• May produce visual fatigue and negative effects

on appreciation-oriented assessments

Artifact: Keystone Effect

47

• Captured data needs to be coded into a format for

delivery and display

• Several methods for coding, depend on the

capture technology

• Most common is subsampling like 2D video

• Degradation Artifacts

– Blocking

3D System: Coding (Su 2010)

48

Coding

• Discontinuity between consecutive

blocks in a picture (Cancellaro)

• One of the most relevant artifacts in

coding (Cancellaro)

Artifact: Blocking

49

• Limited by the specific requirements of

the capturing technology

• Broadcasting

– Bandwidth limitations

• Artifacts in delivery depend on coding

– Packet loss

3D System: Delivery (Su 2010)

50

Delivery

• Variant in terms of duration, intensity,

and occurrence (Boev 2009)

Artifact: Packet Loss

51

• Stereoscopic

– Require eyewear

• Auto-Stereoscopic

– Glasses Free

• Head-Mounted Displays

– Two distinct images in front of the two eyes

–

• Volumetric/Holographic

– Not a flat surface

– No glasses

3D System: Display

52

Display

• The image destined for one eye is visible to

the other eye

• Perceived as a shadow, ghost

• Caused primarily by display technology

• Can lead to visual discomfort

Artifact: Crosstalk (Meesters 2004, Reyes 2010)

53*Boev 2009

Stereoscopic displays and glasses

• Another optical element

• Reduce brightness by at least 50%

• Can be physically uncomfortable

• Awkward for people who need glasses to see clearly

S3D and Glasses

1st frame

L R L R L LR R

• Using polarized glasses

• Seeing visual information for left eye & right eye

at a time.

2nd frame

Polarized

Glasses

L L L L R R R R

• Using shuttered glasses

• Seeing visual information for left eye & right eye

sequentially.

1st frame 2nd frame

Shuttered

Glasses

Film Patterned Retarder (FPR) Shutter Glasses

Shutter glasses (SG)

• DLP technology

• Less Flicker

• Complex timing

– “Cross-talk”

FPR 3D glasses

Polarized Glasses

PolarizerQWPQWP

QWP_LQWP_RQWP_LQWP_RQWP_LQWP_RQWP_LQWP_R

QWP_LQWP_RQWP_LQWP_RQWP_LQWP_RQWP_LQWP_R

•QWP : Quarter

•Wave Polarizer

Film Patterned Retarder

3D without glasses

Autostereoscopic Displays

Parallax Barrier Lenticular Lens

Backup

59

• Fusion limits are affected by eye

movements, stimulus properties, duration

of exposure, amount of illuminance,

individual differences

• Fusion limits decrease with smaller,

detailed, and stationary objects

• Increase with larger, moving objects

Disparity

60

Virtual 3D• Separate images need to be shown to

each eye

• In order to view different depths,

– Vergence is required,

– Accommodation must remain the same

Vergence-Accommodation and 3D

S3D viewing

• Optimal human response to S3D requires different, and

varying, accommodation/vergence ratio than

encountered in real 3D

Vergence-Accommodation

Donder’s Line is the real world

The zone shows the areas of flexibility

Image courtesy of VISERG, Loughborough University

AC/A and CA/C

• Accommodation can drive vergence– AC/A is 4^/D (mean)

• Morgan MW. Clinical measurements of accommodation and vergence. Am J Optom 1944;21:301-313

• Vergence can drive accommodation– CA/C is about 0.5D/6^

• Schiemann and Wick

• And they vary by person

Which link is strongest?• A normal viewing distance of 40 cm requires

– 2.5 D of accommodation (less because of depth of field)

– 15^ of vergence

• It takes 30^ of vergence to induce 2.5D of accommodation

• It takes 3.75 D of accommodation to induce 15^ of vergence

• Neither one is strong enough to induce the necessary amount of the other

• This creates a vergence/accommodation “space” within which – disparity and proximal cues drive the eyes to fuse, and

– blur and proximal cues drive them to accommodate.

How should the visual system respond to 3D?

• Vergence without accommodation?

• Does accommodation always

accompany vergence changes?

• Do all subjects react the same?

– Almost certainly not

• Which subjects are having

symptoms?

Optometrists regularly test the accommodation and

vergence zone (ZSCBV)

• NRC and PRC are changes in vergence with no change in

accommodation

• NRA and PRA are changes in accommodation with no changes in

vergence

Symptoms caused by binocular vision disorders

• eyestrain

• eye fatigue

• general fatigue

• intermittent blurry vision

• losing one’s place while reading

• double vision

Stimulus Differences

Real Life S3D

Blur is a stimulus Blur is not an appropriate stimulus

In fact, it may hinder because it can

be inappropriate

Disparity is a stimulus Disparity is a stimulus

Proximal is a stimulus Proximal is probably weaker

Inappropriate blur

• When blur is inappropriate for the

intended depth,– then the perception of depth in space can be

distorted.

– Watt, Akeley, Ernst, Banks. Focus cues affect

perceived depth. J Vis (2005) 5, 834-862.

Appropriate blur

• When blur is appropriate:

– reaction time to depth stimuli is reduced

– Spatial distortions are reduced

– Fatigue and discomfort are reduced.

• Hoffman, Girshick, Akerley, Banks.

Vergence-accommodation conflicts

hinder visual performance and cause

visual fatigue. J Vis (2008) 8(3).

Symptoms related to Accommodative Disorders

• near blur

• post-work distance blur

• slowness of focus changes

• eyestrain

• general ocular discomfort