* Sharp Laboratories of Europe, Ltd.

Abstract

New three dimensional (3D) displays switching electrically between 2D and 3D modes are discussed. Theprinciples of operation and design considerations for particular applications are described.

Introduction

A display is presented that can be electricallyswitched between an autostereoscopic (no glasses) 3Dmode and a full resolution 2D mode. This 2D/3Ddisplay can be used in a range of products fromcomputer monitors to mobile displays. There aremany applications including 3D games, amusement,image capture and display. The 2D mode allows theuser to enjoy the same performance as currentdisplays, with the added advantage of 3D forenhanced reality and enjoyment. In September 2002, Sharp announced massmanufacture of electrically switchable 2D/3Ddisplays. The first product was launched in November2002: a 2.2" 2D/3D mobile phone for NTT DoCoMo(SH251iS).

1. History

In 1992, SLE began researching 3D displays - an exciting new application that added to Sharp's high qualityLiquid Crystal Displays.The first systems1) comprised two standard LCDs mounted at 90｡ with beam combining optics to send theimage from one LCD to one eye and the image from the second LCD to the second eye. However, thisparticular set-up was too large and probably too expensive for the mass market. In 1994 we achieved ourfirst single-panel 3D display2) based on the parallax barrier method3). These displays were called "3D-onlydisplays" since the 3D effect was permanent. These displays allowed a single user to enjoy 3D from certainpositions. This technology was improved in 1996 with the invention of the "sweet spot indicator" to help theuser find the best 3D viewing position. This indicator is one of the key distinguishing factors of Sharp's 3Dtechnology4).Up until 1997, the 3D displays were not suitable for conventional 2D applications since only 3D images

- 1-

2D/3D Switchable Displays

Adrian Jacobs* Jonathan Mather* Robert Winlow* David Montgomery* Graham Jones*

Morgan Willis* Martin Tillin* Lyndon Hill* Marina Khazova* Heather Stevenson* Grant Bourhill*

Fig. 1 SH251iS Mobile phone with 2D/3D display.

could be viewed. The breakthrough came by using polarisation optics to allow switching between full-resolution 2D and no-glasses 3D modes. 8.4" and 13.8" displays were built that allowed switching between2D and 3D by a simple mechanical method (see5)).In October 2001 a key target was finally achieved: the first prototype of a 2D/3D display that could beelectronically switched between modes. The key advantages are that the 2D image quality is identical to astandard 2D display, whilst the no-glasses 3D mode provides exciting and comfortable to view images withenhanced reality.

2. Principle of an autostereoscopic display

The basis of our 3D display without glasses (autostereoscopic) is the "parallax barrier". The parallax barriercomprises alternating transmissive and non-transmissive columns aligned with the columns of the LCDpixels. The transmissive columns generate two regions in space (or viewing windows) in front of the displayas shown in Fig. 2 and Fig. 3.

Two 2D images are displayed on the LCD. When theobserver places the right eye in one viewing window(one 2D image) and the left eye in the other (thesecond 2D image) then a 3D image will be perceived.Each eye sees a different 2D image and this creates a3D perception. 3D viewing distance can be estimated with theparameters in Fig. 4. The important features are pixelpitch and the distance between parallax barrier andLCD. The 3D viewing distance can be matched withthe best 2D viewing distance so that the user is in acomfortable position for enjoying 3D. This emphasison user comfort is another key distinguishing factorfor Sharp 2D/3D displays.

2･･1 Design considerations for electrically switching 2D/3D display

If normal text is read using a standard 3D display, the text can be distorted. Additionally, as the user movesfrom one viewing window to another, there are often brightness variations. It is therefore important to beable to "switch off" the parallax barrier function to obtain a high quality 2D mode. This switching can be

- 2-

Fig. 2 Viewing windows created in space. Fig. 3 Parallax barrier display.

Fig. 4 Viewing distance calculation.

achieved by use of a patterned retarder parallax barrierand removable polariser5). The switching is achievedmechanically by the addition of a secondary polariser.This mechanical switching is ideal for low-costapplication, but may be impractical for others. Electrical switching can be achieved by the addition of acomponent that can rotate the polarisation of light (3D)or leave it unaffected (2D). This additional componentcould be a simple liquid crystal cell. A possibleconfiguration is shown in Fig. 5.In 2D mode, the two columns of the patterned retarderparallax barrier transmit equally. Importantly, the 2Dimage quality is the same as a standard LCD panel. In 3D mode, one column of the patterned retarderparallax barrier transmits and the other column is opaque, resulting in a 3D mode.Other aspects of the design depend on the particular application. For example, the same 3D effect can beachieved with a parallax barrier at the rear of the display (between backlight and LCD) rather than at thefront (between LCD and user). This rear parallax barrier arrangement is useful for "Advanced TFT"transflective displays that operate in both transmissive and reflective modes. With a rear parallax barrier, thereflected brightness is identical to a standard 2D display. The transmitted brightness is reduced. With afront parallax barrier, both the reflected and transmitted brightness would be reduced.

3. The road to production

In late 2001 the electrically switched 2D/3D technology was shown to Sharp's Communications SystemsGroup, who wanted to integrate 3D into future mobile phones. A key to commercial success was the detaileddiscussion that followed, where the possible technical options were compared with detailed understanding ofcustomer's requirements. In keeping with Sharp's history of high quality "only-one" products, a higher riskbut higher performance option was chosen.A very successful and intensive collaboration followed between CSG, SLE and Sharp's Mobile LCD Group,responsible for mass manufacturing the displays. SLE made initial prototypes of 2" displays and thenimproved the design and performance. Additionally, a number of "test" displays were created to help MobileLCD group define manufacturing tolerances.At this stage the only 3D displays that had been produced were "hand-made" prototypes built at SLE. Thedesign and processing knowledge had to be transferred to Mobile LCD group, with the aim of setting-up amass manufacturing line.In order to be "first to market", a challenging target of 9 months was set to complete this knowledge transfer.To help the information flow between SLE and Mobile LCD BG, two researchers from SLE spent 9 monthsin Sharp Tenri working closely with colleagues from Mobile LCD BG. This 9-month visit was a key part ofthe successful commercialisation.The commercialisation included replicating the SLE fabrication route in Japan. Once all productionproblems had been overcome, the next step was to decrease the production cost, and increase the yield. Forexample, several chemicals are needed to make the parallax barriers. Cost and time reduction in massmanufacture is possible using chemicals that can be applied in the manufacture of both LCDs and parallaxbarriers. SLE and Mobile LC BG continue to investigate novel processes that will simplify manufacture.SLE benefited greatly from spending 9 months at Sharp Tenri by understanding the methods and constraints

- 3-

Fig. 5 Display configuration.

of mass manufacture. This will help us to better consider business group constraints and needs whendeciding future IP.

4. 3D contents

The Sharp 2D/3D display is an integrated system - software is as important as the hardware for good 3D.Alongside the hardware development, a key part of the research at SLE has been understanding anddeveloping protocols to allow comfortable 3D. The protocols control the depth in images. The depthinformation for the protocols comes from several Human Factors studies commissioned by SLE.Another important area for content generation is 3D image capture from stereo digital photography. Oneapproach is to have a single camera with two sensors. Another approach is to have a single camera/sensorand a prism adapter to produce two views. Yet another approach is a single camera mounted on a sliding railto capture two images. Another alternative approach is to take two separate images with a single camera andthen apply software correction of image errors. All these approaches are part of SLE's current and futureactivities in this area.

5. Future challenges

The immediate future should involve the extension ofthe electrically switching technology to a wider rangeof applications. For example, it can be effectivelyapplied to larger area displays such as computermonitors (Fig. 6)For future research, there are many possible advancesto widen the appeal of Sharp's 2D/3D displays. For example, we will aim to widen the position inwhich a 3D image can be viewed and thereforeincrease viewing freedom. This is particularlyimportant for gaming applications. Additionally, wewill aim to develop a method for efficient 3D imagecapture. New manufacturing processes will beexamined to develop cheaper, easier to producecomponents.A further target will be to allow multiple viewers tosee the effect at the same time. This would be a key requirement towards the ultimate goal of 3D LCTV.

Conclusions

A display system has been developed and described that allows simple switching between a full resolution,full colour 2D mode and a high quality, comfortable, no-glasses 3D mode. Close collaboration between research and business groups lead to a successful commercialisation of a newtechnology. The future research in 3D involves 3D image capture and development of a system withenhanced "look-around" capability.

Acknowledgements

The authors would like to acknowledge the strong support and collaboration provided by members of

- 4-

Fig. 6 3D computer monitor.

Mobile LCD Group and Communications Systems Group during the commercialisation of this technology.We would also like to thank Corporate R&D for their continuing support and belief in our work.

References

1) "New Autostereoscopic Display System", Ezra et al. SPIE Vol. 2409 February 19952) "Flat Panel Autostereoscopic Displays - Characterisation and enhancement", Woodgate et al. SPIE Vol3957, January 2000.3) "Autostereoscopic Displays - Past and Future", GB Kirby Meacham, SPIE Vol 624, 1986.4) "Autosteroscopic Display" GJ Woodgate et al. US Patent Application US60550135) "3D display Systems Hardware Research at SLE: an update", Jacobs et al. Sharp Technical JournalAugust 1999.6) Sharp Laboratories of Europe, Ltd.URL : www.sle.sharp.co.ukE-mail :3D@sharp.co.uk

（received Jan. 14, 2003）

- 5-

Evaluation of a Single User autostereoscopic Display System for 3D-TV and PC oriented applications - an example of a user centered

design cycle

Birgit Quante and Klaus Hopf Fraunhofer-Institute for Telecommunication – Heinrich-Hertz-Institut(HHI),

Einsteinufer 37, 15087 Berlin, Germany, http://www.hhi.fraunhofer.de/ Phone: +49 30 31002 677, Fax: +49 31002 213, e-mail: quante@hhi.fhg.de

Abstract We report outcomes of a human factors experiment dedicated to test a prototype single user autostereoscopic (glasses-free) display designed for 3D-TV, internet browsing, and future communication applications. This display is based on a special head-tracking lenticular-screen 3D display principle in order to provide good image quality and free positioning of the viewer. Main objective of the reported study was to identify potential technical shortcomings of the set-up in order to improve critical system parameters in the final version. Ten display experts participated in the experiment. Each subject completed 36 experimental trials. They rated 3D image materials under different viewing conditions in terms of image quality, depth impression, image artefacts, impairment and viewing comfort. Overall, our results show good image quality and depth impression ratings across the tested viewing conditions, especially in a fixed observer position. As soon as the observer moves the assessed variables slightly decrease. Concerning the viewing area our findings indicate to enhance tracking accuracy not only in the centre of viewing zone but also at the border areas in order to improve the viewers feeling of unrestricted positioning. Key words: Autostereoscopic display technology, viewer tracking, depth impression, image quality, visual comfort 1. Introduction The ATTEST 3D-TV broadcast chain At HHI an autostereoscopic single user display has been developed. The R&D work is part of the ATTEST project (Advanced Three-Dimensional Television System Technologies), funded by the European Information Society Technologies (IST) 5th framework program. Main aim of ATTEST is to design a flexible, 2D compatible and commercially feasible end-to-end 3D-TV broadcast chain. The project comprises 3D content generation, 3D coding and autostereoscopic display technology. Research into human perception and usability requirements play a central role throughout the project. Results of formative user centered evaluations support the design process such that further refinements can be incorporated in the final ATTEST prototype. The scope of application areas of the autostereoscopic display technology under development is not only limited to the entertainment section such as 3D-TV ─the target application in

ATTEST─ but also extendable to PC-oriented multimedia applications and video communications. Principle of stereoscopic and autostereoscopic displays The human eyes perceive a real-world scene from slightly different perspectives. Binocular depth perception is largely based on the evaluation of these differences. Accordingly, 3D displays create the 3D effect by redirecting two slightly different images to the right and left eye. In the case of traditional stereoscopic display techniques viewers wear special viewing aids such as anaglyphic, shutter or polarization glasses in order to direct the separated left and right view to the appropriate eye. An important issue and challenge for display designers is to separate the left and right eye view correctly. If both channels are not separated perfectly this may lead to crosstalk, perceived as ghosting (double images) and makes stereoscopic images strenuous to watch. To overcome the constraint of wearing glasses, in autostereoscopic displays the eye-addressing techniques are completely integrated in the display itself (Pastoor and Wöpking, 1997). In general, existing autostereoscopic displays allow only a fixed viewing position in order to channel the left and right eye view in the appropriate eye of the observer. Some others include a limited form of head tracking providing a multitude of fixed positions. Within ATTEST a non-intrusive video based head tracking has been developed that allows viewers to move freely in all directions (x – lateral, y – up and down, z – back and forth). Technical details of the set-up are described in the next paragraph. Technical description of the ATTEST Single User Display For the representation of stereoscopic content, a left eye view and a right eye view must be presented at virtually the same time. The ATTEST single user display consists of a 20 inch flat LC display equipped with a lenticular lens plate to separate the two views, which is mechanically adjusted according to the viewer’s position. That is: For the image separation left and right images are split into vertical stripes which are steered on the appropriate eye by means of cylindrical lenses. Depending on the viewers positioning, this lens plate is tracked in two axes with high precision. The detailed technical design of the 3D display is shown in figure 1.

actualvalue

actual value

motor (x-axis)motors (z-axis)

Processing Unit

Head Detection Unit

video (analog)

PWM

MM

A/D

PID Control System

Power Amplifier

Camera

3D

xz

Lens Plate

LC Panel

Figure1: Technical design of the test set-up

The Tracking System and Viewing Zone As can be seen in figure 1, a camera based head detection device was developed to sense the 3D head position without any user worn devices. In order to detect the eyes of a potential observer the pattern of the captured images are analyzed via a PC based frame grabber system. The processed head location data are sent to a processing unit which supplies a pulse-width modulated data stream for driving several motors. The required real time performance and precision for a spatial positioning of the lens plate is achieved by means of a custom-made PID control system. Figure 2 shows the tracked viewing area in front of the display. The area corresponds to the field that allows correct reproduction of stereoscopic images. The proportions were tested in a pilot test. Up-and-down movements do not require opto-mechanical adjustments since there is no direction selectivity of the cylindrical lenslets in the vertical direction.

700

mm

400

mm

700

mm

320 mm

display

500 mm

800 mm

viewing area

central viewing positionhead

xz

Figure 2: Viewing area Aim of the human factors experiment Aim of the subjective evaluation was to test the single user display quality and head tracking accuracy under critical viewing conditions. Aspect of particular interest was real time tracking behaviour in terms of precision, tracking range and robustness. Open questions were related to whether the tracking system would be fast enough and how it would react to different observer movements. Furthermore, we were interested in user reactions towards the limitations of the viewing zone. The tracked area should allow sufficient possibilities of observer movements in order to provide a comfortable viewing experience and a distortion-free depth impression. 2. Method Stimulus material The test materials being applied in the experiment included stereoscopic still images as well as moving stereoscopic images (produced by NHK, Japan for research purposes on stereo

video). These materials were selected with respect to particular image properties such as binocular disparity values or background attributes. Figure 3 shows the stereoscopic still images. All stills (one photo and two synthetic images) were colored images with a size of 320x399mm in portrait format and a spatial resolution of (2x512)x1280 pixels. This resulted in a spatial resolution of 512x1280 pixels per view.

(1) (2) (3)

Figure 3: Stereoscopic still images: 1. Ladder, 2. Anatomy, 3. Shuttle Figure 4 shows images of the video sequences. The stereoscopic video sequences were captured with standard TV cameras. Post processing of the image material improved the image quality and adapted the format of the sequences to suit the display in a size of 320x213 mm. Each video sequence took 15 seconds. The source materials were in the ITU-Rec. 601 format (International Telecommunication Union), with a spatial resolution of 702x480, 60 Hz interlaced (ITU, 1994).

(4) (5)

(6) Figure 4 : Example frames of the stereoscopic video sequences: 4. Balloons, 5. Flower Pot, 6. Trapeze

The Experimental Design The experimental design comprises two factors: image content (three stills and three video sequences) and observer movements (six distinct head and body movements defined in a pilot test) resulting in 36 sub trials as shown in Table 1. Viewers explored the different stimulus materials while performing different head and body movements in front of the display (see Table 1). The tracking area was highlighted by red bands. The general viewing conditions were set according to ITU-R BT. 500-10 (ITU 2000). Image content: 3D still images Ladder Anatomy Shuttle 1. Fixed position A1 A2 A3 2. upper body slow (x) A4 A5 A6 3. upper body faster (x) A7 A8 A9 4. upper body back and forth (z) A10 A11 A12 5. Head rotation slow A13 A14 A15 6. Head rotation faster A16 A17 A18 Image content: 3D video sequences Balloons Flower Pot Trapeze 1. Fixed position B1 B7 B13 2. upper body slow (x) B2 B8 B14 3. upper body faster (x) B3 B9 B15 4. upper body back and forth (z) B4 B10 B16 5. Head rotation slow B5 B11 B17 6. Head rotation faster B6 B12 B18 Table 1: The experimental conditions: A1-A18 for still images, B1-B18 for video sequences Assessment procedure The single stimulus subjective test method according to ITU-Rec. 500-10 was applied (ITU, 2000). One test stimulus was presented per trial. Subjects explored the images as long as they wanted before they gave their judgement. They were asked to rate each stimulus with respect to image quality on a continuous quality scale ranging from bad to excellent at the end of each trial. The labels “Excellent”, “Good”, “Fair”, “Poor”, “Bad” were printed alongside the scale. The same rating scale was used to measure depth impression and viewing comfort. Moreover, subjects were asked to describe the perceived image degradations and subsequently rated the perceived overall impairment on an impairment scale ranging from very annoying to imperceptible. Each participant completed 36 experimental trials (see Table 1). The entire session lasted 90 minutes, on average. Two practice trials were conducted prior to the main test, in order to familiarise the subjects to the task and the test material. After the main experiment, subjects answered several questions concerning the viewing zone and personal data. Subjects Ten display experts participated in the experiment, all were male and in the age from 28 to 55 years. They had normal or corrected to normal vision (six subjects wore glasses). Seven subjects had prior experience in performing perception experiments such as this display evaluation. Concerning the display experts' knowledge, six subjects indicated to be highly

experienced in display technology and four subjects stated to have moderate or little experience. 3. Results The assessment includes descriptive statistical analyses, a qualitative analysis of the written descriptions of image impairments and a summary of observations and comments. Overall image quality ratings For the stereoscopic still images the subjective ratings indicate a decrease of perceived image quality if the observer moves. In a fixed viewing position, the perceived image quality is on average between good and excellent. The average image quality is rated between fair and good if the observers move in the x- and z-direction, whereas the same stereoscopic images are assessed between poor and fair if the observers rotate their head. Figure 5 shows a histogram of image quality results across the six viewing conditions.

image quality

8,47

6,74

5,39

6,7

3,754,41

7,98,3

6,34

7,21

5,144,75

8,52

6,51

4,965,69

3,72,9

0

1

2

3

4

5

6

7

8

9

10

fixed position slow x-axis faster x-axis z-axis slow headrotation

faster headrotation

viewing conditions

imag

e q

ual

ity

ladder

anatomy

shuttle

Figure 5: image quality of stereoscopic static images across viewing conditions

As can be seen in figure 6, the image quality ratings vary less for stereoscopic sequences. The image quality ratings are on average between fair and good for a fixed viewing position, movements in x- and z- direction and head rotation.

image quality

7,4 7,236,64

7,61

5,45 5,526,19

6,866,41 6,57

5,92 5,615,77 5,52

4,535,01

4,13 4,19

0

1

2

3

4

5

6

7

8

9

10

fixed position slow x-axis faster x-axis z-axis slow headrotation

faster headrotation

viewing conditions

imag

e q

ual

ity

Balloons

Flower Pot

Trapeze

Figure 6: Image quality of stereoscopic sequences across viewing conditions

Except from the observer movements, the perceived image quality seems to depend on shooting conditions and scene content. The perceived image quality decreases if the stereoscopic image material contains excessive disparity values (shuttle), or if a stereoscopic video sequence consists of scene transitions with large varying disparity values (trapeze). On the other hand, observer movements may have less effect on the perceived overall image quality if image distortions are masked due to the scene content, e.g. a dark background (anatomy). Depth impression ratings The depth impression seems to suffer most from head rotations while viewing stereoscopic still images. The tracking system seems to lose track of the viewers eyes, because the interocular distance changes when rotating the head and consequently, the two stereo views can’t be displayed correctly in the right and left eye. In this case the depth impression is, on average, rated between poor and fair. The depth impression is between fair and excellent while viewing the still images in a fixed position or moving in x- and z- direction. While viewing stereoscopic sequences all observer movements resulted in a depth impression between fair and excellent. Viewing comfort ratings Stereoscopic still images with moderate disparity values (ladder, anatomy) are, on average, rated between fair and excellent. The viewing comfort decreases if the static image contains excessive disparity values (shuttle). In this case, for all observer movements the viewing comfort is, on average, rated between poor and fair. The viewing comfort experienced with stereoscopic sequences is rated between fair and excellent, for all observer movements. Although the effect is less apparent in comparison to the still images, a sequence containing excessive disparity values (trapeze) seems less comfortable to view than sequences with moderate disparity values (balloons, flower pot). Overall, observer movements seem to affect the visual comfort ratings less than the shooting conditions and/or scene content. Descriptions of perceived image impairments The written descriptions, observations and comments of the participants gave important details in order to facilitate the further development of the tracking system. No severe image distortions are reported in a fixed position. A slow upper body movement of the viewer leads to crosstalk, perceptible as shadows or dark stripes, and flicker, particularly visible at border areas. The impairments increase when viewers perform a comparatively faster body movement. Movements back and forth the z-axis show that the tracking system operates partly incorrect. Subjects report 3D breakdowns while moving forward. According to observations of the participants, the back and forth motion seems to be not as impairment sensitive as the parallel motion along the x- axis, even though flicker, and black stripes are perceptible, mainly at the border of the screen. Slow and faster head rotations lead to stronger image impairments. The depth impression breaks down, crosstalk becomes visible as black stripes and shadows. All impairments are experienced as unpleasant and strenuous for the eyes. Additionally, acoustical noise caused by the driving system of the lens plate is mentioned across all dynamical viewing conditions and is experienced as an impairment of the visual sensation.

Overall impairment ratings The overall impairment ratings reflect, how annoying the reported artifacts were judged by the participants. The degree of annoyance of perceived impairments depends on the body movements and the scene content. In a fixed position the stereoscopic still images were judged, on average, between slightly annoying and perceptible but not annoying. Body movements in the x- and z-direction as well as head rotations result in average impairment ratings between annoying and slightly annoying. The impairment ratings for stereoscopic sequences were slightly different. Observers seem to be less sensitive to impairments in stereoscopic sequences while moving in the x- or z- direction. In this case the observers judge the overall image impairment similar as in a fixed position, which is, on average, between slightly annoying and perceptible but not annoying. The impairment ratings of stereoscopic static images and sequences are also affected by shooting conditions and scene content. Again excessive disparity values decrease the impairment ratings and observer movements seem to have less effect on the impairment judgments if impairments are masked due to the scene content (anatomy). 4. Conclusions The findings have the following practical implications for the further development of the tracking system: The subjective results indicate that especially fast lateral observer movements in front of the display (x-direction) and head rotations can lead to annoying artifacts. The viewing experience can be significantly enhanced by increasing the tracking speed, and improving the tracking accuracy such that the system can keep track of head rotations, faster observer movements in the x- and z-direction and observer movements at the borders of the viewing area. Furthermore, the test showed that acoustical noise produced by the tracking system should be reduced. This may increase viewing pleasure. Some technical solutions overcoming these deficiencies have already been incorporated in a follow-up display version. Especially the real-time performance and the tracking accuracy have been improved. A revised tracking electronic avoids the occurrence of disturbing acoustical noise. Moreover, our experimental results show that excessive disparity values should be avoided. Otherwise observers may experience visual discomfort or difficulties to fuse the left and right- eye image. In addition, scene cuts resulting in a transition from large to small disparity values or vice versa is unpleasant to the observers and should be limited as much as possible. References

ITU (2000). Methodology for the subjective assessment of the quality of television pictures. Recommendation BT.500-10. International Telecommunication Union. ITU (1994). Encoding parameters of digital television for studios. Recommendation BT.601. International Telecommunication Union. Pastoor, S. and Wöpking, M (1997). 3D Displays: A review of current technologies. Displays, 17:100-110.