Page 1 of 10

Designing and Building the PIT:

a Head-Tracked Stereo Workspace for Two Users

Kevin Arthur, Timothy Preston, Russell M. Taylor II,

Frederick P. Brooks, Jr., Mary C. Whitton, William V. Wright

Department of Computer ScienceUniversity of North Carolina at Chapel HillSitterson Hall , Chapel Hill , NC 27599-3175

Additional information is available at:http://www.cs.unc.edu/Research/graphics/GRIP/PIT.html

Abstract

The PIT (“Protein Interactive Theater” ) is a dual-screen, stereo display system and workspace designedspecifically for two-person, seated, local collaboration. Each user has a correct head-tracked stereo viewof a shared virtual space containing a 3D model under study. The model appears to occupy the samelocation in lab space for each user, allowing them to augment their verbal communication with gestures.This paper describes our motivation and goals in designing the PIT workspace, the design and fabricationof the hardware and software components, and initial user testing.

Motivation and design goals

The PIT display system is the most recent of a series of displays used by the GRIP molecular graphicsproject at UNC [Brooks et al., 1990]. While we intend for the PIT to be generally applicable to otherapplication domains, our initial use is in molecular graphics applications such as protein fitting. The PITdesign is motivated by observations gathered over the years by collaborating with biochemists to developmolecular graphics systems. The following were our primary goals in developing the PIT.

High quality 3D display. To provide strong 3D depth cues, we employ the technique of head-trackedstereo display. For each user, a stereo image is displayed and updated in real time according to theperspective projection determined by the positions of the user’s eyes. The stereo and motion parallax cuesgive users the illusion of a stable 3D scene located in front of them and fixed in laboratory space. Theusers wear LCD shutter glasses and tracking sensors. High-resolution images (four images per frame,each rendered at 1280 × 492 pixels) are displayed on two large rear-projection screens oriented at 90degrees to each other, with one screen corresponding to each user. Rendering is performed on a SiliconGraphics Onyx workstation with InfiniteReality™ graphics. We allow for decoupling the application’sdisplay and simulation loops so that they run as separate processes, in order to maintain a high displayupdate rate that is independent of the complexity of computations the application may be performing. Thedisplay screens may also be oriented at 120 degrees to each other for applications desiring a high-resolution, wide field-of-view panoramic view for a single user.

Including a second user. Over the years, we have observed that our users, who are biochemists andother scientists, quite often work in pairs to conduct their experiments. To allow close collaborationbetween the two users we wanted to provide the second user with a view equal in quality and realism to

Page 2 of 10

that given to the first user. In our past work we found that using a single head-mounted display for thefirst user and a monitor view for the second user was unsatisfactory; the monitor viewer found it difficultto understand the scene. The situation is improved somewhat with large-screen head-tracked displayssuch as the CAVE™ [Cruz-Neira et al., 1993], but, because the perspective images are correct for onlythe first user, the second user will still see an inaccurate view unless he or she makes an effort to followthe other’s movements. More seriously, the second user’s view jumps around when the first user moves.This limits that user’s ability to interact with the virtual scene and to collaborate actively with the otheruser. For the PIT display, we decided to provide two head-tracked stereo views, so that each user has anindependent view of the virtual scene under study, and may optionally apply their own custom viewingand model manipulations.

To create two independent stereo views without perceptible flicker on a single screen using LCD shutterglasses requires a refresh rate higher than that available with current commercially available projectorsand shutter glasses. This drove our decision to have two projectors and opened the question of whetherthe two screens should be side-by-side or in some other configuration. We decided to use two projectionscreens oriented at 90° to each other, and to display one user’s view on each screen. The display givesthe illusion of a stable 3D structure fixed in lab space in front of the two seated users. Because they seethe displayed structure in the same position in physical space, they may physically point to the sameplaces in the model with their hands. The users may work with their two instances of the virtual sceneregistered in this way, or they may alter the view according to their own preference. For example, atoggle button is provided for either user to rotate his view by 90° so that both users are viewing the modelalong the same axis, as if they were both seated at the same virtual position, looking in the same direction.In this case they use virtual icons to point to the scene.

We have optimized the display for local collaboration, where the two users are seated next to one anotherin the same physical place, rather than for distance or tele-collaboration. Additional users can standbehind the two users and view the screens with stereo glasses (without head tracking).

Figure 1. Two users interacting in the PIT.

Page 3 of 10

Access to common devices. Over the years, our chemist collaborators have preferred to use multiplephysical input devices such as dials and buttons rather than use modal or overloaded 2D interfaces. Wewanted to provide a 3D display that would allow the user access to a lab notebook, a keyboard, a smalldisplay for text and GUI elements, physical dials and buttons, and other common devices. This led usaway from opaque head-mounted displays, and towards head-tracked stereo in an environment thatincluded space for a table, portable PC’s, and other equipment.

Input devices may be shared or user-specific. Our intention is to provide devices that are appropriate andnatural for the tasks that need to be performed without restricting the users to particular modes of 3D or2D interaction. Each user has a laptop PC for accessing 2D graphical user interface elements, for readingexperiment plans, and for making lab notebook entries based on observations.

Two tracked 6 degree-of-freedom handheld controllers provide pointing, picking, and other scenemanipulations. Physical dials and buttons are used for shared interactions such as repositioning thedisplayed structure and controlling PIT display modes. A SensAble Technologies PHANToM arm isavailable for force-feedback display.

Table-top workspace within arm’s reach. We found in our earlier experiments using head-mounteddisplays for molecular graphics applications that users did not choose to fly around or walk around“room-fil ling” molecules. The PIT provides a more comfortable environment where the user can sitdown and easily reach and manipulate the model under study. Its size can be made similar to the 2 cm/Åscale of the familiar brass models that chemists have historically used.

Related work

Systems employing head-tracked stereo on multiple large screens for a single tracked user, such as theCAVE, have been actively used in recent years [Cruz-Neira et al., 1993; Deering, 1993]. Single-screenhead-tracked “workbench” style displays have also been widely reported [Agrawala et al., 1997;Czernuszenko et al, 1997; Grant et al., 1998]. Agrawala and others [1997] extended the ResponsiveWorkbench display to allow for two tracked users viewing a single screen. They made custommodifications to their shutter glasses and video system to multiplex the four views on a single projector.Extensions to CAVE systems to allow for multiple tracked users are reported in on-line CAVEdocumentation maintained by the Electronic Visualization Laboratory at UIC. Others have used head-mounted displays for multi -user immersive virtual reality or augmented reality systems [Blanchard, 1990;Szalavari, 1998].

Hardware components

Figure 2 shows a block diagram of the hardware components of the PIT workspace.

Image generation. We use a Sili con Graphics Onyx2 workstation with Infinite Reality graphics togenerate two video output channels of 1280 × 1024 pixels each. The video is displayed in a rear-projection arrangement using two AmPro 3600 projectors positioned to give their smallest focussedimage size. A StereoGraphics CrystalEyes box doubles the vertical refresh rate of the projectors toprovide stereo images synchronized to LCD shutter glasses. Each eye’s view is rendered into a region inthe frame buffer of 1280 × 492 pixels [StereoGraphics, 1997].

Page 4 of 10

Figure 2. PIT hardware components.

In order to do geometrically correct head-tracked stereo, we must accurately measure the physicalpositions of the screen corners. We have done this manually, and have also used a theodolite surveyingdevice to obtain more accurate measurements [Grant et al., 1998]. We have found that we also need toretune the projectors frequently to maintain alignment of the displayed video with the physical screencorners and minimize other distortions. We also require accurate measurements of the tracker sensor-to-head position transformation, and of the users’ inter-pupil lary distance (IPD). We allow the users toadjust the software’s IPD value at runtime using dials.

Screen construction. Figures 3 and 4 show the PIT screen material and frame used for our initialimplementation of the system. The screen surfaces are each 3 ft. high by 4 ft. wide. We chose a screensize that would provide an adequate field of view for a seated user with line of sight intersecting thecenter of the screen, as constrained by the smallest focussed image we could get from the projectors.

Screen material. We chose a vinyl screen material according to its brightness and gain properties. Wewished to maximize image brightness across the range of normal head positions, without brightnessfalloff being noticeable, and without serious inter-reflection between the two screens. A single sheet isused across the two screen areas and is tensioned to make both surfaces vertical and flat. A Naugahydefabric border is stitched around the outside of the screen, and behind the corner seam. The bordercontains grommet holes for tying the screen to the external frame, as well as loops through the material tocontain a rod to help keep the screen material flat. The material is gathered at the corner, between the two

Page 5 of 10

sides of the screen, and re-sealed. A loop in the border behind it is used for support. We desired that theinside seam at the corner be straight and not allow light leakage across the two screens. We also requiredthat it be of minimal thickness so as to give the impression of a continuous display surface with no visibleline at the seam when the screens are in a 120° panoramic display configuration. In our first screen thecorner seam was heat-sealed. In our second, the seam was stitched. Stitching produced a more durableseam that was less susceptible to tearing, but we found its visual appearance to be worse. The stitchedseam wasn’ t as straight as the heat-sealed seam, and light leakage was visible at the holes created by thestitching.

Screen frame and projectors. We required that the frame that holds the screens be made of a non-ferrousmaterial so that its presence would not interfere with the electromagnetic tracker. We chose to use 4-inchPVC pipe for this reason, and for its light weight, rigidity and convenience. Holes are spaced along theoutside of the frame for attaching the screen with adjustable elastic bungee cords passing through thegrommets. The projectors are fixed to the floor behind the screens on mobile wooden frames that haveadjustable bolts for leveling the projectors and adjusting their height.

Screen hinge. The corner of the screen frame is hinged to allow the screens to be placed at angles of 90°and 120°. We fix the screens and projectors in place using pins through holes in the screen frame,projector mounts, and the lab floor. The left screen and projector are fixed at all times and the rightscreen and projector can be easily moved between the two positions and secured to the floor with the pins.To maintain a straight seam at the corner between the two screens we place a vertical glass rod through aloop in the screen border material.

Lessons learned. Were we building the PIT screens again today, we would construct the frame from 3-inch T-section fiberglass impregnated extrusions, for looks and ease of manufacture. This material wouldsimpli fy right angle joints and holes for the bungee cord. The T-section also offers the possibility of aless complex hinge assembly. The glass rod used for tensioning the screen could be made of less fragilematerial, perhaps as simple as a 1”×4” board. Both rod and bungee cord are needed for tensioning thescreen. We find that three pieces of cord on each side are sufficient and that surprisingly little tension isrequired to ensure a flat screen.

Servers for per ipheral devices. We use a Dell 200 Pentium Pro PC running the Linux operating systemas a server for multiple input devices. The PC runs a server process using the locally developed VRPNlibrary (“Virtual Reality Peripheral Network,” described athttp://www.cs.unc.edu/Research/nano/manual/vrpn/). It relays data from trackers, buttons, andother input devices over the local ethernet. Additional PC servers provide optional access to aPHANToM™ force-feedback arm, the UNC ceiling tracker, or a sound server for audio output. We alsouse an SGI button and dial box connected directly to the serial port of an SGI Onyx.

Tracking. For head and hand tracking we have used both a commercial electromagnetic tracker, and alocally developed optical tracker. We use the VRPN server PC to drive an Ascension Flock of Birdstracker with Extended Range Transmitter and four sensor units (two for head tracking and two for handtracking). The server reads records from the four tracker sensors in parallel at an average rate of 100 Hz.We have observed distortion in tracker readings across the PIT working volume, and intend to applycorrection tables using methods developed by Livingston and State [1997].

Page 6 of 10

Figure 3. The PIT workspace in its 120°° configuration.

Figure 4. Detail of screen corner , showing frame hinge, screen mater ial, and glass rod for support.

Page 7 of 10

For more accurate and faster head tracking we have also used the UNC ceiling tracker [Ward et al., 1992;Welch and Bishop, 1997]. We attach a HiBall optical sensor to each pair of stereo glasses (see Figure 5).The sensor views active infrared LEDs on the ceiling above the PIT and processes these views to provideposition and orientation records at approximately 1 kHz. The ceiling tracker’s latency is user-configurable from approximately 1 ms to 100 ms, depending on the level of filtering. We expect that alevel of filtering resulting in approximately 30 ms of tracker latency will be adequate. In our initial tests,the infrared LEDs on the ceil ing occasionally interfered with the infrared synchronization signal sent tothe LCD shutter glasses, triggering the shutters. To solve this problem we plan to investigate methods tofilter or shield the glasses from seeing the ceiling signal.

Figure 5. StereoGraphics shutter glasses with the UNC HiBall tracker .

Software structure

The PIT software library has been designed to facil itate fast conversion of existing applications, and toallow for the same application code to run on different display devices, with either one or two users.Figure 6 shows the basic software structure. The core PIT code handles all functions required to displayhead-tracked stereo images and to allow a small set of viewing manipulations (primarily translate, rotate,scale, “grab world,” and adjust IPD). The user performs these manipulations using dials, buttons, handcontrollers, and GUI control panels displayed on the laptop computers using the X and Tcl/Tk libraries.

Applications link with the PIT API library, which is written in C++ and uses OpenGL and the UNC Vlib,VRPN, and other supporting libraries. We are testing methods for decoupling the simulation and displayprocesses, similar to methods described by Pausch et al. [1994] and Shaw et al. [1992]. Our intent is thatat runtime, PIT applications will be split into two main threads of execution: a model simulation loop,executing application-defined code, and a display loop, executing PIT library code and application-defined callback functions. An application must supply a display callback function that draws the modelin world space using OpenGL. The PIT library sets the necessary viewing projections prior to calling thedisplay function four times per frame (once per user, per eye). The application’s display callback ispassed information about which view is being drawn (left or right user, and left or right eye), and may usethose parameters to customize its model display. In the simplest mode of operation, the display callbackignores these parameters and draws the same geometry for both users. The PIT will also call otheroptional application-specified callbacks: a model update callback to coordinate data sharing between theapplication’s simulation and display code, and a graphics initialization callback, called once duringinitialization in the display execution thread.

Page 8 of 10

Figure 6. PIT software structure. Blocks indicate modules of source code. Ar rows indicate datapassed between the code modules.

The PIT API library also includes functions to modify display parameters and to query the PIT for datafrom the trackers, hand controllers, and other input devices to perform more complicated interactions suchas selecting and moving individual model elements. To provide 3D picking, the application queries thePIT API for the status of the hand controller’s buttons, and for the hand controller’s position andorientation with respect to the application’s model space.

Evaluation and next steps

We have ported the crystallography application CORWIN (for “coupled reciprocal windows”) formembers of the Biochemistry department at UNC to use in the PIT. CORWIN allows users to manipulatemolecules to perform protein-fitting tasks. The proteins can be viewed in real-space or reciprocal-space(Fourier-space) representations. Users manipulate a protein model by moving individual bond residuesand rotating about individual bonds to make the model fit an electron density map. The basic PIT displayprovides controls for changing viewing and world-space transformations. The CORWIN application codehandles application-specific interaction, such as selecting and manipulating individual atoms andresidues, and provides other interactions through a GUI interface on the laptop computers. We plan totest the effectiveness of the PIT version of CORWIN by having UNC biochemists use the system on realdata. We plan to gather additional human-factors results by extracting from CORWIN a simple 3D fittingtask and conducting controlled experiments to measure performance under different viewing conditionsand using different manipulation techniques.

We have also created general polygonal model viewers for the PIT for other applications, such asarchitectural walkthrough or mechanical part design, and are working with outside users to port other

Simulation

Code

Draw

World

Update

Model

API

Query

Functions

Update

View

Handle

Devices

Simulation Process Display Process

Application CodePIT Library Code

Page 9 of 10

molecular graphics applications to the PIT.

For additional information, including a list of material specifications and sources, and additional imagesand video sequences of the PIT, please see the project web page athttp://www.cs.unc.edu/Research/graphics/GRIP/PIT.html.

Acknowledgments

We thank the following people for their contributions to this work: Sumedh Barde, David Harrison, JohnThomas, Ruigang Yang, Hans Weber, Brent Insko, Michael Meehan, Kurtis Keller, Andy Wilson, andmembers of the UNC-CH Tracker and nanoManipulator projects. The PIT screen design was done incollaboration with Fakespace, Inc.

Work on the PIT is funded by the NIH Division of Research Resources grant number RR02170 with othersignificant support from Intel Corporation.

References

Agrawala, Maneesh, Andrew C. Beers, Bernd Fröhlich, and Pat Hanrahan. “The two-user responsiveworkbench: Support for collaboration through individual views of a shared space.” Proceedings ofSIGGRAPH 97. Computer Graphics Proceedings, Annual Conference Series, 1996, ACM SIGGRAPH,pp.327-332.

Blanchard, Chuck, Scott Burgess, Young Harvill, Jaron Lanier, Ann Lasko, Mark Oberman, and MichaelTeitel. “Reality built for two: A virtual reality tool.” Proceedings of the 1990 Symposium on Interactive3D Graphics, Computer Graphics Vol. 24, No. 2, March 1990, pp. 35-36.

Brooks, F. P., Jr., M. Ouh-Young, J. J. Batter, and P. J. Kilpatrick. “Project GROPE: Haptic displays forscientific visualization,” Proceedings of SIGGRAPH 90, Computer Graphics, August 1990, Dallas, TX,pp. 177-185.

Cruz-Neira, Carolina, Daniel J. Sandin, and Thomas A. DeFanti. “Surround-screen projection-basedvirtual reality: The design and implementation of the CAVE.” Proceedings of SIGGRAPH 93. ComputerGraphics Proceedings, Annual Conference Series, 1993, ACM SIGGRAPH, New York, 1993, pp. 135-142.

Czernuszenko, M. D. Pape, D. Sandin, T. DeFanti, G. Dawe, and M. Brown, “The ImmersaDesk andInfinity Wall Projection-Based Virtual Reality Displays,” Computer Graphics , Vol. 31 Number 2, May1997.

Deering, Michael. “Making virtual reality more real: Experience with the virtual portal.” Proceedings ofGraphics Interface 93, Canadian Information Processing Society. pp. 219-226.

Grant, Brian, Aron Helser and Russell M. Taylor II, “Adding force display to a stereoscopic head-trackedprojection display.” Proceedings of VRAIS '98, Atlanta, Georgia, 1998.

Livingston, Mark A. and Andrei State. “Magnetic tracker calibration for improved augmented realityregistration.” In Presence: Teleoperators and Virtual Environments, MIT Press, Vol. 6, No. 5, October,1997, pp. 532-546.

Pausch, Randy, Matthew Conway, Robert DeLine, Rich Gossweiler, Steve Maile, Jonathan Ashton andRichard Stoakley “Alice & DIVER: A Software Architecture for the Rapid Prototyping of VirtualEnvironments,” Course notes for SIGGRAPH '94 course, “ Programming Virtual Worlds,” 1994.

Page 10 of 10

Shaw, Chris, Jiandong Liang, Mark Green, and Yunqi Sun. “The decoupled simulation model for virtualreality systems.” Proceedings of the CHI 92 Conference on Human Factors in Computing Systems, 1992,pp. 321-328.

StereoGraphics Corporation. 1997. StereoGraphics developers’ handbook. StereoGraphics Corporation,San Rafael, CA. Available at http://www.stereographics.com.

Szalavari, Zs., D. Schmalstieg, A. Fuhrmann, and M. Gervautz. “Studierstube - An environment forcollaboration in augmented reality.” to appear in Virtual Reality Journal, 1998. Available athttp://www.cg.tuwien.ac.at/research/vr/studierstube/

Ward, Mark, Ronald Azuma, Robert Bennett, Stefan Gottschalk, and Henry Fuchs. “A demonstratedoptical tracker with scalable work area for head-mounted display systems.” Proceedings of 1992Symposium on Interactive 3D Graphics, Computer Graphics, Vol 25, No. 2, pp. 43-52.

Welch, Gregory and Gary Bishop. “SCAAT: Incremental tracking with incomplete information,”Proceedings of SIGGRAPH 97. Computer Graphics Proceedings, Annual Conference Series, 1997, pp.333-344.