Situational Visualization

David M. Krum, William Ribarsky, Christopher D. Shaw∗, Larry Hodges, and Nickolas FaustGVU Center, Georgia Institute of Technology,

801 Atlantic Drive, Atlanta, Georgia, 30332-0280, USA{dkrum,ribarsky,cdshaw,hodges}@cc.gatech.edu, nick.faust@gtri.gatech.edu

Phone: 404-894-6328, Fax: 404-894-2970

ABSTRACTIn this paper, we introduce a new style of visualization calledSituational Visualization, in which the user of a robust, mo-bile visualization system uses mobile computing resources toenhance the experience and understanding of the surround-ing world. Additionally, a Situational Visualization systemallows the user to add to the visualization and any under-lying simulation by inputting the user’s observations of thephenomena of interest, thus improving the quality of visu-alization for the user and for any other users that may beconnected to the same database. Situational Visualizationallows many users to collaborate on a common set of datawith real-time acquisition and insertion of data. In this pa-per, we present a Situational Visualization system we aredeveloping called Mobile VGIS, and present two sample ap-plications of Situational Visualization.

KeywordsMobile users and collaborators, dynamic databases, syn-chronized databases, real-time acquisition and insertion ofdata, location-specific services, location and time-specificuser input.

1. INTRODUCTIONMany tasks require a detailed knowledge of the local en-

vironment as well as an awareness of rapidly changing andinteracting events. This awareness is termed situationalawareness or situation awareness. It can be formally definedas:

“the perception of the elements in the environ-ment within a volume of time and space, thecomprehension of their meaning and the projec-tion of their status in the near future.” [5]

This type of spatio-temporal knowledge and awareness isimportant in many tasks, including aircraft piloting, mili-

∗Contact author

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.VRST’01,November 15-17, 2001, Banff, Alberta, Canada.Copyright 2001 ACM 1-58113-427-4/01/0011 ...$5.00.

tary maneuvers, law enforcement, fire fighting, and even thedaily automobile commute.

We are defining a set of techniques to provide computer-based support for situational awareness in augmented andvirtual reality systems. We wish to bring large, diverse setsof data into such systems, and manage the limited visual-ization resources of the system and the limited perceptualand cognitive resources of the user. We describe this bodyof techniques as Situational Visualization. Situational Vi-sualizations are responsive to a user’s location in space andtime, as well as her actions and situation. They integraterealtime and static data from a variety of sources and allowusers to view time variant events and objects and projecttrends into the future.

The difference between such Situational Visualizations andother types of visualizations (e.g. information visualizationsand scientific visualizations) is that the user is situated inthe environment and experiencing the situation firsthandand in real time. These elements of Situational Visualiza-tion become especially significant for mobile users who canmove from place-to-place and can encounter new and unex-pected situations. The user must be able to receive timelyinformation associated with the situations encountered.

From the user’s perspective, this implies that the systemmust be:

Unobtrusive: The user must be able to attend to the en-vironment without being distracted by the system in-terface.

Context-Aware: The system must be aware of the user’slocation and circumstances.

Time-Aware: The system must help the user anticipateevents, objects, and trends.

Priority-Aware: The system must help the user prioritizeand attend to the most important events and objects.

Users can also act as data collectors, having a hand in theconstruction of the database. It is now possible for users tocarry a variety of sensors (cameras, various environmentalsensors, etc.) as they move about. The user may activelydirect these sensors, or allow them to operate automatically,providing a rich stream of information to be absorbed intothe database. Furthermore, a user may develop higher levelinterpretations and understanding about the environment.The user can make precise observations and annotationsbased on what she sees. To help capture and share thatinformation, the system must support annotation (the capa-bility to annotate and edit the database) and collaboration

(the ability to communicate and work with remotely locatedusers).

These high level capabilities and attributes translate intothe following system design requirements:

Real-time Data Acquisition: To update the situationalpicture, the latest information must be placed in thedatabase. Real-time acquisition and insertion of datamust be possible from a variety of sources.

Dynamic Data: Because the database is continually beingupdated with spatial data, dynamic data structuresmust be capable of changing on-the-fly as pertinentinformation is received.

Synchronized Data Structures: Synchronized data struc-tures are required, since there may be multiple, net-worked databases and multiple collection points.

User Geo-Location: A key situational variable is user lo-cation, so that users needing situational informationcan see who contributed it, and where it comes from.Users must be geo-located, and must be able to in-sert their orientations and movement vectors into thedatabase. The database will be enriched with these an-notations and, in turn, will permit more accurate userlocations and orientations to be derived using sensorfusion techniques.

Global Geospatial Structure: There must be a global,geospatial structure capable of handling all types ofgeospatial data from all over the Earth. The databasemust be able to handle data from any location, andmust therefore be robust enough to address any loca-tion at any reasonable resolution.

Time and Location-specific Services: This presumes adatabase structure appropriate to the location-specificand temporal nature of the user’s situation and work.With this structure, one can have location-specific ser-vices, which also depend on the time at which relevantevents occur, on what the user is doing, and on whereshe is and where she is going.

Situational Visualization shares many properties of Vir-tual Environments in that the user is immersed in a highlyinteractive 3D virtual environment. Situational Visualiza-tion is also an Augmented Reality application in that one issimultaneously navigating the real world and a correspond-ing annotated 3D virtual replica of the world. There are is-sues of view registration in terms of aligning a virtual viewin the database with the real view seen by the user. Al-though we do not do so in this paper, one could carry theseissues all the way to the simultaneous alignment of virtualgeolocated objects and real ones in the same scene, as is usu-ally done in Augmented Reality. Because of the overlap, wetreat the issues of immersion and interactivity in SituationalVisualization from the standpoint of Virtual Reality.

This paper will examine the design of data and synchro-nization structures, as well as collaboration and awarenessfeatures to support Situational Visualization. We also ex-amine some example Situational Visualization applications.

2. RELATED WORKSome aspects of Situational Visualization are seen in sys-

tems that present information based on the user’s location.A number of tour guide and map navigation applicationshave demonstrated location specific information for smallgeographical regions. Georgia Tech’s Cyberguide [1] is anindoor and outdoor tour guide based on the Apple Newtonhandheld computer. The indoor version presents informa-tion about various demos using an array of infrared trans-mitters for pinpointing location within a large lab. Theoutdoor version uses GPS to present information about lo-cal establishments in Atlanta. A travel diary is created andused to suggest other locations that a user might be inter-ested in visiting.

The Touring Machine [7] is a wearable augmented re-ality system that presents multimedia information aboutthe Columbia University campus with access to departmentweb pages. It combines see-through augmented reality withtablet input and output. The ”map-in-the-hat” system de-veloped by Thomas, et al. [17] is a navigational aid system.It uses a see- through head mounted display to display way-points and a compass for walking direction. The main focusof ”map- in-the-hat” is to guide the users in an orienteeringtask from one point to another.

Nokia’s Context Compass [16] is a wearable computer ap-plication that presents a see-through 2D overview map ofan urban area with roads, buildings, and virtual objects.Virtual objects can represent buildings, other users, or mes-sages, photographs, or waypoints, for example. Virtual ob-jects are displayed on a linear compass and selected by facingthe object, which centers the object in the compass. Eachvirtual object has an associated URL with a web page, im-age, audio, or video.

Our focus is to extend the notion of providing navigationand informational aids with real time data feeds and largedata sets covering large (whole Earth) geographic regions.Various projects for spatialized hypertext such as World-Board [15] and the Real-World Wide Web [11] will utilizelarge information data sets over worldwide geographic re-gions, but our system provides an emphasis on situationalawareness and visualization, requiring a more powerful setof geo-located objects and services than hypertext and otherweb media.

The collaborative and information annotation aspects ofSituational Visualization are related to work at Columbia onindoor and outdoor interfaces for MARS [9]. This follow-onwork to the Touring Machine adds a collaborative interfaceto their mobile augmented reality system that allows station-ary remote experts to monitor and assist mobile users. Theusers can create virtual objects and annotations to pointout interesting objects and direct attention. Another ap-plication that demonstrates context tagged annotations isStickePad on a GPS equipped PalmPilot, which allows usersto create geolocated and time-stamped “stick-e notes” forecology fieldwork [13].

Another project that is closely related to our descriptionof Situational Visualization is the Battlefield AugmentedReality System (BARS) developed at the Naval ResearchLaboratory [10]. It also combines a detailed map of the userssurroundings overlaid with a context and situation awareinformation display. This AR system presents data aboutthe soldier’s environment such as sniper positions or navi-gational waypoints based on the importance of each of the

soldier’s current tasks and needs. However, it currently isdesigned to address the needs of the dismounted soldier inurban landscapes. We are developing a more general frame-work of services to assist a wider variety of users.

Our system is built on our global geospatial data organi-zation and visualization system, VGIS (Virtual GeographicInformation System) [3, 4, 6]. VGIS treats the terrain as asingle, connected surface for rendering using a continuous,view-dependent Level-Of-Detail (LOD) representation. Al-though there are similar approaches in the literature, VGISis the first system to support real-time view-dependent, pixel-accurate multiresolution rendering of high resolution globaldatasets.

The VGIS system has the following capabilities:

• It accepts and integrates diverse types of geospatialdata into a global framework.

• It is scalable to very large data stores and to dis-tributed environments.

• It provides an interactive visualization framework sothat the data store can be navigated as one coherentwhole.

• It supports discovery of new information via naviga-tion in context with unfolding detail.

Our approach starts with a hierarchical structure for op-timal interactivity for data exploration. We use a ”forestof quadtrees” with nested coordinate systems for handlingglobal-scale terrain data and permitting timely paging ofcollections of objects in support of real-time navigation. Wehave found that one can effectively employ this hierarchi-cal structure for a wide range of geospatial data as long asmethods adapted to the specific type of data (e.g., terrainversus buildings) are applied at the lower (detailed) levels.

We have built a quadtree and shared cache data structurethat together constitute the basic components of our globaldata model. The global structure is divided into 32 zones,each 45 degrees × 45 degrees on the Earth’s surface. Eachzone has its own quadtree and all quadtrees are linked sothat objects or terrain crossing quadrant boundaries can berendered efficiently. On top of this structure we have added256 x 256 coordinate systems within each quadtree – onecoordinate system per node, eight levels below each rootnode – resulting in a 1 mm worst case precision.

3. DYNAMIC AND SYNCHRONIZED DATASTRUCTURES

An attribute of Situational Visualization is that data canbe received all the time, either in discrete or streaming fash-ion. Our interactivity requirement means that these datamust be readily available when received. One data struc-ture that can effectively accommodate these attributes isan appropriately tailored universal hierarchy. In the caseof geospatial data, the choice is a global forest of quadtrees,which we have shown can be tailored to handle a wide varietyof data including terrain (from low to high resolution), pho-totextures and maps, GIS information, 3D buildings, andeven time-dependent, volume data (e.g., weather) [3, 4, 6,14]. The forest of quadtrees is used at the top level for eachof these types of data and although each has its specificstructure at higher levels of detail, the specific structures

are quadnode aligned. The quadnode alignment permitsthe handling of dynamic data in the same general fashion,regardless of the type. As the data are received they areorganized to fit the quadtree hierarchy. This entails build-ing links to data chunks at appropriate levels of the hierar-chy and updating quadnodes with the appropriate averages,ranges, error values (for LOD management), and links. Forstreaming data this process may exceed the allotted timebudget, in which case the dynamic data are organized ap-proximately in the hierarchy. The system can then completethe organization during idle times.

For Situational Visualization, data must also be in ”syn-chronized data structures” in the sense that any user maybe collecting data and putting it in her own database, whichmust then be synchronized with databases of her peers.With geospatial data, one of the peers may be quite a bitlarger than the others (e.g., a data server), so the networkingstructure and overall data organization will have aspects ofboth peer-to-peer and client-server structures. Our general,global structure allows us to think in terms of a completeglobal virtual hierarchy. This is the hierarchy for the wholeEarth down to very high resolution (currently 1mm). Noone, not even the servers, has a complete set of data; ev-eryone has some piece of it. But if anyone receives a newpiece of data for any place on Earth, he knows where toput it in the hierarchy and it is a unique location. The treestructure is unique but properties such as links to individ-ual blocks of terrain are not. Thus one peer may have aset of elevation posts at a quadnode that differs from thoseat the same node of another peer. Furthermore, these nodecontents may change over time.

Figure 1: Distributed databases. The two moblesystems on the right have overlapping areas of in-terest.

The distributed set of databases needs a synchronizationmechanism. (see Figure 1.) If a mobile system, for example,moves into a new area and finds it doesn’t have the datait needs, it looks for these data amongst its peers (see Fig-ure 1). On the other hand, it can also be collecting data thatit wants to transmit to its neighbors. To support efficientsynchronization we define 3 types of peers, each with differ-ent attributes: neighbors, servers, and collaborators. Neigh-bors are the set of all peers who may communicate; serversare neighbors with large repositories of data at defined lo-cations, and collaborators are neighbors who are in activecommunication. Note that these groups are not mutually

exclusive and could also change over time. The servers arethe first systems queried for data in their areas of interest,although the rest of the neighbors could also be queried forhigh priority requests. Collaborators directly share annota-tions or acquired data and also such things as interactionsand viewpoints. Servers also get any updates for their ar-eas of interest. As long as a peer gets a fresh download ofdata from the servers when it enters the collection of neigh-bors (but only for those regions in its area of interest thatare marked newer on the appropriate server), it will be syn-chronized. Thus the synchronization is restricted to areasof interest for all peers. The meaning of areas of interest formobile users will be discussed further below.

For efficiency the querying peer can construct an exact ad-dress for the data block it needs. This is one of the attributesof the global virtual hierarchy. For example, a terrain foot-print can be addressed by first obtaining the zone it is in (i.e.,the quadtree from the forest of quadtrees), then finding theleaf cell at the quadlevel appropriate to the footprint size.These are both bin sort operations and quite fast. A uniqueaddress can then be constructed. This querying procedureneeds to take account of the distributed environment. Thusthe footprint covers not only the scene in direct view butalso areas likely to be covered in further navigation. Thisnavigational footprint (and the resolution needed) will bedifferent depending on, for example, whether a mobile useris walking, in a vehicle, or in an airplane. The system con-structs two navigational footprints around the user’s currentlocation (see Figure 2). The outer one defines the region tobe retrieved from the server, if necessary. However, this onlyhappens if the user moves outside the inner footprint in herexplorations. This arrangement insures that thrashing willbe minimized as the user moves back and forth; once the usermoves outside the inner footprint, a new set of footprints isset up centered around the new current position (Figure 2).This arrangement is similar to the one used in [12], exceptthat now it is done in the context of a complete global hier-archy. The footprint address and resolution needed is thentransmitted to, say, the relevant server. The server findsthe appropriate parent node and then does a breadth firsttraversal across child nodes at the appropriate resolution. Alist of data blocks is constructed for transmission (for exam-ple, terrain elevation blocks, or 3D blocks of buildings). Thelist is ordered so that blocks in the immediate vicinity of theuser are sent first, minimizing the delay in receiving data ofimmediate interest. The global hierarchy permits these tobe found quickly. The individual blocks are large enoughthat they contribute significantly to a given view yet smallenough that they are transmitted quickly. Several blockscan be in a view, depending on the viewpoint location. Wehave found that this balance in size makes out-of-core dataretrieval efficient for a variety of data [4, 3].

3.1 Streaming DataSome of the data may stream in continuously. These may

be time-stamped 3D weather data acquired from sensorssuch as Doppler radars, computer simulations, streamingvideo, or other data. A special server collects and organizesthese data, but in addition, mobile users may also collectand transmit streaming data (e.g., video). The overall orga-nizing structure for the streaming data is the same virtualglobal hierarchy discussed above. However, the data are alsoorganized for fast queries of the form, ”Tell me about this

Figure 2: Inner and outer footprints used to man-age retrieval of terrain and imagery data from aserver. Spatial locality of reference helps minimizethe amount of data that needs to be transmitted.

event at this time and location.” To handle these queries weare in the process of developing an event-based hierarchy forthe sequence of steps in the streaming data (e.g., time stepsor image frames) aligned with the quadtree structure. Oneissue we are studying is at what quad level (i.e., what levelof spatial granularity) to insert the event structure. Withthis structure we will be able to ask questions such as, ”Howmany tornadoes occurred in North Georgia during the Sum-mer from 1995 to 1999, where, and with what severity,” or,”At what times and in what locales did this atmospheric pol-lutant exceed this level during Spring and Summer, 1998.”Queries such as this are relevant to the weather applicationinvolving streaming 3D Doppler radar data that we describebelow. The above dynamic structure is discussed more fullyin [8].

4. AWARENESS AND COLLABORATIVESTRUCTURES

Our implementation of Situational Visualization extendsthe spatial model of interaction proposed by Benford andFahlen [2]. In the spatial model of interaction, there arethree regions of note: the focus, aura, and nimbus. Thefocus is the area of interest, or region in which the useris aware of activity and objects. This could be the areadefined by the limit of visibility. The aura is the regionwhere interaction is possible. When the auras of two usersintersect, the users can interact. For example, when twopeople are face to face and within a few feet of each other,they can easily converse. The nimbus is the region in whicha second user can be aware of a first user. For example,on a crowded city sidewalk, a tall person has a much largernimbus than a shorter person. They are easily noticed byothers from longer distance.

There are four ways to modify the regions of focus, aura,and nimbus. The first is implicitly by making changes inlocation or orientation. The second is by explicit modifica-tion of key region parameters, such as radius. The third isby adapter objects, such as a telephone, telescope, or otherreal or imaginary object. The fourth is by boundaries in

space such as walls.This interaction model is employed in networked collab-

orative virtual environments such as MASSIVE and DIVE.However, there are several key differences in between thosevirtual environments and a networked collaborative mobileenvironment, such as a Situational Visualization. The spa-tial model of interaction was developed for virtual environ-ment where users have one virtual location and can eas-ily move in virtual space. Some participants in a Situ-ational Visualization are stationary clients. These clientscould present an essentially unmodified spatial interactionmodel. However, many clients will be mobile. In these mo-bile clients, users will have both a real and a virtual locationthat may or may not coincide. The real or physical locationcorresponds to location reported by GPS. Virtual locationcorresponds to the user’s cursor or viewpoint in the visual-ization. Since a user will often rapidly navigate through thevisualization, virtual location will change faster than physi-cal location. A user will need to be aware of events and ob-jects surrounding their physical location, while they closelyexamine an area that may be a hundred miles away. More-over, the user may wish to monitor several regions withoutexplicitly examining those regions. A user may also want tobe notified of future events that may have significance.

Modifications to the concept of focus would be to createfocus (or awareness) regions for both the user’s real andvirtual locations. The user would also be able to defineadditional regions of focus, in a sense, dropping pieces ofawareness across the terrain or attaching them to particularusers or moving objects. Users who make modifications tothe database should transmit updates to other users whohave an overlapping focus region. Since the spatial modelof interaction defines different region extents for differentmedia (sight, sound, communication, etc.), it follows thatusers should be able to register focus for particular classes ofobjects as well, (weather, traffic, virtual objects, other users,etc.) To deal with warnings about future events, a predictionprocess would determine if objects, such as weather eventsmight intersect a focus region in the future, even if the focusregion is in motion with the user.

In our implementation, auras determine interaction suchas personal communication and database communication.Since physical location typically changes less rapidly thanvirtual location, the concept of aura may be split in two.Transmitting auras will be attached to virtual body and re-ceiving auras will be attached to physical body. When a userwishes to initiate contact, the user will navigate the virtualbody in proximity of another user’s physical body and es-tablish a two-way communication channel. Since users maywish to stay in contact even if users move apart, the channelmust be explicitly closed. In our implementation, personalcommunication is by text messaging. Auras will also applyto database communication. Some users may carry updatedstatic terrain, building, or other virtual object information.For example, one user may have an annotated database ofwith locations of wheelchair ramps, handicapped parkingspaces, automatic doors, and stairways. A second user caninitiate a database synchronization in much the same wayas a communication channel. This allows the second user tofetch and add all the new information from the first user’sdatabase. The second user can also have a previously en-abled database connection that fetches this information fromthe first user as needed. Some special purpose stationary

servers such as a weather information server would projecta location quite different than it’s physical location. Itsdatabase communication aura coincides with the weatherfootprint.

Mobile Situational Visualizations will have nimbus regionsfor virtual location as well as physical location. Nimbus isthe visibility profile of a user. In a standard mode of opera-tion, users would only be aware of the nimbus surroundingthe physical location of other users. Users would only seethe physical locations of other users. If a group of userswere collaborating closely, they may wish to see an avatarrepresenting a user’s virtual location.

5. SITUATIONAL VISUALIZATION APPLI-CATIONS

There are many potential Situational Visualization ap-plications. One class has to do with awareness, being no-tified of time-evolving events based on one’s location, di-rection of movement, or actions. The nano-forecaster de-scribed below is one such application. Another would be a“traffic-wizard” that gives one advice about traffic on po-tential routes as one approaches. Another class would in-volve geo-referenced services, such as telling a mobile userwhere and in what direction restaurants, bus stops, theaters,or other services are based on her current location. Otherapplications could involve annotations and updates of thesynchronized database as determined by real world obser-vations. These might include finding and noting handicapaccess points, noting positions of trees or shrubs and thenplacing appropriately scaled models, noting incompletenessor inaccuracies in building models and then capturing im-ages to be linked to the models, capturing geo-located andoriented video, and many other applications. There are alsocollaborative applications, such as two mobile users investi-gating a large building site and coordinating their informa-tion collection so that it is complete. Other collaborationsmight involve mobile users and those at some central lo-cation or command post. The user at the central locationmight act as an overall coordinator, directing the mobileusers and sending or receiving relevant information.

Our prototype Situational Visualization system consistsof a set of central computers and a mobile system. The cen-tral computers include both desktop machines and a virtualworkbench. The latter enhances collaborations since multi-ple people can be at the workbench and more than one caninteract. All systems are connected by an 802.11b WaveLANwireless network. To reach mobile users, WaveLAN anten-nas have been mounted on the exteriors of two buildings(Figure 3) providing a communication path of about a half-mile through a central part of the Georgia Tech campus. Anominal 11 Mb Ethernet runs on the wireless network. Theoutdoor portion of the network is subject to interruptionsand points where communication may be blocked. This,however, permits us to try out predictive tracking, use ofcached events, or other ideas for handling interrupted ser-vice. The mobile computer is an IBM 850 MHz Thinkpadwith 512 MB RAM and an ATI Mobility 128 AGP graphicschipset with 16 MB video RAM. This configuration runsthe VGIS global geospatial visualization system at interac-tive rates.

To evaluate the Situational Visualization concept and be-gin building the needed infrastructure, we have identified

Figure 3: The Georgia Tech wireless testbed for Sit-uational Awareness. To reach mobile users, Wave-LAN antennas have been mounted on the exteriorsof two buildings providing a communication path ofabout a half-mile through a central part of the Geor-gia Tech campus.

two prototype applications. The first is an awareness appli-cation, the Nano-Forecaster. The second is an annotationapplication, the Wheelchair Assistant.

5.1 The Nano-ForecasterUp to now the best weather reports that a person can

receive is a set of ”micro-forecasts” developed for variousareas of a geographic region. These are not specifically cor-related to a viewer location and do not provide accurateinformation such as ”When is a storm going to hit me,from what direction, and with what intensity?” However,such information can be made available to a user if inte-grated in the above Situational Visualization context withappropriate geospatial model and weather data. We havean accurate model for North Georgia with high resolutioninsets for parts of Atlanta including the Georgia Tech cam-pus and the weather data (in this case, real-time streaming3D Doppler data from multiple radars covering the north-ern half of Georgia). Similar models have been or could bedeveloped for other locales. We have previously shown how3D, time-dependent data can be inserted on-the-fly into thegeospatial hierarchy [8]. The Doppler radars are operatedby the National Weather Service and are distributed acrossthe country, especially around urban or tornado-prone ar-eas. Thus the components are available that could makepossible wide use of the nano-forecaster.

The mobile visualization system is shown in Figure 4,where the user has navigated to a ground level view as-sociated with her location. We have found that a GPS-controlled icon, as shown in Figure 5, gives reasonably ac-curate and easy to follow position updates in an overviewas a user walks through the same area. Further, the systemcan be made to automatically navigate to a view encom-passing the user’s current position if its viewpoint is initiallysomewhere else. The moving position information is enoughto dynamically update nano-forecaster queries. Since thetime-dependent, 3D Doppler data are in the global geospa-tial hierarchy, the parts in the user’s current area of focus

Figure 4: Prototype laptop Situational Visualiza-tion system. Mobile VGIS is running on the laptop,showing a location on Georgia Tech campus plusbuildings in midtown Atlanta.

can be quickly located on the streaming data server. Asso-ciated with these 3D data is a set of storm analyses that aregenerated at the same time as the data are collected. Thesetrack windfield intensities, reflectivity, and spectral width.These quantities can be transformed into severe storm signa-tures (including tornadoes), and precipitation intensity andtype (e.g. , rain or hail). For signature above certain val-ues, features called mesocyclone cells are generated, whichhave a location, width, height, and average values for thesignatures. These cells move over time and can be tracked.Certain cells that have the signatures of tornadoes, for ex-ample, can be flagged. The tracking permits a simple, butaccurate phenomenological model to be applied for predict-ing cell motion. A severe storm passing over Atlanta, withboth 3D radar data and mesocyclones, is shown in Figure 6.

In the current nano-forecaster prototype, the forecastingagent looks only at the subset of mesocyclones, with theirpredicted paths, in the current area of focus. It determineswhether the path of any mesocyclone will pass over the user’sprojected position (taking into account trajectory informa-tion from previous positions). The area of focus includes atime frame of interest; if the intersection is within that timeframe and the mesocyclone signatures are above a selectedthreshold, the user is notified. Notification can be as sim-ple as an arrow, indicating the direction of the approachingweather, labeled with type and severity of storm and time ofarrival, as shown in Figure 7. Alternatively, the user couldrequest a detailed view of the 3D Doppler data in her area offocus, as in Figure 6. The user’s area of focus and trajectoryand the storm’s development are all continuously updated.Since the key notification involves very little data, it couldeven be displayed on a PDA with minimal display capabil-ities, perhaps just showing a fixed image of the user’s areawith her current position and the storm’s labeled vector su-perimposed.

Figure 5: Screenshot of Mobile VGIS tracked byGPS. The arrow at the center indicates the user’sreal position.

5.2 Wheelchair AssistantThe Wheelchair Assistant is an application that allows

wheelchair users to navigate appropriately through an out-door urban landscape. Users can annotate the building andterrain database with icons denoting locations of wheelchairramps, handicapped parking spaces, automatic doors, andstairways. Users will avoid having to double back because ofan outdoor staircase or use a heavy door when an automaticdoor is readily nearby.

The user can add wheelchair annotations by selecting anannotation type and clicking in the VGIS window for thelocation. The user can also move to the proper location andthen select an annotation type. After a short delay, the an-notation will be made. One of the advantages of displaying ahighly detailed set of satellite phototextures, as in VGIS, isthat a user can accurately place an annotation even thoughGPS location information is often inaccurate. The user caninput the position of a wheelchair ramp in relation to itemsin the satellite photo. Furthermore, the user can adjust theGPS position to more accurately reflect the user’s actualposition.

Through database synchronization, users can share theseannotations to create a complete picture of wheelchair accessfor an area. This allows other wheelchair users to benefit.It also allows an organization like a business, school, or cityto visualize and assess how wheelchair users are constrainedor assisted.

A similar application mapping locations of emergency tele-phones, police stations, and manned security desks mightalso be a part of crime prevention and assessment projects.

6. CONCLUSIONSIn this paper, we have introduced a new style of visual-

ization called Situational Visualization, based on our mo-bile Virtual Geographic Information System, VGIS. Ourapproach to issues surrounding how to deploy augmented in-formation spaces to mobile users is to start from the basis ofa rich source of information about the real world, such as ter-rain elevations and aerial photography supplied by various

Figure 6: Overview of Doppler Radar readings overNorth Georgia. The column of cylinders near At-lanta to the upper left is a series of mesocyclones,or large circular wind patterns.

government agencies. This allows us to build an augmentedinformation source that already has a great deal of groundtruth. This Situational Visualization system allows the userto enhance the visualization and any underlying simulationby inputting the user’s observations of the phenomena ofinterest, thus improving the quality of visualization for theuser and for any other users that may be connected to thesame database. The example applications show how such asystem can afford a rich information space that can be usedto enhance the user’s situational awareness.

7. ACKNOWLEDGMENTSThis work is supported by a grant from the NSF Human-

Computer Interaction Program and the DoD MURI pro-gram administered through the Army Research Office. This

Figure 7: Closer orbital view of Georgia Tech andmidtown and downtown Atlanta with a single meso-cyclone plus its predicted path over the next 5 min-utes.

work is also supported by a grant from the NSF Large Sci-entific and Software Data Visualization Program.

8. REFERENCES[1] G. D. Abowd, C. G. Atkeson, J. Hong, S. Long,

R. Kooper, and M. Pinkerton. Cyberguide: A MobileContext-Aware Tourguide. Wireless Networks,3(5):421–433, 1997.

[2] S.D. Benford and L.E. Fahlen. A Spatial Model ofInteraction in Large Virual Environments. InProceedings of Third European Conference on CSCW(ECSCW’93). Kluwer, October 1993.

[3] D. Davis, T.Y. Jiang, W. Ribarsky, and N. Faust.Intent, Perception, and Out-of-Core VisualizationApplied to Terrain. IEEE Visualization ’98, Rep.GIT-GVU-98-12, pages 455–458, October 1998.

[4] D. Davis, W. Ribarsky, T.Y. Jiang, N. Faust, andSean Ho. Real-Time Visualization of Scalably LargeCollections of Heterogeneous Objects. IEEEVisualization ’99, pages 437–440, October 1999. ISBN0-7803-5897-X. Held in San Francisco, California.

[5] M.R. Endsley. Situation Awareness Global AssessmentTechnique (SAGAT). In Proceedings of the IEEE 1998National Aerospace and Electronics Conference,volume 3, May 1998.

[6] N. Faust, W. Ribarsky, T.Y. Jiang, andT. Wasilewski. Real-Time Global Data Model for theDigital Earth. In Proceedings of the InternationalConference on Discrete Global Grids (2000), Rep.GIT-GVU-01-10, 2000.

[7] S. Feiner, B. MacIntyre, T Hollerer, and A. Webster.A Touring Machine: Prototyping 3D Mobile andAugmented Reality Systems for Exploring the UrbanEnvironment. In Proceedings of the First InternationalSymposium on Wearable Computers 1997, October1997.

[8] B. Hannigan, R. Mitchell Parry, Nickolas L. Faust,T. Y. Jiang, and William Ribarsky. HierarchicalStorage and Rendering of Real-Time 3D Data. In NickFaust, editor, Visualization of Temporal and SpatialData for Civilian and Defense Applications III,volume 4368A, page 8 pages. SPIE Press, 2001.

[9] Tobias Hollerer, Steven Feiner, Tachio Terauchi, GusRashid, and Drexel Hallaway. Exploring MARS:Developing Indoor and Outdoor User Interfaces to aMobile Augmented Reality. Computers and Graphics,23(6):779–785, 1999.

[10] Simon Julier, Marco Lanzagorta, Yohan Baillot,Lawrence Rosenblum, Steven Feiner, Tobias Hollerer,and Sabrina Sestito. Information Filtering for MobileAugmented Reality. In Proceedings of IEEEInternational Symposium on Augmented Reality 2000,October 2000.

[11] Rob Kooper and Blair MacIntyre. The Real-WorldWide Web Browser: An Interface for a ContinuouslyAvailable, General Purpose, Spatialized InformationSpace. In Proceedings of the 2nd InternationalSymposium on Mixed Reality, March 2000.

[12] Michael R. Macedonia, Michael J. Zyda, David R.Pratt, Donald P. Brutzman, and Paul T. Barham.Exploiting reality with multicast groups. IEEEComputer Graphics and Applications, 15(5):38–45,

1995.

[13] J. Pascoe. Adding Generic Contextual Capabilities toWearable Computers. In Proceedings of the SecondInternational Symposium on Wearable Computers1997 (ISWC ’98), October 1998.

[14] Christopher D. Shaw, R. Mitchell Parry, WilliamRibarsky, Anthony A. Wasilewski, and Nickolas LFaust. Interactive Volume Rendering of Non-UniformData. In Nick Faust, editor, Visualization of Temporaland Spatial Data for Civilian and Defense ApplicationsIII, volume 4368A, page 8 pages. SPIE Press, 2001.

[15] J. C. Spohrer. Information In Places. IBM SystemsJournal, 38(4):602–628, 1999.

[16] Riku Suomela and Juha Lehikoinen. ContextCompass. In Proceedings of the Fourth InternationalSymposium on Wearable Computers 2000 (ISWC2000), October 2000.

[17] B. Thomas, V. Demczuk, W. Piekarski, D. Hepworth,and B. Gunther. A Wearable Computer System withAugmented Reality to Support Terrestrial Navigation.In Proceedings of the Second International Symposiumon Wearable Computers 1997 (ISWC ’98), October1998.