DoppelLab: Tools for Exploring and Harnessing MultimodalSensor Network Data

Gershon Dublon, Laurel S. Pardue, Brian Mayton, Noah Swartz, Nicholas Joliat, Patrick Hurst and Joseph A. ParadisoResponsive Environments Group, MIT Media Lab, Cambridge, MA 02139

Email: {gershon, punk, bmayton, swartzcr, njoliat, phurst, joep}@media.mit.edu

Abstract— We present DoppelLab, an immersive sensor data browserbuilt on a 3-d game engine. DoppelLab unifies independent sensornetworks and data sources within the spatial framework of a building.Animated visualizations and sonifications serve as representations of real-time data within the virtual space.

I. INTRODUCTION

Our environments are increasingly filled with sensor networksdesigned to provide specific data and solve predetermined problems,but information from embedded sensors remains, for the most part,locked in closed-loop control systems and out of reach of the majorityof users. In a technology-driven rush to collect new and ever morestreams of information, sensor networks have been deployed soquickly and broadly that users, be they consumers monitoring theirpersonal energy usage or corporations tracking their global supplychains, can barely manage the data influx, much less know how tointerpret the data. Rapid developments in sensor networks researchare enabling real-time, mobile and distributed sensing, bringingthese technologies into new domains, from homes and offices toconstruction sites, where users might face challenges integrating real-time information into existing practice. At the same time, sensorfusion algorithms that would make sense of these data require aposteriori knowledge, which in many cases is lacking or non- existentfor new deployments. When models are available, these algorithmstend to reduce data to single dimensions of interpretation, separatedfrom other data streams that may provide valuable context.

Motivated by increasingly dense deployments of disparate sensornetworks and an absence of interface tools that can truly span acrossnetwork-siloed sensor data, we present DoppelLab, a scalable, open-ended platform for creating collocated representations of general sen-sor network data. This platform supports an audio-visually immersive,3-d virtual space in which users can explore real-time sensor data andinferences of all types, as well as long-term, aggregated informationand knowledge. Building on a game engine, DoppelLab transformsarchitectural models into browsing environments for real-time sensordata visualization and sonification, proposing to organize these virtualrepresentations by the physical space from which they originate, andthereby enabling both broad and specific queries about the activities,systems, and relationships in a complex, sensor-rich environment.By providing tools for rapid parsing, visualization, and applicationprototyping that can take advantage of the platform’s horizontalrelationship to otherwise independent sensor networks, DoppelLabaims to drive new interfaces to data and physical-world actuation.

DoppelLab lets users walk through and interact with sensor datawithin a building, or fly out of the building to see it buzzing withlayers of activity and information. To date, we have incorporated hun-dreds of building-wide distributed sensors at the MIT Media Lab andother instrumented spaces, and are planning for further deployments.Our visualizations are based on intuitively metaphorical animationsand are easily adapted or swapped for different applications, a keyconsideration in our system design that allows developers to quickly

Fig. 1. Translucent view of the MIT Media Lab in DoppelLab.

prototype their own visualizations and customize existing ones. Wehave also begun exploring sonification as a method for increasinguser immersion: our system incorporates live, voice-scrambled audiofeeds from across the Media Lab complex, and spatializes them inthe virtual model. We are planning to mix this audio with abstractsonifications of sensor data, providing a means for users to hear fully3-d representations of information.

Intuitively and scalably collocating representations of dense sensornetwork data presents a significant design challenge. We elucidatethese challenges, and draw on visualization design strategies to helpsolve them. Behind the scenes, our back-end server and databaseframework collects data from multiple, disparate networks and storesit generically, providing a means for aggregation and analytics thatcut across individual sensor networks. More broadly, this paper laysthe groundwork for exploratory modes of user interaction with sensornetwork data. By enabling users to move fluidly between convergentdata streams and the higher-level inferences that fuse them together,this work aims to develop interfaces that foster better understandingof sensor-driven context and the data that produce it.

Our work draws on a broad set of fields, including cross-reality,visual analytics, sensor network user interfaces, data sonification,and sensor data management. Sec. II of this paper reviews therelevant literature from these fields, and situates the work withinthem. Sec. III presents the DoppelLab system, describes the sensornetworks, visualizations, and sonifications we have developed, anddetails the client-side and back-end architectures that constitute theplatform. DoppelLab is an ongoing research effort, and Sec. IV brieflyoutlines our future plans. Sec. V concludes the paper.

II. RELATED WORK

There have been a number of recent efforts to create general-purpose sensor network user interfaces. The Tricorder [1] and Ubi-corder [2] are a family of location- and orientation-aware handheldwireless sensor network browsers; the latter provides an interfacefor defining inference rules. Both systems feature a graphically-

augmented 2-d floor plan. Other work has focused in particular onuser interfaces for defining inference rules on data streams throughscripting [3]. A number of commercial enterprises have developeduser interfaces to sensor data, for consumer applications like homeenergy monitoring [4], [5]. These products center on 2-d graphicalrepresentations that can become quite complex as networks grow.

Recent efforts to approach this problem theorized cross-realityenvironments, where ubiquitous sensor networks would interact withpervasively shared virtual worlds [6], [7]. These efforts focused onthe point of exchange between physical and virtual environments,and hinged on the concept of massively multiplayer virtual worlds,like Second Life. Many of the challenges faced by this research areparticular to that concept of pervasive sharing in virtual worlds, chiefamong them the vacancy problem, which results from a combinationof asynchronous usage and typically low levels of user interest. Incontrast, this paper focuses on encouraging and enriching individualusers’ experiences of sensor data in 3-d environments.

Several other papers explore the use of game engines for in-formation visualization. Brown-Simmons, et al. use game engineaffordances to encourage exploration of environmental data andcommunicate a specific point of view [8], [9]. Kot, et al. apply agame engine to information visualization [10]; but focus on gamingmetaphors such as shooting, limiting their scope.

The burgeoning field of visual analytics develops interactive visualinterfaces that support analytical reasoning and decision-making.Visual analytic designs aim to to transmit large data sets concisely,enable comparison between multiple streams of information, andencourage user interaction with visualizations [11]. Users choose be-tween modular representations, assess the efficacy of various designs,and aid in the fabrication of new designs [12]. This ease of interactionfacilitates feedback workflows from design to inference and back.

DoppelLab incorporates live, spatialized audio streams into thevirtual environment. This work has an early precedent in systems likeCAVE [13], which created an immersive soundscape by spatializingsounds within a 3-d virtual space. We are beginning to investigatesonification of real-time sensor data streams over a network. In[14], Lodha et al. develop a generative music system based onWestern tonal harmony, and emphasize the importance of using amusically pleasing sonification to improve user experience and reducefatigue. In [15], Hunt and Hermann show the benefits of low-latency,direct-manipulation interfaces in sonification. We address this throughclient-level spatialization which responds to movement in the virtualspace. Other work has shown that pitch-mapping sonifications can beas effective as visualization in conveying properties of functions ofone variable [16].

III. DOPPELLAB

DoppelLab is an immersive, cross-reality virtual environment thatserves as an active repository of the multimodal sensor data producedby a building and its inhabitants. Built on the Unity3D game engine,the platform leverages physics and lighting, interactive animations,and other game engine affordances such as distance culling tostructure and encourage exploration of the data.

A. Visualizations

In DoppelLab, visual representations of data take the form ofmetaphorical animations that express absolute and relative magni-tudes, rates of change, and higher-level inferences that relate dataacross time, space, and sensing modalities. Visualizations take placewithin the walls of the building, corresponding to the physical loca-tions of the sensors, and the walls can be toggled into translucence

Fig. 2. An instrumented ping pong table is visualized in DoppelLab asa 2-d game of Pong (center). In the background, temperature and humiditymeasured on a dense grid in the atrium are shown as colored shapes and fog.

to expose the relationships that span across the building. Organizingdata in this way provides a largely intuitive platform that emphasizesthe relationship between people and their physical environments.While projections and transformations to other spaces can producemore efficient representations of data for many kinds of queries, thephysical space serves as a consistent and relatable starting point.

We draw on Edward Tufte’s seminal works to identify a number ofvisual principles that structure our interface design [17]. Adherenceto the physical space provides a means for self-representing scales,that normalize the size (and relative significance) of any visual cueagainst the architecture itself. In DoppelLab, a large and fast growingset of data sources poses a visual and interface design challenge. Assuch, we seek visual representations that engage with users to revealinformation and expose functionality in response to their explorationthrough the virtual space, making the density of information in arepresentation a function of the user’s virtual proximity to it. Thisnotion of macro and micro design makes the platform scale tolarge numbers of data sources and analytical visualizations withoutoverwhelming the user. To the same end, we aim for visualizationsthat make use of every graphical element to show data; thesemulti-functioning elements consolidate conceptually related sensingmodalities into compact forms, using multiple properties of eachobject or animation.

In its current form, DoppelLab renders data from an increasinglylarge and dense set of building-wide distributed sensors at the MITMedia Lab complex and one researcher’s instrumented living space.Fig. 2 includes a dense network of 45 temperature and humiditysensors suspended in a large atrium represented by nodes whose colorrepresents temperature and shape reflects the rate of change; thesenodes are surrounded by foggy clouds of particles that track levels ofhumidity, where dense, red fog indicates high levels, and blue, sparsefog the opposite. DoppelLab supports multiple visual interpretationsof the same data that can be toggled using keyboard modifier keys.Shown in Fig. 3, a system of several hundred thermostats at the MediaLab is represented by flame animations whose colors reflect absolutetemperature, and by spheres whose color and size reflect the localdeviation of the temperature from the set point; both visualizationshighlight anomalously large differences between the two.

DoppelLab connects to a network of distributed sensor nodescarrying motion, temperature, humidity and light sensors, as wellas stereo microphones. Some of these nodes are linked to daughterboards with barometric pressure, light color, ozone, volatile organic,and particulate sensors, for monitoring machine shop activity. Coin-cident audio levels and motion are represented as a set of spheresthat accumulate into circle as audio levels increase (resembling atraditional level meter), and undulate as motion increases, combining

Fig. 3. Thermostats are represented by colored flames (left). An anomalybetween the setpoint and measured temperature is indicated by a pulsatingsphere (center) and an alternate visualization (right) shows the magnitude ofthe discrepancy.

Fig. 4. Social activity visualizations: at left, an audio level-meter rippleswith the amount of coincident motion, while the typical levels float overhead;the arrows reflect an unusually high level of social activity that is trendingupward. At right, RFID tags are detected and faces are shown.

the two into an indicator of social activity. When a user approachesthe visualization, the typical sound and motion levels for the currenthour and current day of the week appear for comparison as a ghostlycopy of the original. As shown in Fig. 4, when the activity levelover the preceding 10 minutes exceeds the typical value by somemargin, the visualization spawns a cloud of arrows whose densityand orientation reflects the trending social hot-spot. DoppelLab alsomakes use of an extensive network of RFID readers, showing thefaces of people carrying tags as they move through the building. Inaddition, Twitter messages appear in DoppelLab as they are broad-cast, localized to individuals’ virtual offices using an institutionaldirectory. The position of a ping pong ball on a sensor-equippedtable is shown as a continuous game of 2-d pong (Fig. 2). Finally,DoppelLab incorporates external factors, like the position of the sun,under the assumption that it may impact the HVAC and other systems.

B. Sonification

We aim to provide an informative and pleasing data sonificationto complement DoppelLab’s graphical display. We have designed atiered architecture for audio processing, spread across sensor nodes,centralized servers, and the client.

The first layer of this architecture performs node-level speech ob-fuscation, preserving the character of the sound but rendering speechunintelligible to protect privacy. We prototyped two algorithms: thefirst reverses sequential segments; the second stores 5 recent segmentsand selects from them randomly. The first has the benefit of leavingsounds slower than speech, including some music, mostly unchanged,and listeners found the output smooth. However, reversing drasticallyalters transient sounds, and can be easily decoded by an eavesdropper(although randomizing the buffer size could mitigate this problem).The second technique preserves transients, and is significantly harder

to decode. However, listeners noted that this algorithm sounded more“busy” and “choppy”, and longer structures like sentences were moredisrupted. We are presently investigating the potential for granularsynthesis to address some of these challenges.

After obfuscation, audio is streamed to a dedicated DSP server,where further data sonification algorithms run. This server supportsrapid programming in Max/MSP. On the client, we have designed acustom plugin to download, decode, and spatialize the streams.

C. Interface

DoppelLab is built on the Unity game engine, which makes useof Mono, the open-source version of Microsoft’s .NET Framework.Scripts are written in C# and UnityScript, a JavaScript-based lan-guage. The system is organized hierarchically; centralized scripts dealwith server communication and parsing, and pass messages to lower-level scripts that manage visualizations of each network. Centrally,the update loop periodically polls the data server, and the serverresponds with an XML file containing the requested data. If no data isfound for any given sensor, the corresponding animation is disabled.The system relates its coordinate system to the building floorplan, andpasses messages and data accordingly; lower-level scripts that managevisualizations and animations use those data to control propertiesof objects in the environment. These parameters are visualizationdependent, and include object properties like color, shape and size,or in the case of particle systems, systemic properties like emissionrate and lifetime.

A simple development process is central to our system design,allowing developers to quickly prototype their own visualizationsand customize existing ones. The application makes a number ofanimations “drag-and-drop” ready so that developers need not re-peatedly reinvent existing designs, and to produce a certain levelof consistency in the user experience. The goal is that DoppelLabsupport a rapid development-to-visualization-to-development cycle,whereby visualizations suggest relationships and correspondences tousers, and the environment enables on-the-spot prototyping of newapplications. One such application, designed for building managers,reveals anomalies in the thermostat data–specifically, large deviationsfrom the local set point. This idea came about because a visualizationof absolute temperatures showed what appeared to be stronglycorrelated and unusually high temperatures in a series of adjacentrooms; significantly, the result exposed a previously unknown faultin the building HVAC system.

In addition to streaming in real-time, data are also stored, aggre-gated, and analyzed on the server, enabling users to speed throughweeks or months of data in minutes. We created two preliminaryinterfaces to this functionality on the client side, supporting a simple(exploration) mode and an advanced mode. The exploration mode,shown in Fig. 5, is modeled on video editing timelines, where the topslider sets the system time within a 24 hour range, and the bottomslider moves that range across a span of days and months. Data canbe played back at 4 different time scales–real-time, 1 minute persecond, 10 minutes per second, and 1 hour per second. The slidersadvance with the time, together with an analog clock. The advancedmode supports input of specific times into a text box.

D. Server and Database Architecture

The data used by DoppelLab is generated by multiple independentnetworks. The DoppelLab architecture includes a server that bringstogether data from these independent networks, storing it in an SQLdatabase. This provides the client access to both current and historicalsensor data from many different sensor networks in a generic format.

Fig. 5. Interface for exploring historical sensor data.

The database distinguishes between two types of sensor data,samples and events. Samples are numeric values produced by a sensorat regular intervals, and are stored as a triplet of a unique sensor ID,a timestamp, and the numeric value. Events occur irregularly, andmight contain larger amounts of data; for example, the appearance ofa tag (with an associated username) at an RFID reader, or a messageposted to a Twitter feed, are both stored as timestamped events.

Metadata about each sensor is also stored in the database, includingthe sensor’s type and location. Sensors are organized into logicalgroups to facilitate higher-level queries: a thermostat, for example,has both a temperature sensor and a setpoint, and a node in a sensornetwork might have a collection of various sensors. We are alsobeginning to implement facilities for sensors with changing locations,where the location of a sensor is itself a sensed value.

Scripts on the server periodically request the data from each datasource and store the values in a generic format in an SQL database.Simultaneously, the server also computes hourly averages which arecached in a separate table. This aggregation permits fast execution ofqueries over longer timescales, such as ‘What is the average readingon this sensor at 3 PM on Mondays?’, or ‘What days of the weektend to have the highest average motion activity at noon?’

The DoppelLab client accesses the data stored on the server viaHTTP requests, which allow queries for aggregate data over timeintervals ranging from one second to ten minutes, and over hours.

IV. FUTURE WORK

DoppelLab is evolving quickly as we add sensors and features, aswell as new buildings and environments, including our own instru-mented living spaces. We are integrating real-time data from wearabledevices and fine-grained location systems, as well as distributedvideo. As we deploy interfaces in new built environments, we havebegun testing the system with facilities managers, who have foundDoppelLab’s spatial organization of sensor data particularly usefulfor tracking the state of building systems.

Our hierarchical and modular system simplifies the design anddevelopment process for visualizations and sonifications, as well asback-end analytics. An extension of our efforts to facilitate rapiddeployment of new interfaces is the design of a markup languagethat will codify and further streamline this process.

Sonification in the DoppelLab environment is very much a work-in-progress. We are exploring the use of generative music techniquesto encode further data in our sonification, in addition to spatializedaudio recordings. For example, by applying resonant filters to theaudio streams, we can differentiate the streams from one another,while also mapping new data parameters to the cutoff frequenciesof the filters. Finally, we are beginning an arts residency with acomposer, and hope to draw on his musical experience in our design.

V. CONCLUSION

Motivated by the increasing availability of real-time data and fewmeans for users to explore these layers of information, we devel-oped DoppelLab, a comprehensive browsing interface that enablesexploration across independent sensor networks and data streams.

DoppelLab is an open-ended and intuitive environment that trans-forms data into interactive visual and auditory experiences, aimingto foster better understanding of sensor data and their relationshipsacross space, time, and modality. We see this work leading to atransformative moment in ubiquitous computing, where interfaces andapplications built atop distributed sensor streams intimately connectpeople to the rapidly emerging Internet of real-time data.

More information about DoppelLab can be found in [18], and aninteractive web player is available at http://doppellab.media.mit.edu.

ACKNOWLEDGMENTS

The authors would like to thank the hard-working and creativeundergraduate researchers who contributed to this work: AnishaJethwani, Jeffrey Prouty, Turner Bohlen, and Tanya Liu. Specialthanks to Mary C. Murphy-Hoye of Intel for her valuable insights.This work was supported by Intel, the Things That Think Consortiumand the research sponsors of the MIT Media Lab.

REFERENCES

[1] J. Lifton, M. Mittal, M. Lapinski, and J. A. Paradiso, “Tricorder: Amobile sensor network browser,” in Proceedings of the ACM SIGCHIconference on Human Computer Interaction, 2007.

[2] M. Mittal and J. A. Paradiso, “Ubicorder: A Mobile Device for SituatedInteractions with Sensor Networks,” IEEE Sensors Journal, 2011.

[3] A. Bamis and A. Savvides, “STFL: a spatio temporal filtering languagewith applications in assisted living,” in Proceedings of the ACM 2ndInternational Conference on PErvasive Technologies Related to AssistiveEnvironments (PETRA ’09), 2009.

[4] Google PowerMeter. [Online]. Available: http://google.com/powermeter[5] Microsoft Hohm. [Online]. Available: http://www.microsoft-hohm.com[6] J. Lifton, M. Laibowitz, D. Harry, N.-W. Gong, M. Mittal, and J. A.

Paradiso, “Metaphor and Manifestation—Cross-reality with UbiquitousSensor/Actuator Networks,” IEEE Pervasive Computing, vol. 8, no. 3,Sept. 2009.

[7] B. Coleman, “Using Sensor Inputs to Affect Virtual and Real Environ-ments,” IEEE Pervasive Computing, vol. 8, no. 3, pp. 16–23, 2009.

[8] G. Brown-Simmons, F. Kuester, C. Knox, S. Yamaoka, and D. Repasky,“Kepesian Visualization,” in Proceedings of Connectivity, the TenthBiennial Arts and Technology Symposium. New London, CT: UC Irvine,Jan. 2006, pp. 25–36.

[9] F. Kuester, G. Brown-Simmons, C. Knox, and S. Yamaoka, “Earth andplanetary system science game engine,” Transactions on Edutainment II,Lecture Notes on Computer Science (LNCS), vol. 203-218, pp. 203–218,2009.

[10] B. Kot, B. Wuensche, and J. Grundy, “Information visualisation util-ising 3D computer game engines,” Proceedings of the ACM SIGCHIconference on Computer-human interaction (SIGCHI ’05), 2005.

[11] J. J. Thomas and K. A. Cook, “A visual analytics agenda,” IEEEComputer Graphics and Applications, vol. 26, no. 1, pp. 10–13, Jan.2006.

[12] M. Beaudouin-Lafon, “Designing interaction, not interfaces,” in Pro-ceedings of the working conference on Advanced visual interfaces.ACM, 2004, pp. 15–22.

[13] R. Bargar, I. Choi, and S. Das, “Model based interactive sound foran immersive virtual environment,” in Proceedings of the InternationalComputer Music Conference (ICMC ’94), 1994.

[14] S. Lodha, J. Beahan, T. Heppe, and A. Joseph, “Muse: A musicaldata sonification toolkit,” in Proceedings of International Conferenceon Auditory Display (ICAD ’97), 1997.

[15] A. Hunt and T. Hermann, “The importance of interaction in sonification,”in Proceedings of the Int. Conference on Auditory Display (ICAD 2004),S. Barrass and P. Vickers, Eds., International Community for AuditoryDisplay (ICAD). Sydney, Australia: ICAD, July 2004.

[16] J. H. Flowers, D. C. Buhman, and K. D. Turnage, “Data sonificationfrom the desktop: Should sound be part of standard data analysissoftware?” ACM Trans. Appl. Percept., vol. 2, pp. 467–472, October2005. [Online]. Available: http://doi.acm.org/10.1145/1101530.1101544

[17] E. R. Tufte, Envisioning Information. Graphics Press, May 1990.[18] G. Dublon, “Beyond the Lens: Communicating Context through Sensing,

Video, and Visualization,” S.M. Dissertation, Massachusetts Institute ofTechnology, Media Arts and Sciences, 2011.