1

Big Signal: Information Interaction for Public Telerobotic ExplorationPeter Coppin*, Alexi Morrissey*, Michael Wagner*, Matthew Vincent**, Geb Thomas

* The Big Signal InitiativeThe Robotics Institute

Carnegie Mellon University, Pittsburgh, PA15213

http://www.bigsignal.net/

The GROK Lab** Department of Industrial EngineeringUniversity of Iowa, Iowa City, IA 52242

http://grok.ecn.uiowa.edu/

AbstractObstacles must be overcome in order to

make viable public distribution of interactiveremote experiences via the Internet. Highlatency and the fact that there are many moreusers than robots make traditional forms oftelerobotics difficult. The Big Signal projectseeks to overcome these obstacles usinginformation interaction tightly coupled to alive autonomous rover mission.

Information interaction allows users toengage in a rich exploratory experiencewithout affecting the robotics mission.Additionally, information interactionadheres to the Internet standard ofclient/server models that allow many usersto interact with one data set of information.In December 1998, Big Signal deployed aprototype project by providing aneducational interface that allowed studentsand the public to participate in remotetelescience.

1 IntroductionFactors of Success for the Internet =

Obstacles for Internet TeleroboticExploration.

Interactive television, video telephonesand the like were the subject of sciencefiction stories and research labs for yearsuntil the low data requirements anddecentralized client/server configurations ofthe Internet converged with suitable modemspeeds that could accommodate primarilytext based information distribution. Soonimage compression techniques alloweddigital images to be transferred within a

reasonable amount of time. Now virtualcommunities and computer gaming allow adecentralized public to interact in virtualspace using standardized Internettechnologies.

These attributes, which are primecomponents of success for the transfer ofInternet data packets, are the coffin nail for aremote user attempting to engage in a real-time telerobotics scenario.

1.1 Problem: Communication BottlenecksControlling an electromechanism in the

real world in real time requires monitoring asituation remotely through sensors, sendingcontrol commands to the electromechanism,and finally observing the result in enoughtime to direct the physical machine to itsgoal.

In the case of the Internet directtelerobotic closed loop control is difficult,because many users participate over lowbandwidth connections such as modems. AnInternet based remote experience must bestructured in such a way that time lags do notaffect an exploratory experience.

1.2 Problem: Many People, Few RoversThe second problem is a problem that is

not inherent to Internet telerobotics, but topublic telerobotics in general. As a publicevent, there are usually many more peoplethan there are robots. Several public roboticexperiences have attempted to resolve thissituation in several ways.1.2.1 Traditional Approach: One Operator,One Robot

2

A traditional approach to teleroboticexperience is the one operator, one robotparadigm (see figure 1.1). Though suitablefor personal robotic experiences, this isimpractical for public experiences whereother users must wait in line in order to gainthe precious and rarified experience due to itssingular nature.

1.2.2 Democratic Approach: VotingA second paradigm used for public

telerobotic experiences is a “democratic”voting scenario (see figure 1.2). In thissituation, users vote on the direction that arobot may travel or look. An example of thisproject was part of the public outreach effortfor NASA/CMU’s Atacama Desert Trek. [1-3]

In this project, CMU worked withPittsburgh’s Carnegie Science Center toproduce a public interface for the Trek. LivePanoramic imagery was projected onto ahemispherical screen that surrounded atheater equipped with buttons at each seat inthe theater. By pushing buttons on these

seats, users were able to vote on the nextdirection that Nomad would travel. Otherbuttons controlled the pan/tilt direction of thepanoramic image that was projected onto thescreen.

Though thrilling to see full colorimmersive imagery from 5000 miles awayupdating in real time, this voting scenariorequired the collective collaboration of alarge group of people, resulting in the neglect

of individual urges in the audience.1.2.3 Hybrid Approach: One UserControls, Others Watch

Another approach to public remoteexperiences is a one operator / many viewersapproach (see figure 1.3).

For example in a project called RoverTV. This venture was an "interactive"television show that allowed viewers toactively explore a remote environment bygiving TV watchers control of a the NomadRover that was deployed in the Atacamadesert of Northern Chile 5000 miles away.Viewers observed live imagery from theRover and gained control by dialing anumber with their touch-tone telephones.Users controlled both camera movements andsteering of the rover.

Figure 1.2: A democratic approach

Figure 1.3: One user controls, many

Figure 1.1: A single operator and robot

3

2 Solutions Using Autonomous RoverSystems and Information Interaction

2.1 Solution: Eliminate CommunicationBottlenecks through Autonomy Systems

Time lag is not a new issue fortelerobotics. Space relevant rover technologyhas focused on the time lag issue for yearsbecause of the amount of time it takes forsignals to get from the Earth to off worldlocations and back again. Space roverscontain autonomy systems to keep themrunning between information bursts fromEarth bound controllers. Some of thesesystems focus on autonomous navigation andscience. One such project that explores thecreation of autonomous navigation andscience systems is the NASA/CMU RoboticSearch for Antarctic Meteorites.[4-6] Thisproject seeks to develop robotic technologythat allows the autonomous discovery andclassification of rocks and meteorites onAntarctic ice fields. This technology willenable a robot to deploy in an ice field,search this field to some level of precision,spot a potential target, autonomouslynavigate to it, and classify it as a type ofrock or meteorite. Multiple sensor modalitiesare required to classify a meteorite moreeffectively than a human scientist, but therobot has limited energy available. Therefore,to efficiently search an area without wastingtime or energy requires science autonomy – acapability that does not just safely guide arobot to a waypoint, but actually createsmission plans from higher-level goals.

To further this research, the project sentan expedition team and the robot Nomad toPatriot Hills, Antarctica in October 1998.During this expedition, the polar navigation,ice traversal and meteorite / rockclassification capabilities of Nomad weretested. Data from a wide number of sourceswere sent back to CMU:

• High-resolution images of rock /meteorite targets

• Panoramic images• Spectrometer results

• GPS information• Telemetry from Nomad’s numerous

pose, weather, and other sensors

2.2 Solution: Information InteractionThese many types of information had to

integrate into one seamless experience by BigSignal. The limited bandwidth fromAntarctica to CMU and bad weather inAntarctica made communications andinfrequent; however, each new transmissionaugmented the data set at CMU’s server.These data were processed into a form thatcould easily be integrated into and publiclyaccessed in the Big Signal web site. Thistransformed robotic activity from a one rover/ one user model to a client/server typesystem, optimized for the Internet. Thepublic would now be able to access thewealth of information rather than the robotitself: a model we call informationinteraction.

An archive of data on a stateside servercontinuously downloading from a remoterover provided the perfect framework for ademonstration of public information accessto a telerobotic mission.

3 Pilot Project: An EducationallyRelevant Remote Science Interface

In the fall of 1998, the Big Signal teamcreated a prototype project consisting of aninteractive web site that linked selected pilotclassrooms with the 1998 RAMS mission.This project provided a perfect platformupon which to test and demonstrate the useof information interaction for remote sciencein a public setting.

Classroom teachers, interface designers,web designers, illustrators, videographersand robotics researchers worked together toboth create content and adapt robot telemetryto web formats that students and teacherscould access from the classroom. Pilotclassrooms focused on the areas of physics,remote geology, and technology education.

Students using Big Signal encounteredseveral interface features that engaged them

4

in the active experience of remoteexploration.

3.1 Remote SensingThe most visible connection to

Antarctica was a continuously updatingpanoramic image taken by a camera onboardthe rover and sent with other robot data viathe communications link to the project webserver in Pittsburgh. This updating imageallowed students to gain a tangible view ofthe most recent condition of the environmentin Antarctica (see figure 3.1)

Students viewed these panoramic imagesover the web using Apple’s QuickTime VRPlug-ins. This downloadable cross-platformstandard virtual reality technology plug-inwas the perfect counterpart for Nomad’sunique imaging system specifically designedto acquire complete 360-degree images, asshown in figure 3.2.

Custom software converted thepanospheric images acquired by Nomad intomercador projection (panoramic) imagesrequired for input into the web friendly andhigh performance QuickTimeVR Movieconversion program. Use of this VRtechnology created an immersive experiencefor the students while using minimalcomputational resources to support the usercontrolled photorealistic movie capability.Figure 3.3 illustrates the method by whichthe raw panospheric image converts to aQuickTime VR format.

The ultimate goal of creatively applyingvirtual reality technology to quickly andreliably exchange information between therobot and the students was the ultimate goalof this aspect of the user interface. Thepanospheric to QuickTime VR interfacedesign approach was a significant successfactor in achieving this goal.

3.2 Data ReductionIn addition to direct links to sensory

information, Big Signal also reduced rawdata sent from Antarctic. Therefore,technical facts could be presented in amanner easily understood by students and thepublic. There were three main aspects of BigSignal that used reduced data.

Figure 3.2: Unaltered 360-degreepanoramic photograph.

Figure 3.1: Panoramic images provided a tangible link to the remote environment.

5

Panospheric to QTVRConversion

Panospheric to QTVRConversion

Image Data:Panospheric

Image

PerspectiveConversion:

Panospheric toMercadorProjection

Locate "ImageCeter" based on

Axis of Symmetry

CameraCalibration Data

ImageReorientation& Cropping for

input intoQTVR

Con

trol

Tran

sfr

SG

Is

Run non-GUI QTVRMakePanoVR

input is Mercador

URL link to newQTVR file

Controltransferto any

QTVR-enabledplatform

QTVRrepresentation of

robotenvironment

Figure 3.3: Panospheric to QuickTime VR conversion methodology.

6

3.2.1 An “I Robot” InterfaceA scrolling “I Robot” text bar

automatically presented late breaking eventscoming from the rover. By clicking on wordsin this scrolling text bar, students were ableto link to more detailed backgroundinformation that allowed them to understandlate breaking events.

The text displayed in the “I Robot”scrolling text bar was comprised ofprocessed network messages inside the robot.These messages were recorded as telemetryand sent back to Pittsburgh. Software atCMU automatically analyzed the telemetry.Interesting combinations of networkmessages converted into English sentences,structured in first person perspective tosound as if the robot were personallyinforming the web user.

Although the text generation process wasnot elaborate, an “I Robot” style ofinteraction is capable of enhancing theexperience of a telerobotic user. First, therobot presented itself in a personal, humanmanner, making it more engaging to noviceusers. Second, it is an effective way tocommunicate mission updates, because itposts data immediately to the center of theuser’s screen in an easy to understandformat.3.2.2 Interactive Map

“I Robot” data mining provided rawmaterials for other interface elements. Robotposition information was correlated withscience finds and images, then plotted onto acontinuously updating interactive map thatacted as a visual, clickable interface to theprogress of the mission.

Icons on this map represented currentand past positions of the rover. Clicking mapicons activated datalogs, science data, andscientist reports from previous days in theform of daily report pages.

Map visualization was created withVirtual Reality Markup Language (VRML);a cross-platform dynamic three-dimensionalstandard that was interfaced with roverdatabase protocols. Remote users explored

interactively the dynamically updated virtuallandscape. The modeling language enablesthe creation of a 3D scene in which the userobserves the entire expedition area (seefigure 3.4). The user can select icons thatpresent further scientific information oractivate the QuickTime VR window with aVR scene from that exact robot position onthe map. This holistic view of the explorationallowed a vantage point by which users cansee the total progress of the exploration aswell as have a sense of orientation to locallandmarks, before entering the QuickTimeVR robot environment. User disorientationwas prevented by restricting mapmanipulation orientation angles, while stillallowing “zoom in” and “zoom out”capabilities. Figure 3.5 diagrams the methodused in converting incoming GPS data intoVRML map icons.3.2.3 Daily Report

The daily report functioned as both arepository for robot data types and as asource for news about the team and theexploits of the robot from a human point ofview. Research external to the robot wasalso posted in the Daily Report. The DailyReport was designed to interface to usersaccessing Big Signal from the twelve pilotlibraries in Allegheny County, Pennsylvania.This aspect of the page also proved useful toresearchers affiliated with

Figure 3.4: An interactive map acted asan interface to the mission over time.

7

GPS data coversion to VRMLGPS data coversion to VRML

Site Map often not scanned inas a perfectly parallel square.

(trapezoid shape likely)

SkewedMap

SkewedMap

GPS coordinateslocal coordinatesbase unit coordinatesvelocity

(4) Points (inPixel locations)

of knownLatitude andLongitude

Vector Placement ofPanospheric Image(mapped to pixel

location on site map)

Image given URL linkto QTVR file for that

exact location

User can "fly" overhead thesite map and specify a

location along the robot pathfor closer examination in a

QTVR environment

Location assignedcolor code to represent

time of day.

Controltransferto anyQTVR-enabledplatform

cont

rol

trans

frS

GIs

Figure 3.5: GPS data conversion to VRML

8

the project who were not participating in thefield test. The Daily Report section was acombination of data automation andinformation created manually by theAntarctic expedition team (see figure 3.6).

Users could select from various datesfrom the mission to gain an understanding ofthe field operations through basic researchnotes and journal entries posted by theexpedition team and automatic data.

3.3 Novice User Information andVisualization

As a public interface for a teleroboticsmission, users could not be expected to haveprior knowledge of telerobotics, remotescience, rover operations, or scienceoperations. The Big Signal site containedbackground information that focused ontelerobotics concepts such as autonomy and

Figure 3.6: Daily reports provided a daily archive of visual information, automaticallygenerated information, human generated information, and science data.

Fig 3.7: Explanation of Antarctic meteorites

9

remote science, mission information, andinformation about Antarctica.

A variety of information types such astext explanation, video clips, and VRML 3Dvisualization were used for explanations (seefigures 3.7 and 3.8).

The “I Robot” scrolling text bar was alsolinked to explanations. Thus as a viewer waspresented with new information due to latebreaking events, clicking the text bar wouldactivate pre-written information about achosen topic.

3.4 Classroom ActivitiesThe three pilot classrooms were chosen

based on the particular systems and missionplan of the Nomad robot. Professionaleducators met with the Big Signal team andthe researchers who built the robot to designclassroom activities based on existing highschool standards set by the state ofPennsylvania for Science and Technology.This approach led to a sustainablerelationship with the teachers so that the BigSignal Interface could be used as a teachingtool over a period of three weeks.3.4.1 Earth Science and Astronomy

Big Signal allowed for telescience bygiving students access to data sets from theAntarctic that were relevant to their course ofstudy. For example, figure 3.9 shows a thinsection of a terrestrial rock that was cut insitu by planetary scientist Pascal Lee fromthe NASA Ames Research Center and sent tothe Daily Report section of the interface viasatellite.

3.4.2 Technology EducationTechnology education students

investigated Nomad’s mechanical and

Fig 3.8: VRML rover explanation.

Figure 3.9: Thin section of a rock, usedin Big Signal (Courtesy NASA Ames

Research Center)

Figure 3.10: Students investigate roboticchassis designs

Figure 3.11: High school physicsexperiment

10

electromechanical system. Pre recorded videoof the robot along with diagrams andanimations available on the web interfaceshowed how various functions of the robotworked. Figure 3.10 shows a student teamas they analyze their preliminary design foran expanding chassis.3.4.3 Physics

Physics students in figure 3.11 measurethe footprint of a four-wheeled vehiclesimilar to the robot for their pressure VSarea assignment. Physics students used acombination of live data from the field aswell as prerecorded web archives off BigSignal to fulfill their work.

4 Continuing ResearchSeveral proposed research thrusts in Big

Signal and at the Robotics Institute ofCarnegie Mellon University will greatlyaugment the capabilities of informationinteraction. Big Signal seeks to apply thesetechnologies to future educational and publicoutreach experiences based on informationinteraction.

4.1 Information Request ServicingSystems

Public participants in the Robotic Searchfor Antarctic Meteorites were unable to addinformation to the existing data set. Toparticipate, a user must be able to requestinformation that is currently not in the dataset. However, as observed above, one rovercannot simultaneously handle requests frommany users. Therefore, software must bedeveloped to handle, intelligently andautonomously, requests to fill informationinto the existing data set. This softwarewould analyze the information requests ofmany users to create a mission plan that mostefficiently services all requests. This missionplanner would seek to create a plan that isleast costly with respect to aspects such astime or energy. Techniques developed for thescience autonomy system of Nomad could beused to evaluate the energy and time costs ofa particular information request.

Additionally, the software could takeadvantage of any requests that could beserviced with a single action (for instance,requests for medium resolution images oftwo rocks could be handled by one aerialimage).

This type of software would seamlesslyfill gaps in the existing data set, allowingsmooth information interaction. Unlikecurrent mission planning systems, the verygoal of the robot would be to facilitateinformation interaction.

4.2 Advanced Virtual WorkspacesA proven technology born from early

investigations into the science of human-machine systems is telepresence. Untilrecently, telepresence has been regarded asthe use of low-bandwidth, low-resolutionvideo as a forum to create 2-D imagery forthe remote control of robots in dangerousterrain. Recently, in the Mars Pathfindermission, telepresence evolved, resulting in afull 3-D reconstruction of the landing site.This proved that a full remote experience is acaptivating and indispensable missioncomponent. The resulting 3-D maps wereused by the rover drivers and missionscientists, as well as enjoyed by the generalpublic on the World Wide Web. Virtualreality technologies were advanced to theforefront of telepresence frontiers. Whilethese technologies are currently highlysuccessful, they must now be re-thought andtransformed in order to realize their fullpotential.

Carnegie Mellon, The Iowa GROK Lab,and NASA Ames Research Center seek toaddress some of the fundamental researchproblems in 3-D virtual reality discoveredduring Mars Pathfinder and testing ofChernobyl’s Pioneer. In addition, theypropose to transform the existingtechnologies to the point where constructionof large scale, integrated photo-realisticmodels is possible. These models willincorporate orbital, descent, and ground-based acquisitions into a model of the virtual

11

workspace that exploits the full range andredundancy of information currentlyavailable to the science and operations teams,but in a novel form that is both compact andaccessible.

The use of three-dimensionalvisualization tools creates new opportunitiesfor public outreach, education, and publictelerobotics in a way that was not previouslypossible. Such representations of remote sitesallow the public to experience a mission in auser-friendly way through the activevisualization of mission data within thepublic realm of the World Wide Web.

4.3 Advanced “I Robot” Interface withUser Context

Human communication depends on anunderstanding of context. Human speakerschoose appropriate communicative modesbased on the situation and on the distinctqualities of the audience. Effective speakersimplicitly take into account the intelligence,motivations, knowledge, interests andexperience of their audience. This awarenessallows the speaker to create sentences thatare quickly understandable and contain high-level meaning.

Human/machine interfaces typicallytransmit large quantities of raw, numericaldata because computers excel at very fastand accurate measurements. Users maybecome overwhelmed by large amounts ofdata, or may be distracted by manycompeting sources of information. Althoughthe active field of human-computerinteraction (HCI) has developed datapresentation techniques to ease the use ofhuman/machine interfaces, they do not usecontext to communicate. A “one size fits all”interface that attempts to meet the lowest

common denominator of user will not besufficient for complex future human/machinecooperation. An interface centered on thecontext of a human user rather than onebased on robotic technology must be createdin order to maximize the benefits of human-computer interaction.

Researchers at Carnegie MellonUniversity seek to develop “I Robot” intohuman/robot interface model that createsself-motivated robotic agents designed toinform a wide-ranging user base. It willproactively communicate high-levelinferences and explanations tailored tocontext-specific human goals and interests.The types of users that contact the robot mayrequire different information in differentformats, so explanations will be presented informats appropriate to the needs of particularusers (see figure 4.1).

This objective encapsulates two researchgoals. The first goal addresses the need forrelevant and naturally understandableexplanations. Determining relevance andunderstandability requires that wecharacterize the goals of people that will usethis type of interface. This research willtherefore seek to define three main featuresof an “I Robot” interface user: motivations,areas of interest, and communication formatrequirements. The second research goal willbe to develop an information processingarchitecture that facilitates the kind ofhuman/robot partnerships required to enablethe creation and delivery of roboticexplanations. This interface architecturemust be modular to support many types ofdata sources, such as sensors, as well asinformation destinations, such as graphicaldisplays.

12

0

50

100

Imag

es

Spec

tra

Mag

net

Energy

Time

Info Gain

“I Robot”World Model

Publicweb user

Humanworking

with robotScientist

UserProfile

UserProfile

UserProfile

“I must switchto backup batteries… ”

Figure 4.1: An enhanced “I Robot” interface will serve the communications needs ofmultiple user profiles from a single world model. The same sensor information will be

presented differently depending on a particular user’s needs

13

AcknowledgementsAdditional contributors to this paper and

supporters of the Big Signal Initiativeinclude: the Grable Foundation, theElectronic Information Network, the FrickFund of the Buhl Foundation, the HowardHeinz Endowment, Bob Namestka, DawnLambeth, Scott Thayer, the Robotic Searchfor Antarctic Meteorites, and the FieldRobotics Center at Carnegie MellonUniversity.

References[1] Shillcutt, Kimberly and William

Whittaker, "Modular Optimization forRobotic Explorers," Integrated Planningfor Autonomous Agent Architectures,AAAI Fall Symposium, Orlando, FL,October 23-25, 1998.

[2] "The Atacama Desert Trek - Outcomes,"with D. Bapna, M. Maimone, J. Murphy,E. Rollins and D. Wettergreen.Submitted to the 1998 IEEEInternational Conference on Roboticsand Automation.

[3] "Lessons from Nomad's Trek in theAtacama Desert," DimitriosApostolopoulos, Red Whittaker. 1999International Topical Meeting onRobotics and Remote Systems for theNuclear Industry, Pittsburgh, PA, Apr1999 (accepted)

[4] "Pattern Search Planning and Testing forthe Robotic Antarctic Meteorite Search,"Kimberly Shillcutt, DimitriosApostolopoulos, Red Whittaker 1999International Topical Meeting onRobotics and Remote Systems for theNuclear Industry, Pittsburgh, PA, April1999 (accepted)

[5] "Sensing and Data Classification forRobotic Meteorite Search," LiamPedersen, Dimitrios Apostolopoulos, etal. Proceedings of the XIII SPIEConference on Mobile Robots, Boston,MA, November 1998

[6] "Field Validation of Nomad's RoboticLocomotion," Ben Shamah, DimitriosApostolopoulos, Eric Rollins, RedWhittaker Proceedings of the XIII SPIEConference on Mobile Robots, Boston,MA, November 1998