ApplicationsEditor: Mike Potelhttp://www.wildcrest.com

14 July/August 2006 Published by the IEEE Computer Society 0272-1716/06/$20.00 © 2006 IEEE

U ntil recently, scientists knew little about whatwhales did underwater. Studying the underwater

behavior of marine mammals is difficult; light doesn’ttravel far through water, and divers can’t keep withinvisual range of an animal capable of sustained speedsof 5 knots. Scientists can use sonar technologies toimage marine mammals underwater, but sonar recordsprovide only occasional brief glimpses of whales under-water.

A new collaboration between visualization experts,engineers, and marine biologists has changed this. Forthe first time, we can see and study the foraging behav-ior of humpback whales. This is important not just forpurely scientific reasons. Whales are dying because ofship collisions and entanglements with fishing gear inever-increasing numbers. Understanding their behav-ior could lead to changes in shipping regulations or inthe nature and deployment of fishing apparati.

Our study’s primary objective was furthering the sci-ence of marine mammal ethology. We also had a secondobjective: field testing GeoZui4D, an innovative test-bench for investigate effective ways of navigatingthrough time-varying geospatial data. The study

involved two expeditions, one in 2004 and another in2005, in which we tagged whales and recorded theirmovements.

Recording whale behavior The digital recording acoustic tag,1 shown in Figure 1,

is the key technology allowing us to see underwaterwhale behavior. DTAG is a recording device containingseveral instruments. Three axis accelerometers provideinformation about the gravity vector’s direction, threeaxis magnetometers measure the direction of the earth’smagnetic field, a pressure sensor provides depth infor-mation, and a hydrophone continuously records sound.Except for sound, which is recorded at 64 kHz, all instru-ments record at 50 Hz. DTAG merges and compressesdata records to be stored on 3 gigabytes of flash memo-ry. DTAG is about 10 centimeters long, excluding theantenna, and is attached to a whale’s back using suctioncups. After some preset interval, typically 10 to 22 hours,DTAG automatically releases suction and floats to thesurface. Once it’s on the surface, a ship can locate it byits radio beacon. The ship picks up the DTAG, and sci-entists download the data.

To place the tag, scientists attach it loosely to the tipof a 45-foot carbon fiber pole, mounted (as Figure 1shows) with a gimbal on a rigid hull inflatable boat’s(RHIB) bow. Placing the tag on a whale isn’t easy giventhat a whale usually only surfaces for a few seconds ata time. The operation requires careful timing and coor-dination between the boat driver and pole handler.

Scientists analyze the DTAG data to produce a pseudo-track. This is only an approximate estimate of the ani-mal’s path because DTAG doesn’t give information aboutspeed through the water. Therefore, scientists mustderive the pseudotrack using a form of dead reckoning.The cross-product of the magnetometer vector with thegravity vector (from the accelerometers) provides head-ing information. Together, heading and gravity provide3 degrees of freedom of orientation for the tag.

It’s still necessary, however, to deduce the tag’s place-ment on the whale. This involves selecting a part of thetrack where the whale is at the surface and assumingthat the animal is horizontal and moving forward. Directobservation can verify this assumption. After scientists

Colin Ware,Roland Arsenault,and MatthewPlumleeUniversity of NewHampshire

David Wiley Stellwagen BankNational MarineSanctuary

Visualizing the Underwater Behavior of Humpback Whales __________________________________

1 The digital recording acoustic tag (DTAG) is attached to a whale usingsuction cups. Scientists place the DTAG on the whale from an inflatableboat using a 45-foot carbon fiber pole.

IEEE Computer Graphics and Applications 15

have obtained a continuous recordof whale orientation, they assumethat the animal is moving forwardat a constant velocity except whenit’s diving steeply. In this case, depthchanges from the pressure sensordata provide direct informationabout downward velocity. Scientistscan correct the pseudotrack basedon sightings of the whale when itsurfaces.

We also gathered data about sev-eral vessels in the vicinity. Thisincluded GPS time-series data fromthe US National Oceanic andAtmospheric Association’s NancyFoster and the two RHIBs. Weobtained the positions of fishingboats within a radius of approxi-mately 10 nautical miles using theNancy Foster’s radar.

Replaying whale behaviorwith GeoZui4D

Our first approach to visualizing the data was to con-struct a tool for interactively replaying whale motion,together with sound recorded by DTAG. To accomplishthis, we added features to GeoZui4D, a software appli-cation we developed to investigate methods for inter-acting with time-varying geospatial data.2

As a basic navigation method, GeoZui4D uses whatwe call center of workspace navigation. The user posi-tions the cursor on an object or region of interest andclicks the middle mouse button. This causes the pointto move to the center of the workspace, a point justbehind the screen in the viewing frustum. While holding the button down, the user can scale up or down around that point by moving the mouse. Three-dimensional widgets let the user rotate the scene aroundthe point of interest. A suite of interaction techniquesextend center-of-workspace navigation, and haveproven extremely useful for analyzing DTAG data.

Attaching to a moving object Figure 2 shows the GeoZui4D interface. With a mouse

click, the user can attach the center of workspace to amoving object. This lets the user follow the whale on itspseudotrack and watch its motions while listening tothe various sounds picked up by the DTAG. The user cansimply attach the center of the workspace to the whale,or follow along behind, changing orientation as thewhale changes its orientation. Clicking on a point noton the whale instantly changes the view to a world frameof reference, providing a different perspective and dif-ferent insights. In this case, however, the whale willoften swim out of the view field.

Users can create an extra window through a menuselection and attach it to a moving object using thesame center-of-workspace one-click operation. (InFigure 2, the main window view is attached to the fore-ground whale and the subwindow view is attached toa whale shown in the distance in the main view.) The

second window has several uses. For example, it pro-vides a continuous view of a different whale or tracksthe path of a ship in the vicinity. GeoZui4D also pro-vides range markers showing one object’s relative bear-ing and distance relative to another. These features helpscientists understand shipping’s possible effects onwhale behavior.

We added the capability to play back recorded soundsin such a way that the sounds are in temporal synchronywith various other time-varying objects (such as boats,which are also represented in the scene). This gives sci-entists studying marine mammals an unprecedentedway to watch and listen to whales as they forage.Although ethologists studying land animals take thiscapability for granted, it has not previously been avail-able to those studying marine mammals.

Space-time notesWe implemented space-time notes specifically to sup-

port the ethological analysis of whale behavior. Ourspace-time notes encapsulate the following variables:

■ a point in space (the center of the workspace wherethe note was created) or “look-at” position,

■ a view direction relative to the look-at point,■ a view scale,■ a point in time at which the note was created,■ a playback rate,■ a name for the note (entered by the user), and■ typed-in textual information.

Space-time notes mark and record interesting behav-iors. They’re also effective in presentations because theylet presenters select a few interesting or representativeactions from hours of recording.

The panel on the right-hand side of Figure 2 is thespace-time notes interface panel, which supports notecreation and activation. Fields in the panel support the

2 GeoZui4D showing two tagged whales together with the whale tracks and the tracks ofseveral surface vessels.

Applications

16 July/August 2006

behavioral coding performed by ethologists. Clickingon the “save” button produces a file with a simple XMLtagged format.

TrackPlotAlthough GeoZui4D is an effective tool for observing

whales’ underwater behavior, it suffers from the prob-lem of all visualizations that show time-varying data asan animated replay. Showing instantaneous behaviorsmight not be the best way to reveal patterns occurringon longer temporal scales.

Often, the best way to look at temporal sequence datais to turn it into a spatial pattern. Accordingly, we devel-oped TrackPlot, a tool that uses a custom-designed rib-

bon to represent tracks. TrackPlot effec-tively reveals a variety of behaviors.

The ribbon’s center is the pseudotrackcenter. It’s two meters wide and is twist-ed around the along-track direction toshow roll behavior. A pattern of chevronson the top surface of the ribbon revealstravel direction and gives an additionalorientation cue. Figure 3 shows a ribbonrepresenting several hours of foragingbehavior. As the figure shows, the whaleis either at the surface getting air, or onthe bottom⎯about 90 meters below thesurface⎯searching for food. Ascentsand descents between the surface andthe bottom are steep.

In GeoZui4D replays, we occasional-ly saw cyclic attitude changes that wetook to indicate the whale’s flukestrokes. We added a feature to TrackPlotto make these more visible. We differ-entiated the pseudotrack twice about alateral axis to obtain angular accelera-tions. The sawtooth patterns in Figure4 (red above, blue-green below the dor-

sal and ventral sides of the ribbon) represent these angu-lar values. The sawtooth’s amplitude reveals the amountof angular acceleration at any given instant.

This visual representation reveals several interestingbehaviors. Figure 4 shows a clear difference betweenswimming on ascent and on descent. On ascent, thewhale swims strongly up to within about 20 meters fromthe surface, then glides the rest of the way. Conversely,on descent (only one of which is shown, the third trackfrom the left), the whale swims down for only a fewmeters and glides the rest of the way as its lungs collapseand it loses buoyancy.

In Figure 5, the angular acceleration plot reveals burstand glide swimming, a mode of swimming that con-serves energy. Whales generally use this mode for longertraversals.

Prior to our study, researchers had hypothesized thathumpback whales sometimes forage for sand lance (asmall fish species) by rolling on their sides and openingtheir mouths while at the seabed. One piece of evidenceof this behavior was scarring on some whales along oneside of their mouths.

GeoZui4D showed side rolls near the bottom, asshown by the whale’s orientation in the main windowin Figure 2. But TrackPlot was much better at bringingout patterns of side-roll behavior, revealing that theyoccurred on almost all dives, for all whales, and usual-ly with several per dive. To make side rolls more distinctin the visualization, we color-coded them yellow whenthe roll exceeded 40 degrees. Figures 6a, 6b, and 6cshow side-roll patterns from three different animals. All

3 TrackPlot showing several hours of whale foraging behavior. This is an oblique viewlooking down on the water. The animal is mostly either at the surface or on the bottom,with steep transitions between.

4 The red and green sawtooth patterns above andbelow the ribbon reveal whale fluke strokes.

5 Burst and glide swimming patterns.

IEEE Computer Graphics and Applications 17

of them exhibited this behavior but in the context of sev-eral different foraging patterns. The ubiquitous natureof side rolls leads us to believe that the whales taggedin 2004 were almost exclusively engaged in foraging forsand lance on the seabed.

We found one other foraging pattern in the whalestagged in the 2004 season. One of the whales showedthree patterns we believe represent bubble-net feeding.Figure 6d shows the TrackPlot view illustrating thisbehavior. Here, the whale swims in a circle at about 30meters depth emitting bubbles from its blow hole. Thebubbles form a net containing the fish and the whalethen lunges up with its mouth open to engulf thetrapped fish.

We centered the TrackPlot interface both conceptu-ally and literally on the ribbon at a particular point inspace and time designated the current_point. TrackPlotalways translated this point to the workspace center. Wenavigate forward and backward along the track by tabbing and shift-tabbing, respectively, causing the current_point to move forward and back along the trackwhile remaining locked to the workspace’s center. Userscan rotate the view around the center using the mouseand the left mouse button.

To analyze track sections, we can set a marker at thecurrent_point with a key selection, causing a smallsphere to appear at that point on the track. When wemove the current_point, the marker defines a section of the track (between the marker and the new current_point). We can then subject this section of trackto various kinds of analysis through menu selections.For example, we can automatically compute flukingwavelength or the number of side rolls.

Different behaviorThus far we’ve concentrated on visualizations devel-

oped to help analyze the behavior of whales tagged in2004. In the 2005 expedition, the whales showed verydifferent behaviors.

We thought the whales tagged in 2005 to be foragingmainly for krill, which were abundant at that time.When foraging, they were usually only a few metersbelow the surface, as Figure 7 shows. This behaviormakes them highly vulnerable to ship collisions, because

6 Four examples of foraging patterns: (a) straight lines with side rolls, (b) repeated side-roll loops with inversion,(c) tightly clustered swimming patterns with side rolls, and (d) bubble-net production.

(a) (b)

(c) (d)

7 The pink transparent vertical band shows the whale, foraging for krill, tobe extremely vulnerable to ship strikes from vessels such as the tanker (top)that we spotted in the shipping lanes departing from Boston.

Applications

18 July/August 2006

they’re mostly out of sight just beneath the surface. Toillustrate this vulnerability, we added a pink verticalcolor band running though the whale track to a depth of8 meters. This is about the draft of a modern tanker orfreighter, making it dramatically clear that the whale isspending most of its time in the danger zone.

Future directionsWe plan another tagging project for June 2006. One

thing we hope to accomplish is a better understandingof how the foraging whales interact with the seabed.No detailed maps of the seabed existed for the region ofthe 2004 expedition (it’s a little-known fact that wehave much better maps of the entire surface of Marsthan of 95 percent of the ocean floor). This year, we’llhave a state-of-the-art multibeam sonar on the ship thatwill let us map seabed features, such as large sandwaves, and incorporate them into the visualization.This should help us understand how the whales maneu-ver with respect to the terrain beneath them. In themeantime, we’re continuing our investigation into thebest ways of letting users interact with time-varyinggeospatial data. ■

AcknowledgmentsWe’re grateful to the many scientists and engineers

who gave us the opportunity to participate in this pro-ject. US National Science Foundation grant 0081292and National Oceanic and Atmospheric Associationgrant NA17OG2285 funded Colin Ware’s visualizationwork. The Stellwagen Bank National Marine Sanctuaryfunded the whale-tagging studies.

References1. M.P. Johnson and P. Tyack, “A Digital Acoustic Recording

Tag for Measuring the Response of Wild Marine Mammalsto Sound,” IEEE J. Oceanic Eng., vol. 28, no. 1, 2003, pp. 3-12.

2. R. Arsenault et al., “A System for Visualizing Time-VaryingOceanographic 3D Data,” Proc. Oceans 2004, IEEE Press,2004, pp. 743-747.

Contact author Colin Ware at cware@ccom.unh.edu.

Contact editor Mike Potel at potel@wildcrest.com.

By Donald J. ReiferReifer Consultants Inc.

This guide offers a step-by-step process for estimating software costs. Built on public-domain information and cost models, it gives the formulas software developers need to estimate the effort and time required to design, develop, test, and deliver their software projects. $19 www.computer.org/ReadyNotes

The Poor Person’s Guide to Estimating Software Development Costs

IEEE ReadyNotes