the director’s corner · the director’s corner the next few months will be notable ones here at...

2 SPRING 2002 NAVO MSRC NAVIGATOR

The Director’s Corner

The next few months will be notable ones here atthe NAVO MSRC. We are completing a sweepingseries of center enhancements, designated asTechnology Insertion for FY02 (TI-02), acrossseveral major technology areas within the MSRC.The centerpieces of TI-02 are the acquisition of anew IBM POWER4 HPC system and completionof the NAVO MSRC Remote Storage Facility(RSF). When complete, the TI-02 enhancementswill provide almost 8.5 teraflops of aggregatepeak computing capability with commensuratelybalanced storage and networking capabilities.The RSF will permit us to offer what is perhapsone of the most resilient and best performingHPC storage environments in the world today.This enormous computational capability, coupledwith a sustained 11-year NAVOCEANO focus onsupporting the largest and most demanding DoDcomputational applications, will continue toenable unparalleled advances in the DoD scienceand technology areas served by the HPCModernization Program (HPCMP).

With all of this diverse computational capabilitythat's been fielded across 20+ shared resourcecenters (SRCs) by the HPCMP, it has become crit-

ically important for us to redouble our efforts inassessing and implementing common user envi-ronments, practices, and tools within and acrossthe SRCs. Your continuing individual and collec-tive user feedback makes it clear that you consid-er this to be one of your highest priorities for theSRCs. In response, the SRCs have intensified andmade significant progress toward strategic cross-cutting collaborative efforts in technical areassuch as mass storage and archival, metacomput-ing, HPCMP-wide shared information environ-ments, and security. Here at the NAVO MSRC,we are redoubling our efforts to supplementthose activities by strengthening the links to thenew HPCMP Programming Environment andTraining (PET) program that's more tightlyfocused than ever on user environment, tools,and productivity.

My staff and I look forward to seeing you in Juneat the 2002 HPCMP Users' Conference in Austin,Texas. As always, please take every opportunityto let us know how we can better serve you —your feedback is critically important to us and tothe HPCMP.

About the Cover:

This image was generated by the POP model running on HABU, an IBM RS/6000 SP located at the NAVO MSRC. Thesimulation represents a state-of-the-art eddy-resolving effort that will eventually be used as the ocean component of acoupled global air/ocean/ice prediction system for Navy needs as well as for short-term climate studies.

Steve Adamec, NAVO MSRC Director

Sweeping Series of EnhancementsEngulf NAVO MSRC

3SPRING 2002NAVO MSRC NAVIGATOR

The Naval Oceanographic Office (NAVO)Major Shared Resource Center (MSRC): Delivering Science to the Warfighter

The NAVO MSRC provides Department ofDefense (DoD) scientists and engineers with highperformance computing (HPC) resources, includ-ing leading edge computational systems, large-scale data storage and archiving, scientific visuali-zation resources and training, and expertise in spe-cific computational technology areas (CTAs).These CTAs include Computational FluidDynamics (CFD), Climate/Weather/OceanModeling and Simulation (CWO), EnvironmentalQuality Modeling and Simulation (EQM),Computational Electromagnetics and Acoustics(CEA), and Signal/Image Processing (SIP).

NAVO MSRC Code N71002 Balch BoulevardStennis Space Center, MS 395221-800-993-7677 or [email protected]

NAVO MSRC Navigatorwww.navo.hpc.mil/navigator

NAVO MSRC Navigator is a bi-annual technicalpublication designed to inform users of the news,events, people, accomplishments, and activities ofthe Center. For a free subscription or to makeaddress changes, contact NAVO MSRC at theaddress above.

EDITOR: Gioia Furness Petro, [email protected]

DESIGNERS: Cynthia Millaudon, [email protected] Townson, [email protected] Yott, [email protected]

Any opinions, conclusions, or recommendations inthis publication are those of the author(s) and donot necessarily reflect those of the Navy or NAVOMSRC. All brand names and product names aretrademarks or registered trademarks of theirrespective holders. These names are for informa-tion purposes only and do not imply endorsementby the Navy or NAVO MSRC.

Approved for Public ReleaseDistribution Unlimited

Contents

The Director’s Corner

2 Sweeping Series of Enhancements Engulf NAVO MSRC

Feature Articles

4 Active Swirl and Fuel Modulation to Control Combustion Instability in Gas Turbine Engines

7 NAVO MSRC Visualization Center Supports the Retrieval of the Ehime Maru

11 Knowledge Management: A Case for Intelligent Data Archives

Scientific Visualization

13 Improvements in the Use of Color Wheel Visualization of Global Modeling Data

14 Towards a High-Resolution Global Coupled Navy Prediction System: The Ocean Component

Programming Environment and Training

18 VSIPL/ERI: An Enhanced Reference Implementation of the Vector Signal and Image Processing Standard

20 NAVO MSRC PET Update

The Porthole

22 A Look Inside NAVO MSRC

Navigator Tools and Tips

24 File Transfer Techniques: Transferring Between HABUand Other Systems

Upcoming Events

27 Conference Listings

Combustion-generated pollutants suchas nitrous oxide (N2O), carbon monox-ide (CO), unburned hydrocarbons(UHC), and soot can be reduced signif-icantly if lean burning combustion sys-tems can be used in gas turbineengines. However, as the fuel-air mix-ture becomes lean, small perturbationsin the flow can change the unsteadyheat release pattern, and if the heatrelease is in-phase with the acousticfluctuations in the combustor, the pres-sure (acoustic) fluctuations can growinto high-amplitude, low-frequencyoscillations. These oscillations cancause flame extinction (often calledLean Blowout, or LBO), reduce enginelife span, and/or cause catastrophicstructural damage under extreme cir-cumstances. If these oscillations can becontrolled and even leaner mixturescan be burned stably, then not only canemissions be reduced but the over fuelconsumption can also be decreased sig-nificantly. Since these oscillations occur

in a dynamic manner, passive

control techniques(e.g., using struc-tural changes)cannot deal withall possiblechanges, andtherefore, in orderto suppress theseoscillations, activecontrol strategiesare needed. Thisarticle describes thestudy of two active con-trol methods applied to pre-mixed combustion in a gas tur-bine engine: modulation of theswirl imposed on the incoming fuel-air mixture (which is at a fixed equiva-lence ratio) and modulation of the fuelcontent in the premixed mixture for afixed swirl condition. These studieshave been conducted using a state-of-the-art Large-Eddy Simulations (LES)model developed at the GeorgiaInstitute of Technology.

The first active control technique stud-ied here employs modulation of the

inlet swirl velocity of the incomingpremixed mixture. In a typical

gas turbine engine, swirl isintroduced into the air

from the compressorby swirl vanes (typical-ly small airfoils at highangles of attack) in thepremixer located

upstream of the combustor. Fuel is theninjected into this swirling airflow in thepremixer so that a swirling, premixedfuel-air mixture enters the combustor.Swirling inflow is used in dump com-bustors to exploit a fluid dynamic phe-nomenon called Vortex Breakdown(VB) that makes the flame zone com-pact and stabilizes the flame. As theswirling flow expands into the combus-tor, an adverse pressure gradient isformed in the flow, and this slows downthe axial motion of the mixture. For agiven geometry, when the swirl intensi-ty exceeds a critical value, a recircula-tion bubble is formed (called VB) alongthe centerline of the combustor. Thisbubble acts as an effective bluff body in

4 NAVO MSRC NAVIGATOR

Active Swirl and Fuel Modulation toControl Combustion Instability in GasTurbine Engines Chris Stone and Suresh Menon, Georgia Institute of Technology

Figures 1(a)/(b). Swirl modulation effect on flame-vortexdynamics: (a) high-swirl condition, (b) low-swirl condition.At the high-swirl state, the vortex rings (purple) rapidlybreak down, and the flame surface (gray) is compact. Onthe other hand, at low swirl, the vortex rings are morecoherent, and they drag the flame with them, making theflame longer. No VB occurs when the swirl is low, andtherefore, the flame in the low-swirl case oscillates and isunstable, whereas at high swirl, VB stabilizes the flame.

Figure 1(b)

SPRING 2002

Figure 1(a)


the flow, and along the centerline, theflow from the inlet actually stops (stag-nates) just upstream of the bubble. Thisphenomenon helps to stabilize theflame just upstream of the stagnationpoint. However, in current gas turbineengines, since the swirl vanes are fixed,the swirl introduced into the flow is stat-ic and is typically optimized for onlyone operating condition. This studyexplores how swirl magnitude can beactively changed to stabilize the com-bustion as the fuel air mixture ischanged from rich to lean. The second technique studied here isthe dynamic adjustment of the fuelcontent in the fuel-air mixture. For afixed mass flow rate and a fixed inletswirl intensity, we dynamically changethe fuel-air mixture ratio by changingthe equivalence ratio of the incomingmixture. The goal of this study is todetermine how changing the fuel con-tent (which, in turn, will affect theamount of heat released and peak tem-perature reached) will affect the com-bustion dynamics in the combustor.

The LES methodology is used here tosimulate the combustor response toboth control techniques. In LES, thelarge scales of motion are fully resolvedin both space and time, and only thesmall scales (smaller than the grid reso-lution) are modeled. Premixed com-bustion is modeled using a thin-flamemodel in which a scalar field that repre-sents the propagating flame front isevolved, along with the LES momen-tum and energy transport. To modelthe fuel-air mixture change (in the sec-ond control technique), an additional(conserved) scalar equation is mod-eled, which tracks the change in localequivalence ratio. The local equiva-lence ratio is then used to calculate theburning rate and combustion tempera-ture.

The LES algorithm (LESLIE3D) is par-allel (using Message Passing Interface)and performs very well on a variety ofdistributed (Cray T3E (SEYMOUR),RS/6000 SP3 (HABU), etc.) and

shared-memory parallel platforms (SGIOrigin 2000/3800). In fact, due to itsscalability, LESLIE3D is now a bench-mark code for all Department ofDefense (DoD) High PerformanceComputing (HPC) sites and is beingused to evaluate new systems’ per-formances. The current studyemployed 1.3 million grid cells andrequired approximately 1000 CPUhours (on HABU) for one flow-throughtime. Typically, 10-15 flow-throughtimes are simulated to obtain sufficientdata for statistical analysis.

The first investigation looked at chang-ing the level of swirl imposed at theinlet. In an open-loop study conductedearlier, the pressure fluctuation wasshown to reduce when swirl intensity

was increased. To dynamically repro-duce this effect, a simulation was start-ed with a low-swirl S (0.56), and thenswirl was steadily increased so that Sreached a final value of 1.12. Analysisshows that after a small delay, the pres-sure oscillation amplitude dropped by6.6 decibels (dB). Figures 1 and 2,respectively, show the effect of swirl onvortex-flame dynamics and on themass flow-rate fluctuations. Resultsshow that for a fixed fuel-air mixture,

an increase in swirl stabilizes the flameand reduces the fluctuation in both theinflow mass flow rate and in the pres-sure fluctuations.

The governing physics of combustioninstabilities is quite complicated owingto the fact that not one, but severalphysical processes are coupled and incontention. The dominant process forcombustion instabilities is the interac-tion of acoustic waves (pressure fluctua-tions) with the unsteady heat source (inthis case, the flame surface). When thepressure (p') and heat-release fluctua-tions (q') are "in-phase" (the product ofthe two being positive), energy isadded to the instabilities. As the fuelconcentration is decreased, two impor-tant controlling parameters, the com-

bustion temperature (flame tempera-ture) and the burning velocity (flamespeed), are subsequently reduced. Thenearer one is to the LBO limit, themore pronounced is this effect (SeeFigure 3a). At a lower burning velocity,unsteady flames are more susceptibleto large-scale flow perturbations sincethe recovery time (i.e., the time need-ed to return to the mean or stable

Figure 2. Mass flux measured at the inlet during swirl modulation. At low swirl,the fluctuating mass flux is nearly 100% but drops sharply as the swirl vaneangle is increased (i.e., when the swirl velocity is increased). The pressure fluc-tuation in the combustor is also decreased as the swirl intensity is increased.

Article Continues...


location) is lengthened. In the pres-ent simulation, the magnitude offlame fluctuations is reduced at high-er burning velocities (See Figure 3b).Another contributing factor to theincreased stability is the modificationof the relevant time scales. As thecombustion temperature is altered,the acoustic velocity (speed ofsound) is also modified, resulting in ashift in the phase between the pres-sure and heat-release fluctuations.This effect can be significant in eitherincreasing or decreasing the instabili-ties, depending on the phasebetween the p' and q'. In some com-bustion devices, instabilities are moreintense for higher equivalence ratiosdue to this effect, while, as simulatedhere, the reverse may also occur.

Simulations such as these can beused to understand the complexnature of combustion instability thatis a result of nonlinear interactionsbetween the three modes of wavemotion in a compressible reacting

flow in a confined domain: acousticwaves, vortex motion, and unsteadyheat release due to combustion.Active control of this instability, asdemonstrated here, can then be used

not only to achieve stable combus-tion in a regime that is not possibleotherwise, but also to understandhow the dynamics have beenchanged due to control actuation.

AcknowledgementsThis work was supported in part by General Electric Power Systems and Army Research Office (ARO). Computational supportwas provided by the DoD High Performance Computing Modernization Program Office (HPCMO) at HPC Major Shared ResourceCenter (MSRC) sites (Naval Oceanographic Office (NAVO), U.S. Army Engineer Research and Development Center (ERDC), andArmy Research Laboratory (ARL)), under an ARO Challenge Project.

Figure 3(a)/(b). Instantaneous viewof fuel modulation control. Shownare the flame surface (red isosur-face), vortex rings (gray rings), prod-uct temperature (color contours),and fluctuating pressure signal(shown below the figure). In (a), thefuel mixture entering the combustorhas become lean, and this results ina decrease in temperature (bluecore region), and the pressure oscil-lation increases. This time corre-sponds to the far right of the p'(t)signal. In (b), the inflow fuel mixtureequivalence ratio is increased (richmixture), and as a result, the flametemperature increases, resulting in ahotter core region (red temperaturecontours). At this phase of the con-trol cycle, the pressure fluctuationsbegin to decrease (first quarter ofthe p'(t) signal).

Figure 3(b)

0

5

-5

0

5

-5

Figure 3(a)


On 9 February 2001, a collisionbetween a U.S. Navy submarineand a Japanese fishing trawler sentthe fishing vessel to the bottom ofthe Pacific Ocean about 10 milessouth of Diamond Head,Oahu, Hawaii. The decisionwas made to retrieve the ves-sel, which lay in approxi-mately 1800 feet of water. ACrisis Action Team wasformed, and the NAVOMSRC Visualization Centerwas asked to provide visuali-zation support for the EhimeMaru Retrieval Operation.There are many supportaspects in an operation likethis, but the first challengewas to visualize the high-res-olution model output gener-ated by the Shallow-WaterAnalysis and Forecast System(SWAFS) a three-dimension-al (3D) ocean circulationmodel that produces time-series images with correct

speed and direction, temperature,and salinity fields. Initially, a 2-kmgrid that covered the entireHawaiian Islands was generated, butthis model provided input to a high-

er resolution (500-m) nest, thatbounded the operation area justsouth of Oahu. The model was runon the newest and most powerfulsupercomputer at the NAVO MSRC,

the RS/6000 SP3 (HABU).

Part of the NAVO MSRCVisualization Center mission isthe development of analyticalsoftware environments for theDepartment of Defense (DoD)research community. Some ofthese tools, designed to ana-lyze ocean model output,were modified to analyze theNavy's operational model out-put. These software tools pro-vided significant diagnosticcapability, which assisted inthe validation of the modeloutput in a very complexenvironment.

While virtual environmentsmay not provide all of theanswers to data analysts, the

NAVO MSRC Visualization CenterSupports the Retrieval of the Ehime MaruPete Gruzinskas, NAVO MSRC Visualization Center

Vertical profile of SWAFS over the Ehime Maru.


SWAFS surface currents in the operational area.

Screen capture of SWAFS probe compared tothe ADCP current measurements.

fact of the matter is the real worldenvironment is in 3D. Features with-in the environment have 3D struc-ture that can be difficult, if notimpossible, to realize in two-dimen-sional (2D) space. The same tech-nology that was built to helpresearch and development modelersscrutinize their model outputwas applied to an opera-tional scenario where thespeed and direction ofthe ocean currents werecritical factors.

An additional featureadded to the application,unique to the EhimeMaru retrieval operation,was the display of anAcoustic Doppler CurrentProfiler (ADCP) buoy.The display of the ADCPdata provided analysts adirect comparisonbetween model outputand in-situ current meas-urements in near-realtime. Another significantfeature of the virtualenvironments built for

ocean model analysis is portability— the ability to run on a variety ofhardware architectures, includinglaptop computers.

Portability is accomplished by savvyapplication of graphics techniques,as well as smart Input/Output andmemory management. This ability

was critical to the provision of ananalysis environment and data toforward-deployed personnel on-scene at the recovery site. Othersupport products were developed torender the recovery area in 3D fromhigh-resolution bathymetry (waterdepth) data to delineate critical

areas. Conceptual animationswere built to demonstratethe mechanics of anextremely difficult recoveryoperation.

The operation was success-ful, and all mission objec-tives were accomplished.This recovery operationdemonstrates the excellentsynergy that has developedbetween the operationalNavy and the DoDresearch infrastructure builtby the High PerformanceComputing ModernizationOffice (HPCMO), and is astellar example of theNAVO MSRC realizing theHPCMO maxim of"Delivering Science to TheWarfighter."


Screen capture showing ocean currents at depthover an exaggerated (3:1) model of the Ehime Maru.

Another snapshot of currents at depth. Note the leg-end has been "tuned" to accentuate faster currents.

3D shaded relief of OPAREA using high-resolutionbathymetry collected by the USNS SUMNER. Imageryshows where the ship was resting, the shallow-waterretrieval area, and the final relocation site.

Shallow-Water Recovery Site

21º 17.52’ N

157º 56.40’ W

Deep-Water Recovery Site

21º 04.85’ N

157º 49.46’ W

Speed (cm/s)

ParticlePick On

ParticlePick On

02 days22 hours59 minutes

0 45 Speed (cm/s)0 45

02 days22 hours59 minutes

Final Relocation Site

21º 1.00’ N

158º 7.86’ W


Still image showing Ehime Maru retrieval over realbathymetry data. The animation was designed to concep-tualize/visualize what the recovery would look like in thecontext of the real data collected by the USNS SUMNER.

Still image from conceptual animation showing the lift pro-cedure. A conceptual animation was generated depictinghow the towing harness would be attached to the EhimeMaru, which rested in approximately 1800 feet of water.

Still images from conceptualanimation showing retrieval ofthe Ehime Maru by the vesselRockwater.

Oceans 2002Conference & Exhibition

October 29-31

Mississippi Coast Coliseum & Convention Center

Biloxi, Mississippi

www.OCEANS2002.com

Proposed technical sessions at the OCEANS2002 MTS/ IEEE Conference wi l l focus on thefo l lowing areas:

I. Advanced Marine Technology

II. Marine Resources

III. Ocean and Coastal Engineering

IV. Marine Policy and Education

V. Fisheries Technology

VI. Information Technology

VII. Ocean Modeling

VIII. Integrated Ocean Observing Systems

IX. Communications and Navigation

X. Underwater Acoustics Technology

XI. Non-acoustic Signal and Image Processing

XII. Shallow Water Environmental Technology

The technical program will include a studentposter competition and tutorial sessions. An excit-ing exhibition featuring innovative marine technol-ogy products and services will be held at theMississippi Coast Coliseum & Convention Center.Exhibitors will be given passes to attend the technical sessions.


Knowledge Management: A Case forIntelligent Data ArchivesDouglas D. Cline, Ph.D., Program Manager, Northrop Grumman Information Technology

There is little doubt that one of themost significant challenges facing theDepartment of Defense (DoD) HighPerformance ComputingModernization Program today is intel-ligently dealing with the sheer magni-tude of data that are created, collect-ed, analyzed, and manipulated

throughout the MSRCs andDistributed Centers (DCs). Increasingcomputational capabilities, as demon-strated by faster HPC systems andlarger system memories, enable high-fidelity simulations that provide criticalinformation to support the nation'swarfighters. In addition, throughadvances in sensor technologies, sci-entific data are being collected andarchived at an unprecedented rate.These two factors create what isknown as "infrastress" — a fundamen-tal stress applied to the current systemsand data storage infrastructure. As of 1March 2002, the NAVO MSRC dataarchive surpassed 375 terabytes (TB),with a projected growth rate ofapproximately 14 TB per month.

As the data archive approaches thepetabyte range, it is clear that datamanagement initiatives must beundertaken by the DoD to createstructured, intelligent data archivesthat provide a broad range of func-tionality. Such initiatives will provedifficult to implement without first

empowering end-users with a suite ofhigh-level data management tools fororganizing, browsing, and accessinglong-term archival data. Developmentof these tools will require new per-spectives on the MSRC program toleverage existing storage architectureswith newer, more robust technologies.

TRADITIONAL ARCHIVAL STORAGE

Archival storage provides the meansto preserve large-scale computationaland experimental data indefinitely. Aconceptual framework for a traditionalarchival storage system is shown inFigure 1, with "Data" representing thefoundation of the system. The variousinfrastructure elements — tape silos,tape media, Hierarchical Storage

Management (HSM) software, net-working, and fileservers — are all crit-ical components of the system.Scalability is essential if future capaci-ty demands are to be met throughincremental expansion of the compo-nents. As data capacity expands withtime, so do related storage costs. If thegrowth in data storage is exponential,then, arguably, storage costs will alsogrow exponentially. Over time, "datainertia" can become so great that theflexibility needed to migrate the activearchive to newer, lower cost storagetechnologies is problematic at best.

From an end-user's perspective, thetraditional archival system is inherent-ly limited when supporting higherlevel access to data. Generally, accessto data stored in the archive is low-level via HSM-specific commands. Forexample, get /path/file, put/path/file,and copy /path/file are all command-driven directives for accessingarchived data. Each directive requiresspecialized knowledge of the HSMcommands (e.g., get, put, copy) aswell as a specific pathname thatpoints to a particular file (e.g.,/path/file). This knowledge is typicallylimited to a single user or small com-munity where the specialized knowl-edge has been shared. An individualoutside this group, not privileged withthis special knowledge, would find itvery difficult to locate the data set,much less extract useful informationfrom it. Therefore, one of the primarylimitations of the traditional storagearchive is its inability to support"Information Discovery," a criticalmissing capability for long-term per-sistent data storage.

Figure 1. Conceptual Framework for the Current MSRC Archival StorageSystem.


KNOWLEDGE MANAGEMENT SYSTEMS

By leveraging the MSRC's currentstorage infrastructure, higher levelfunctionality can be achieved withnominal investment to create a morerobust, intelligent data archive. For thepurposes of this article, the more gen-

eral type of archival storage willbe referred to as a KnowledgeManagement System.

In principle, the KnowledgeManagement System should be capa-ble of supporting InformationDiscovery, including browsing, search-ing, and retrieving archived data. Theconceptual framework of a KnowledgeManagement System is shown inFigure 2. All essential infrastructureelements of the archival storage systemare present, as in Figure 1. However,a new foundation layer is introducedin our Knowledge framework,Middleware, which supports a set ofhigher level infrastructure elements.

The Middleware provides a softwarelayer to interpret high-level dataaccess functions into low-level HSM-specific commands. Moreover, it actsas an interface between sophisticated

software tools and applications to iso-late them from the implementationdetails of the underlying storage archi-tecture.

Middleware typically consists of ObjectBrokers such as Common ObjectRequest Broker Architecture

(CORBA), or, more specific to storagearchitectures, the Storage ResourceBroker (SRB), developed at the SanDiego Supercomputing Center. In theKnowledge Management framework,changes to the storage architecturecan be introduced without impactingthe functionality of user tools andapplications, thereby protecting soft-ware investments and preservingoperational continuity.

Of critical importance to theKnowledge Management System ismetadata, or "Data about the Data."The metadata represents a databaseof data attributes, with linkages tophysical file locations in the archive.Data attributes may contain informa-tion such as data types, data formator representation, data descriptors,and date of creation. It is through themetadata that higher level functions

such as Information Discovery aresupported, by referencing data filesthrough their data attributes ratherthan physical file location.

BEYOND DATA ARCHIVES - DIGITAL LIBRARIES

This article has examined the basicframework for two storage systems —a traditional storage archive represent-ing the MSRC's current architectureand a more robust framework repre-senting a Knowledge ManagementSystem. The Knowledge ManagementSystem supports a broad range ofdata access tools and applications tofacilitate data manipulation fromremote clients. The key aspect of theKnowledge Management System isthat it leverages the MSRC's existingstorage infrastructure (i.e., silos, file-servers, and networks) by introducingMiddleware to deliver a new class ofhigher level data services to the end-user.

Beyond these simple frameworks liemore sophisticated storage architec-tures. Digital libraries offer the poten-tial to create online knowledge centersfor discipline-specific application areasthat are important to the DoD. Theselibraries generally provide sharedinformation resources, coupled withhigh-speed networks, to deliver high-level information content directly tothe desktop. Client-based tools andapplications can be specificallydesigned to enable data manipula-tions for analysis and interpretation.

The technical challenges posed bylarge-scale data management pres-ents, in reality, new opportunities for infrastructure development within theDoD modernization program. Inexploiting these opportunities, new requirements and capabilities will bedefined that will ultimately deliver anew class of data services for users ofthe MSRCs and DCs.


References

1. Chaitanya Baru, Reagan Moore, Arcot Rajasekar, and Michael Wan, "The SDSC Storage Resource Broker," Proc. CASCON'98 Conference, Nov. 30-Dec. 3, 1998, Toronto, Canada.

Figure 2. Conceptual Framework for the MSRC Knowledge ManagementSystem.

The color wheel method of two-dimen-sional (2D) vector field visualizationuses the Hue-Saturation-Value (HSV)color space to map vector direction tohue (color) and magnitude to saturationand value. This visualization techniqueis better than traditional graphics primi-tives (arrows) in high-resolution modelsbecause it offers less visual congestion.In addition, the color wheel techniquemore effectively displays eddy forma-tion and vector magnitudes in circula-tion models. Two major changes to the initial colorwheel technique, developed atMississippi State University as a compo-nent of the High PerformanceComputing Management OfficeProgramming Environment andTraining Program working within theClimate, Weather, and Ocean Modelingarena, are used in POPFlow (an inter-active graphical viewer of ParallelOcean Program (POP) modeling data)to achieve interactive graphics perform-ance. First, the Alpha channel is linkedto the vector magnitude. Normalizingvector magnitude gives a direct correla-tion to transparency or opacity. As the

magnitude of the vector approaches themaximum, the pixel contains morecolor, and vice versa. Second, a simplercolor lookup table is generated. TheHSV colorspace complicates calcula-tions with converting the saturation andvalue components to the Red-Green-Blue (RGB) colorspace, the de factocolorspace in OpenGL. The hue com-

ponent has 360degrees directly cor-related to direction(i.e., red is 0 degreesand is due east).Previous color wheeltechniques eitherconverted the HSVto RGB or used sim-pler color wheellookup tables. ThePOPFlow versionuses 360-colorentries convertedfrom the HSV color-space. POPFlow isbuilt on the OpenGLand GLUT applica-tion programminginterfaces (API) with

hardware alpha and blending. ThePOPFlow application using the APIs,and the color wheel optimizations easilyshow a year’s worth of vector data,allowing the user to interact with ease. Color wheel optimizations in POPFlowshow impressive global animations ofmajor effects like the Gulf Stream cur-rents, the Kuroshio currents off the east-ern coast of Japan, and equatorial stria-tions. Figure 1 is a global image of thePOP model using the original colorwheel method. Figure 2 shows theenhanced color wheel technique in thePOPFlow application. Notice that inPOPFlow, the alpha blending of thecolor wheel layer does not occlude thebathymetry below. In addition,POPFlow has interactive adjustmentsfor vector magnitudes to allow users toset threshold wheels that serve to filterthese magnitudes. Future efforts will include dynamic colormap manipulation and incorporation ofa similar technique to show convectionin fluid flow models. For more information please contact the NAVOMSRC Visualization Center staff [email protected].


Figure 1. Global image of POP model data diplayed using the original color wheel method.

Figure 2. Demonstration of the enhanced colorwheel technique used in the POPFlow application.

Improvements in the Use of Color WheelVisualization of Global Modeling Data Ludwig Goon, NAVO MSRC Visualization Center


The vision for Navy meteorological and oceano-graphic forecasting is that of a high-resolutionglobal coupled air/ocean/ice prediction systemwith very-high-resolution regional air/ocean cou-pled models nested into the global system at keylocations. The development of such a coupledprediction system is a computational Challengeproblem requiring commensurate supercomput-ing resources. Towards this goal the authors arespinning up an eddy-resolving global oceanmodel; upon completion it will be delivered tothe Fleet Numerical Meteorology andOceanography Center (FLENUMMETOCCEN)for future transition to the operational environ-ment.

Any ocean model to be used in such a predictionsystem must be capable of simulating high-fre-quency (days to several months) and short-scale(10 to 1000 km) processes. In particular, it mustbe capable of producing realistic mean and vary-ing surface and thermocline flows. High horizon-tal and vertical resolutions are required to pro-duce not only mesoscale variability, but also thecharacteristics of strong, narrow currents such asthe Gulf Stream. Additionally, the model must becapable of producing realistically varying thermo-haline structures in the upper water column andtherefore requires a mixed-layer parameterization.Satisfying these requirements is computationallydemanding; for this simulation we have aDepartment of Defense (DoD) High Performance

Computing Modernization Office (HPCMO)Challenge Grant. Its execution is taking place onthe Naval Oceanographic Office's RS 16000 SP3(HABU).

OCEAN MODEL DESCRIPTION

The ocean model used in this study is the LosAlamos National Laboratory (LANL) ParallelOcean Program (POP). It is a primitive equationz-level model with a free-surface boundary condi-tion. Approximations to governing fluid dynamicsequations permit a decoupling of the model solu-tion into barotropic (vertically averaged) andbaroclinic (deviations from vertically averaged)components; these are solved using an implicitelliptic scheme and an explicit parabolic equationsystem, respectively. It is written in Fortran90 andis designed to run on multi-processor machinesusing domain decomposition in latitude and lon-gitude. Message Passing Interface (MPI) is usedfor inter-processor communications on distributedmemory machines and Shared Message Interface(SHMEM) on shared memory machines.Benchmarking shows the code to be highly scala-ble onto a large number of processors, providedthe processor sub-grid is large enough.1 Furthertechnical details and references regarding thecode and its adaptation for massively parallel



computers can be obtained fromhttp://climate.acl.lanl.gov.

A global 0.1-degree, 40-level configuration ofPOP is being spun-up on a displaced pole grid(See Figure 1), whereby the North Pole is rotatedinto Hudson Bay to avoid a polar singularity. Thegrid consists of 3600x2400x40 grid points, andthe spacing is about 11 kilometers (km) at theequator, decreasing to about 3 km in the ArcticOcean. At mid-latitudes this spacing is 5-7 km.A blended bathymetry was created from Smithand Sandwell,2 International Bathymetric Chartof the Arctic Ocean (IBCAO),3 and BritishAntarctic Survey (BEDMAP)4 products. Allimportant channels and sills were checked andmodified to facilitate correct flow.

The model was initialized from an ocean statedepicted by temperature and salinity from theNavy's 1/8-degree January Modular Ocean DataAssimilation System (MODAS) climatology out-side of the Arctic and the University ofWashington's Polar Hydrography winter clima-tology in the Arctic (http://psc.apl.washington.edu/Climatology.html). To produce

an energetically realistic ocean, synoptic surfaceforcing is used whenever possible during a 20-year adjustment of the model to its initial state.This forcing was largely constructed fromNational Center for Environmental Prediction(NCEP) fluxes for 1979-1998.5 A mixed layer for-mulation is active.6 To date, about a decade ofthe simulation has been completed.

THE GLOBAL SIMULATION

The realistic simulation of sea surface tempera-ture (SST) will be important both to the predic-tion of fronts and eddies and for coupling to theice/atmosphere components of the future predic-tion system. A snapshot of temperature in theuppermost model level on 1 February 1987 isseen in Figure 2. To display the whole globe at thefull model resolution, the field is plotted in modelcoordinates; hence, the distortion in the northernhemisphere. Well-defined frontal structures associ-ated with the western boundary currents

and tropical instability waves are observed.The large meander of the Kuroshio Currentseen to the south of Japan is one of itsknown meander states at this location;movie loops indicate that the model alsoreproduces the other observed states.Mesoscale features are apparent in allocean basins; particularly striking are theAgulhas Current eddies that are shed southof Africa.

To more fully appreciate the mesoscaleactivity in the model, the flow in the upperlevel of the model is seen in Figure 3. Herethe direction and speed of the flow arerepresented by color and hue, respectively.

The Agulhas Retroflection Current system is enlarged tohighlight the shedding of eddies, their sense of rotation,and their northwest movement across the South AtlanticOcean. The characteristic reversing zonal flows of thetropics can be easily identified along with the associatedstanding meander patterns of the North EquatorialCountercurrents in the western Atlantic and Pacificoceans. Rich eddy fields are associated with the otherboundary flows and the Antarctic Circumpolar Current(ACC) in the Southern Ocean.

To understand the interactions between thelarge-scale flows of the Western Pacific, theAsian marginal seas, and the IndonesianThroughflow, a snapshot of surface velocityfrom 1 June 1987 is displayed (See Figure4). This is a challenging region for an oceanmodel to simulate realistically, and is there-fore a good test of the model's capabilities.Again, direction and strength are indicatedby color and color intensity, respectively.Grid distortion is apparent in the northernpart of the domain, particularly in theKuroshio Extension. The flow in this regionis complex and punctuated by mesoscaleactivity; regardless, details are in qualitativeagreement with observations.7

The North Equatorial Current (NEC), iden-tified by the northwestward (blue) flow off-shore of the Philippines, bifurcates near the


Figure 1. Spacing (km) of 0.1°, 40-level globalPOP grid.

Figure 3. Snapshot of surface velocity from the uppermostmodel level (5 m) on 1 January 1983. Color and hue representcurrent direction and intensity, respectively. The expandedregion demonstrates eddy shedding by the AgulhasRetroflection.

Figure 2. Snapshot of temperature in the uppermost model level(5 m) on 15 January 1987.

Sea Surface Temperature ( ºc)

A v e r a g e G r i d S p a c i n g ( k m )2400

2100

1800

1500

1200

900

800

300

600 1200 1800 2400 3000 3600

2.5 3 3.5 4 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.55 6 7 8 9 10 11 12

0 5 10 15 20 25 30


coast to flow northwestward as thePhilippines Current and southwardas the Mindanao Current. TheKuroshio is identified as the strongnortheastward current off Taiwan; tothe north an intrusion of the currentis seen in the East China Sea. TheTsushima Current enters the JapanSea and continues in a northeast-ward direction along the coast ofJapan. The Mindanao Current turns southeastward to enter the CelebesSea, and southward flow is seenbetween Borneo and Sulawesi. Flowinto the Indian Ocean is seenthrough all the Indonesian straits tothe south of these islands. The circu-lation in this region is dominated bythe seasonal monsoons, and south-ward flow into the Indian Ocean isconsistent with observations at thistime of year.8 In February, the sur-face flow reverses, and the net flowis away from the Indian Ocean. TheIndonesian Throughflow is the majorlow-latitude conduit of propertiesbetween the Indian and Pacificoceans and is therefore important onboth short-term and climatic timescales. Quantitative evaluations andanalyses of the spin-up are under-

way using data sets with global ornear-global coverage such as thoseobtained from satellite altimeters,surface drifting buoys, and verticalprofilers. We are comparing modeledand observed mean flows, transportsthrough key passages, energy levels,intrinsic scales, and mixed layer vari-ability, among others.

In this manner we are able to assessthe progress of the spin-up and

make adjustments as appropriate.Following the 20-year spin-up, adecade-long simulation starting inthe 1990s will be performed usinghigh-frequency Navy OperationalGlobal Atmospheric PredictionSystem (NOGAPS) surface forcing.Further details of the model run,benchmarking, results, and movieloops of the spin-up are available athttp://www.oc.nps.navy.mil/navypop.

References1. McClean, J.L., W. Maslowski, and M.E. Maltrud, 2001: Towards a Coupled Environmental Prediction System, ComputationalScience - ICCS 2001 (Eds: Alexandrov, Dongorra, Juliano, Renner, and Tan), Lecture Notes in Computer Science 2073,Springer-Verlag, pp 1098-1107.2. Smith, W.H.F., and D.T. Sandwell, 1997: Global seafloor topography from satellite altimetry and ship depth soundings,Science, 277, p. 1957-1962. http://topex.ucsd.edu/marine_topo3. Jakobsson, M., N.Z. Cherkis, J. Woodward, R. Macnab, and B. Coakley, 2000: New grid of Arctic bathymetry aids scientistsand mapmakers, EOS Trans., American Geophysical Union, 81 (9).4. British Antarctic Survey (BEDMAP), http://www.antarctica.ac.uk/bedmap5. Doney, S.C., S. Yeager, G. Danabasoglu, W.G. Large, and J.C. McWilliams, 2002: Modeling global oceanic interannual vari-ability (1958-1997): Simulation design and model-data evaluation. "J. Climate," submitted.6. Large, W.G., J.C. McWilliams, and S.C. Doney, 1994: Oceanic vertical mixing: a review and a model with a nonlocal bound-ary layer parameterization. Rev. Geophys., 32, 363-403.7. Tomczak, M., and J.S. Godfrey, 1994: Regional Oceanography: An Introduction. Pergamon Press, New York, 422 pp.7. Figure8.6; http://www.es.flinders.au/~mattom/regoc/text/8circ.html8. Gordon, A.L., and J.L. McClean, 1999: Thermohaline stratification of the Indonesian Seas-model and observations. J. Phys.Oceanogr., 29, 198-216.

AcknowledgementsThis work is supported by the Office of Naval Research, the National Science Foundation, and the Department of Energy (CCPPProgram). Computer time is provided through the Department of Defense High Performance Computing Modernization Office atthe Naval Oceanographic Office (NAVOCEANO) as part of a Grand Challenge Award. The NAVO MSRC Visualization Centerdeveloped the applications used for the representation of the velocity fields. Detelina Ivanova (NPS) created the graphics.

Figure 4. Snapshot of surface flow on 1 June 1987 in the WesternPacific, the Asian marginal seas, and the Indonesian Throughflow.


As a result of the NAVO MRSCProgramming and EnvironmentalTraining Program (PET) Tiger TeamCollaboration Final Report,1 the cre-ation of a new performance VectorSignal and Image Processing Library(VSIPL)2 that interfaced with opti-mized signal processing and linearalgebra libraries was assigned byPET to MPI Software Technology Inc.(MSTI). The goal of the four-monthproject was to begin with the TacticalAdvanced Signal Processing (TASP)reference implementation of VSIPLand to "wrap" function calls to theLinear Algebra Package (LAPACK),Basic Linear Algebra Subroutines(BLAS), and Fastest FourierTransform in the West (FFTW) opti-mized libraries. This new softwarewas entitled VSIPL EnhancedReference Implementation(VSIPL/ERI).

BACKGROUND

VSIPL, a community standard, con-solidated and streamlined existingmathematical libraries and definedstandard functions for the scientificand engineering community. VSIPLfunctions include Linear Algebra,Signal Processing (i.e., Fast FourierTransform (FFT) Filters), and Scalar,Vector, and Matrix functionality. Themain features are its portability,object-based description, and run-time modes for both developmentand production. The library also fea-tures opaque objects such as blocks,views on the blocks (i.e., vectors,matrices, and tensors), explicit mem-ory/algorithm hints, and public andprivate data arrays. As an activemember of the VSIPL forum, MSTIensures that its current implementa-tions of the libraries conform to theVSIPL standard.

As a baseline of what is available forembedded computing, MSTI's com-mercial implementation of VSPILuses altivec and assembly languageoptimizations to achieve high per-formance on element wise and signalprocessing functions. It containsadvanced C++ programming

concepts, and the source is actually aC++ implementation with C APIs.MSTI supports the VSIPL CoreProfile for several operating systemsand the PowerPC (G4) and Intel(PIII) processors. However, such software has not, as yet, been madeavailable for the MSRCs’ currentplatforms. Thus, an interim solution for higherperformance VSIPL is required.VSIPL/ERI3 satisfies this requirement.In particular, VSIPL/ERI is freelyavailable middleware for UnitedStates (U.S.) GovernmentDepartment of Defense (DoD) MSRCsites. The project requirements were

to enhance the performance of theTASP4 VSIPL reference implementa-tion by using commercial-off-the-shelf (COTS) lower level libraries,instead of the TASP kernels.

This was accomplished by replacingsections of the TASP VSIPL referenceimplementation with wrappers to

BLAS, LAPACK, and FFTW. Suchlibraries are available on many sites.This work involved the identificationof the appropriate functions in thelibraries to wrap, the creation andextraction of pointers to the appro-priate data from the VSIPL datastructures, compliance with VSIPLdata alignment restrictions (withregards to memory management),and the resolution of problems asso-ciated with the C-to-Fortran inter-face. The project required the port ofthe VSIPL/ERI software to four plat-forms: IBM AIX (Power 3);SOLARIS/SPARC; Linux (x86); andMIPS (SGI, R10K). Currently, the

VSIPL/ERI: An Enhanced Reference Implementationof the Vector Signal and Image Processing StandardGary Boudreaux and Dr. Tony Skjellum, MPI Software Technology, Inc.


library has been deliveredfor SPARC, Linux, and AIX.MSTI has been grantedaccess to the NAVO SGImachine, and the entireproject was completedbefore 15 September 2001.

RELATED TECHNOLOGIES

The VSIPL/ERI library uti-lizes the FFTW library for allFFT functions, LAPACKlibrary for solving advancedsystems of linear equations,and the BLAS library forbasic linear algebra toenable it to exceed TASPVISPL performance.

FFTW4, developed by Dr.Matteo Frigo and Steven G.Johnson at theMassachusetts Institute ofTechnology (MIT), is a Csubroutine library for com-puting the Discrete FourierTransform (DFT) in one ormore dimensions, of bothreal and complex data, andof arbitrary input size. Thelibrary is known for its fastperformance and excellentportability between architec-tures. The latest official ver-sion is 2.1.3, and in 1999FFTW received the J. H.Wilkinson Prize forNumerical Software.5

In addition to FFTW,VSIPL/ERI employsLAPACK5 routines. As aFortran-77 subroutine

library, the well-knownLAPACK ApplicationProgramming Interface(API) routines providesolutions to systems ofsimultaneous linear equa-tions, least-squares solu-tions of linear systems ofequations, eigenvalueproblems, and singularvalue problems. In addi-tion, LAPACK alsoincludes other matrix fac-torizations such as LogicalUnit (LU), Cholesky, QR,and SVD, and its library isequipped to handle bothdense and banded matri-ces.

For all routines, LAPACKprovides functionality forreal and complex matricesin both single and doubleprecision. These routinesare written so that themaximum amount of com-putation is handledthrough calls to the BLAS.LAPACK was designed toexploit the Level-3 BLAS6

— a set of specificationsfor Fortran subprogramsthat perform various typesof matrix multiplicationand the solution of triangu-lar systems with multiplesright-hand sides. Thecoarse granularity of theLevel-3 BLAS operationscauses it to promote highefficiency on many high-

References1. NAVO MRSC PET Tiger Team Collaboration Final Report http://www.sdsc.edu/PET/NAVO/reports/vsip/VSIPL_final_report.html2. VSIPL http://www.vsipl.org3. VSIPL/ERI http://www.mpi-softtech.com/VsiplERI4. TASP http://www.tasp.org5. FFTW http://www.fftw.org6. LAPACK/BLAS http://www.netlib.org

ContactsGary Boudreaux, [email protected] and Dr. Tony Skjellum, [email protected]

Article Continues Page 26...


NAVO MSRC PET UpdateEleanor Schroeder, NAVO MSRC Programming Environment and Training Program(PET) Government Lead

The new PET program is approaching the end of its first year.As with any new start, it has been a tremendous learning expe-rience for all involved.

At the NAVO MSRC, as reported in the last issue of theNavigator, we are primarily responsible for 3 of 15 PETFunctional Areas (FAs). Those three areas are the twoComputational Technology Areas (CTAs) of Climate, Weather,and Oceanography Modeling (CWO) and EnvironmentalQuality Modeling and Simulation (EQM). The NAVO MSRCPET program is also responsible for taking the lead on thecross-cutting functional area of Computational Environments(CEs). More information about these functional areas can befound on the High Performance Computing ModernizationProgram Office (HPCMO) web site under the PET link.

Most of our positions have been filled or are in the process ofbeing filled. We have had a few changes, so the following is awho's who at the NAVO MSRC PET program:Government Lead: Eleanor SchroederComponent Point of Contact (POC): Dr. George Heburn

o Executive Assistant: Tammy Smitho Technologists: Andrew Schatzle and Brian Tabor

CWO FA POC: Dr. Jay Boisseauo CWO On-site (NAVO MSRC): Dr. Tim Campbello CWO On-site US Army Engineer Research and

Development Center (ERDC): Dr. Phu Luongo CWO On-site (Monterey): Applicant being inter-

viewedEQM FA POC: Dr. Mary Wheeler

o Environmental Quality Modeling and Simulation (EQM) On-site (ERDC): Drs. Lea Jenkins and Victor Parr (interim)

o EQM On-site (NAVO MSRC): Position to be filledCE FA POC: Dr. Shirley MooreCE On-site (NAVO MSRC): Dr. Tom Cortese

The three FA POCs have submitted the PET strategies foreach of their functional areas. Those strategies can be foundin the sidebar to this article.

One of the highlights of PET is the refurbishing (many thanks tothe NAVO MSRC) of the equipment in our PET classrooms. Wenow have 20 new single-processor Pentium 4 workstations oper-ating at 1.8 gigahertz with 1 gigabyte of memory and a 20-giga-byte removable hard drive, which gives us the capability of host-ing either Linux-based or Windows-based classes. The equip-ment was given its test run at the HPCMO-sponsoredService/Agency Approval Authority meeting that we hosted herein February.

One final note, PET will undergo its first review in May 2002 inVicksburg, MS.

CWO STRATEGY

DR. JAY BOISSEAU, UNIVERSITY OF TEXAS, AUSTIN, CWO FA POC

The MOS team supports the DoD climate/weather/ocean(CWO) modeling community in the new PET program. TheCWO support strategy was developed by the MOS team in thePET recompetition and has been refined during the programwith input from many DoD users. The MOS team for CWOincludes:

❏ George Heburn (Mississippi State University (MSU)),Component Lead for the MOS team overseeing CWO,EQM, and CE

❏ Jay Boisseau (Texas Advanced Computing Center(TACC)/University of Texas (UT) at Austin), coordinatorfor MOS CWO support activities

❏ Tim Campbell (MSU) and Phu Luong (ERDC), on-siteCWO support staff

❏ DoD CWO User Advisory Panel: Alan Wallcraft (NRL-Stennis), Jane Smith (ERDC), Rich Hodur (NRL-Monterey), and Frank Ruggiero (AFRL), plus the CTAleader for CWO, Bill Burnett (CNMOC)

The PET CWO support strategy is focused on the develop-ment, implementation, and support of advanced computingtechnologies that are needed to enhance the current and nextgenerations of strategic CWO applications. Since the DoDCWO community is among the world leaders in atmosphericand ocean physics, the PET CWO support strategy is focusedon advanced computing technologies. Many of these technolo-gies are critical for enhancing the performance, scalability, andmodeling capabilities of strategic CWO applications and foranalyzing the resulting data. The MOS team supports thedevelopment and direct application of these technologies toCWO research problems and the integration of these tools intoCWO codes and research infrastructure. Some particular activ-ities requested by DoD users include: (1) porting codes to thenewest architectures and optimizing them for maximum per-formance and scalability, (2) evaluating various grid/meshtechniques (unstructured grids for coastline, different verticalcoordinate systems) for accuracy and performance, and (3)


developing and implementing code coupling techniquesfor ocean codes with atmospheric and/or ice models, butalso for integrating other kinds of models (aerosol trans-port, biology/chemistry, etc.)

CE STRATEGY

DR. SHIRLEY MOORE, UNIVERSITY OF TENNESSEE,KNOXVILLE, CE FA POC

The CE team is led by the Innovative ComputingLaboratory (ICL) at the University of Tennessee-Knoxville(UTK). ICL, under the leadership of DistinguishedProfessor Jack Dongarra, is a world-recognized leader inmany areas of high performance computing (HPC). TheCE FA POC is Dr. Shirley Moore, who is the AssociateDirector of Research for ICL. Dr. Moore is assisted by Dr.David Cronk, a research scientist with ICL, and by Dr.Thomas Cortese, the onsite CE lead at NAVO MSRC.Other highly qualified personnel at UTK and other PETteam institutions are involved in CE projects.

The CE vision is to improve user productivity and applica-tion performance through the provision of effective parallelprogramming tools as part of a consistent well-document-ed CE. Achieving this goal involves a number of activities.CE staff are collaborating with Shared Resource Center(SRC) systems staff to implement a CE across the SRCsthat is as consistent as possible. The CE includes compil-ers, message-passing libraries, debugging and performanceanalysis tools, and data management and visualizationtools. CE staff are working to ensure that all componentsof the environment are operating correctly in the SRCbatch-queuing environments and that they are adequatelysupported and documented. Cross-platform tools are beingmade available wherever possible so that users do notneed to learn a different tool interface for each platform.These tools include the TotalView debugger, VampirMessage Passing Interface (MPI) performance analysis tool,and KAP/Pro Toolset for OpenMP programming, all ofwhich are commercial tools. Also included is the cross-plat-form library Performance Application ProgrammingInterface (PAPI) to hardware performance counters, as wellas freely available performance analysis tools for sites andusers who do not have access to commercial tools. Theenvironment will be updated regularly to keep pace withchanges in architectures and operating systems.

CE core support consists of ongoing activities in the areasof assessing and meeting user needs and delivering effec-tive training in HPC architectures and parallel program-ming tools. Core support activities include organization ofa CE user advisory panel to help determine CE strategicgoals and user requirements and to help with evaluation ofCE tools, documentation, and training materials. Core sup-port personnel are also involved in evaluation and beta-testing of new tools and new versions of existing tools. The

CE training strategy involves development of a compre-hensive training curriculum to give DoD users the knowl-edge and skills they need to effectively use HPC platformsat the SRCs. Training topics include specific programminglanguages, parallel programming models, performanceoptimization techniques, and debugging and performanceanalysis tools. Another CE activity is to anticipate and planfor CE needs in the context of emerging HPC trends andsystems, such as cluster computing, meta-computing, andinteractive and real-time requirements.

CE projects target specific areas where critical user needshave been identified. Areas that have been identified bythe CE user advisory panel include application portability,application performance evaluation and tuning, a consis-tent well-documented CE, data management, and paralleland asynchronous input/output. Current CE projects areaddressing a consistent CE and deployment of PAPI forthe purpose of application performance analysis. Projectproposals to address these and the other newly identifiedneeds are currently being formulated.

EQM STRATEGY

DR. MARY WHEELER, UNIVERSITY OF TEXAS, AUSTIN,EQM FA POC

The EQM FA under PET has defined three primary strate-gic efforts. The first effort involves the development andimplementation of new, accurate parallel discretizationswith adaptivity that will be based on a posteriori error esti-mators. In addition, this effort will include robust scalableparallel solvers.

A second effort involves code couplings of different physi-cal processes (e.g., flow, transport, reactions, and mechan-ics) which occur within the same physical domain, or whendifferent physical regimes (e.g., surface/subsurface, andfluid/structure) interact through interfaces. The issues to beconsidered include (1) appropriate physical quantities ortransmission conditions that are dependent on the applica-tion for coupling across physical domains, (2) developmentof algorithms for modeling transmission conditions, (3)time-stepping selection for treating couplings of differentphysical models, (4) grid projection strategies, and (5) com-puter science tools for implementing coupled processes.

The third effort involves the integration of emerging com-putational tools for handling large data sets, grid/metacom-puting, interactive steering, scientific visualization, meshpartitioning, and dynamic adaptive grids.

It is anticipated that the algorithms and tools developedunder these efforts will be applied to EQM/CWO softwaresuch as Adaptive Hydraulics (ADH), CE-QUAL-ICM,ADCIRC, and CH3D.

Left:NAVOCEANO

Change ofCommand

Ceremony 8March 2002.

Participants L-R,CAPT Philip G.

Renaud (incoming),RDML Thomas Q.

Donaldson, V,Commander, NMOC,

and CAPT TimothyMcGee

Left:RDML Thomas Q.

Donaldson, V, andmembers of

the U.S. Commissionon Ocean Policy.

Left:(L-R) CDR Augusto Mourao Ezequiel,Technical Director,Portuguese HydrographicInstitute, VADM JoseTorres Sobral, GeneralDirector, PortugueseHydrographic Institute andDave Cole, NAVO MSRC,Carol Nichols, CNMOC.

Left:(L-R) LCDR Erhan Gezgin,Project Officer, TurkishNavy Department ofNavigation, Hydrographyand Oceanography(TNDHNO), CPT AliKaplan, Head ofTechnology Group,TNDHNO, CDR AhmetTurker, Chief ofOceanography Division,Michael Jeffries,NAVOCEANOInternationalDivision.

Right:Steve Adamec,Director, NAVO

MSRC, briefs visit-ing members of

the U.S.Commission on

Ocean Policy.

Right:(L-R) CDR Raul Martinez Sanchez,

Director of HydrographicDepartment, Hydrographic and

Cartographic Office (DIGADHICAR),LCDR Rafael Ponce, Director of

Tides and Maritime Publications,DIGADHICAR, and RADM Anastacio F.

DeAbiega Gamez, Director General,DIGADHICAR, Eric Villalobos,

NAVOCEANO International Division.

Right:(L-R) Dave Cole, NAVO

MSRC, CPT Eric Fraser,British Naval Staff,

British Embassy, and LCDR Don Ventura.

Right:Members of the Basic

Officer AssessionTraining course at theNaval Meteorology and

Oceanography ProfessionalDevelopment Center.

Left:(L-R, facing) LTC Jeff Pritchard(USAF), LCDR Dave Manero(USN), MAJ Jim Hecker (USAF)of Senator Trent Lott’s staff.

Left:Visit of Steve Hall, National Imageryand Mapping Agency Gulf of Mexicodirector.

Left:Members of the editorial and

reporting staff of The SunHerald (Biloxi, MS) attending a

presentation by PeteGruzinskas of the NAVO

MSRC Visualization Center.

Right:(L-R) Frank

Lovato, NAVOMSRC Security

Officer, Doug Lee,Director, Louisiana

Research andTechnology

Park Baton Rouge,and companions.

Right:(L-R) Dave Cole, LCDR

Henry Howell, UnitedKingdom (UK) Personal

Exchange Program(PEP) Officer, FNMOC,and Tom Crew, Allied

Cooperation/Technology Transfer

Division, NAVOCEANO.

Right:(L-R) ADM V.E. Clark,

Chief of NavalOperations and Steve

Adamec, DirectorNAVO MSRC.

Left:Members of the

PET MOS Team arefamiliarized with the

NAVO MSRC by DaveCole.


Navigator Tools and Tips

This article describes a method for transferring filesbetween HABU and other systems via the transfer queueusing a stream of job steps in a single job command file.The transfer queue was created as a shared usage queuefor users to transfer input and/or output files before/afterthe executable program as serial jobs.

This means the transfer queue shares the interactive nodeswith other users. In order to submit a job to this queue,you must specify the following in the job step:

To specify a stream of job steps, you need to list each jobstep in the job command file. Each job step must be fol-lowed by the "queue" statement. The "dependency" key-word is used to let LoadLeveler know that the job steps arenot independent and that the next job will run based onthe exit status of the previously run job. In the examplebelow, there are three job steps:

1. A serial job step to transfer the input file via the transferqueue.

2. A parallel job step to run a sample POE script via thebatch queue.

3. A serial job step to transfer the output file via the transferqueue.

Note: Each job step must have a unique name for all out-put and error files.

Example job command file:

JOB STEP 1 EXECUTABLE

#@ class = transfer#@ node_usage = shared

f18n13e% more get_file.ksh#!/bin/kshlet RC=0# *** IDENTIFY NODE RUNNING SERIAL JOB STEP ***test_script.ksh# *** REMOVE FILE FROM PREVIOUS RUN ***rm test_filelet RC=$?if (($RC != 0))then /bin/echo "$0-ERROR: Removing test_file; RC=$RC" exit

f18n13e% more multistepjobscript#!/bin/ksh# ***** STEP 1 - SERIAL *****#@ account_no=NA0101#@ class = transfer#@ step_name = step1#@ executable = get_file.ksh#@ input = /dev/null#@ output = $(jobid).step$(stepid).log#@ error = $(jobid).step$(stepid).log#@ environment = COPY_ALL#@ initialdir = /scr/shecar/multisteps#@ job_name = multistep#@ job_type = serial#@ node_usage = shared#@ notification = never#@ notify_user = shecar#@ shell = /bin/ksh#@ wall_clock_limit = 300#@ queue# ***** STEP 2 - PARALLEL *****#@ account_no=NA0101#@ class = batch

#@ dependency = (step1 == 0)#@ step_name = step2#@ executable = poe_script.ksh#@ input = /dev/null#@ output = $(jobid).step$(stepid).log#@ error = $(jobid).step$(stepid).log#@ environment = COPY_ALL; MP_LABELIO=yes#@ initialdir = /scr/shecar/multisteps#@ job_name = multistep#@ job_type = parallel#@ node = 4#@ tasks_per_node = 4#@ node_usage = not_shared#@ notification = never#@ notify_user = shecar#@ shell = /bin/ksh#@ wall_clock_limit = 600#@ queue# ***** JOB STEP 3 - SERIAL *****#@ account_no=NA0101#@ class = transfer#@ node_usage = shared#@ dependency = (step2 == 0)#@ step_name = step3#@ executable = put_file.ksh#@ input = /dev/null#@ output = $(jobid).step$(stepid).log#@ error = $(jobid).step$(stepid).log#@ environment = COPY_ALL#@ initialdir = /scr/shecar/multisteps#@ job_name = multistep#@ error = $(jobid).step$(stepid).log#@ environment = COPY_ALL#@ initialdir = /scr/shecar/multisteps#@ job_name = multistep#@ job_type = serial#@ notification = always#@ notify_user = shecar#@ shell = /bin/ksh#@ wall_clock_limit = 300#@ queue# ***** END OF multistepjobscript SCRIPT *****

File Transfer Techniques: TransferringBetween HABU and Other SystemsSheila Carbonette, NAVO MSRC User Support



Which runs the following:


Use the "llsubmit" command to submit the multistepjob:

When this job command file is submitted to LoadLeveler,three jobs will be placed in the queue. This can be verifiedby using the "llq" command.

Initially the first job is queued, and the second two willwait until the previous one finishes.JOB STEP 1 OUTPUT

JOB STEP 2 OUTPUT

JOB STEP 3 OUTPUT

$RCfi# *** GET FILE FROM JULES ***/usr/bin/rcp jules-hip0:/u/a/shecar/test/test_file test_filelet RC=$?if (($RC != 0))then

/bin/echo "$0-ERROR: RCPing test_file FROM Jules; RC=$RC"else

/bin/echo "File Staging FROM Jules Completed SUCCESSFUL-LY!"fiexit $RC# *** END of get_file.ksh script ***

f18n13e% more 41673.step0.logHostname = f18n13eDate = Thu Mar 21 09:37:16 CST 2002File Staging FROM Jules Completed SUCCESSFULLY!

f18n13e% more poe_script.ksh#!/bin/ksh# *** Run POE job ***poe test_script.ksh# *** END of poe_script.ksh script ***

f18n13e% llsubmit multistepjobscriptllsubmit: Processed command file through Submit Filter:"/loadldir/local/bin/ll_submit_filter".llsubmit: The job "f18n13s.navo.hpc.mil.41673" with 3 job steps hasbeen submitted.

f18n13e% more put_file.kshf18n13e% more test_script.ksh#!/bin/kshlet RC=0/bin/echo "Hostname = `hostname -s`"let RC=$?if (($RC != 0))then

exit $RCfi/bin/echo "Date = `date`"let RC=$?exit $RC# *** END of test_script.ksh ***

f18n13e% more put_file.ksh#!/bin/kshlet RC=0## *** IDENTIFY NODE RUNNING SERIAL JOB STEP ***#test_script.ksh## *** COPY FILE TO JULES ***#/usr/bin/rcp test_file jules-hip0:/u/a/shecar/test/test_filelet RC=$?if (($RC != 0))then\

/bin/echo "$0-ERROR: RCPing test_file TO Jules; RC=$RC"else

/bin/echo "File Staging TO Jules Completed SUCCESSFULLY!"fiexit $RC# *** END of put_file.ksh script ***

f18n13e% llq | egrep "^I|shecar"

f18n13e% more 41673.step2.logHostname = f15n13eDate = Thu Mar 21 09:53:22 CST 2002File Staging TO Jules Completed SUCCESSFULLY!

IDf18n13s.41673.0f18n13s.41673.2f18n13s.41673.1

Ownershecar

shecar

shecar

STI

NQ

NQ

PR50

50

50

Classtransfer

transfer

batch

Submitted3/21 09:11

3/21 09:11

3/21 09:11

f18n13e% more 41673.step1.logATTENTION: 0031-408 16 tasks allocated by LoadLeveler, continuing

1:Hostname = f21n03e7:Hostname = f21n04e

11:Hostname = f21n05e12:Hostname = f21n06e0:Hostname = f21n03e5:Hostname = f21n04e8:Hostname = f21n05e

14:Hostname = f21n06e2:Hostname = f21n03e6:Hostname = f21n04e9:Hostname = f21n05e

13:Hostname = f21n06e3:Hostname = f21n03e4:Hostname = f21n04e

10:Hostname = f21n05e15:Hostname = f21n06e2:Date = Thu Mar 21 09:51:24 CST 20027:Date = Thu Mar 21 09:51:24 CST 20028:Date = Thu Mar 21 09:51:24 CST 2002

14:Date = Thu Mar 21 09:51:24 CST 200211:Date = Thu Mar 21 09:51:24 CST 200215:Date = Thu Mar 21 09:51:24 CST 200210:Date = Thu Mar 21 09:51:24 CST 20023:Date = Thu Mar 21 09:51:24 CST 20025:Date = Thu Mar 21 09:51:24 CST 2002

12:Date = Thu Mar 21 09:51:24 CST 20020:Date = Thu Mar 21 09:51:24 CST 20026:Date = Thu Mar 21 09:51:24 CST 20021:Date = Thu Mar 21 09:51:24 CST 2002

13:Date = Thu Mar 21 09:51:24 CST 20024:Date = Thu Mar 21 09:51:24 CST 20029:Date = Thu Mar 21 09:51:24 CST 2002


VSIPL...continued

performance computers. Efficientmachine-specific implementations ofBLAS are available for several high-performance computers.

APPROACH AND RESULT

The VSIPL TASP library was modi-fied to call all three libraries (i.e.,FFTW, LAPACK, and BLAS).Another feature is a header file thatallows the user to switch betweenbuilding the ERI library and the stan-dard TASP library. The final deliv-ered library will include a perl scriptso that the end-user can downloadeach of the reference libraries andrun the script to add the necessarymodifications. The user will thenbuild each library, and MSTI will pro-vide some example codes andbenchmarking routines.

The initial results for VSIPL/ERI areencouraging: Figure 1 shows somecomparative benchmarks for fourFFT functions using float and doubledata types. The top table is theVSIPL/ERI numbers, and the lowertable is the TASP numbers. The fourfunctions are real-to-complex FFT

out-of-place (rcfftop), complex-to-realFFT out-of-place (crfftop), complex-to-complex FFT out-of-place(ccfftop), and complex-to-complexFFT in-place (ccfftip). The numberswere generated using the LinuxVSIPL/ERI implementation on an800-MHz PIII with 256M RAM (men-tioned since processor speed andRAM size affect the performance).Figure 2 shows some comparativeLinear Algebra benchmarks. The sixfunctions are as follows: Choleskydecomposition (symmetric(Hermitian) positive definite linearsystem) for real (Cholesky_f) andcomplex (Ccholesky_f); LU decom-position (Gaussian decomposition)for real (LUD_f) and complex(CLUD_f); general matrix product(compute the general product of twomatrices and accumulate) Gemp_f;and matrix product (product of twomatrices) Mprod_f. The upper tablerepresents the VSIPL/ERI numbers,and the lower table shows the TASPVSIPL numbers. Most of the func-tions require two input values, andfor these cases, the same number(column heading number) was used

for both inputs. These benchmarkswere generated on the same 800-MHz PIII with 256M RAM.

SUMMARY AND CONCLUSION

It is important to note that theLAPACK routines are called onlywhen the matrix majors are columnand the stride is equal to one, other-wise the TASP VSIPL code is execut-ed. It was found that the MPRODand GEMP functions yielded no per-formance improvement over theTASP VSIPL implementation. Theother Linear Algebra functionsshowed increased performance,especially for the larger matrix sizes.Each of the VSIPL/ERI FFT functionsexecuted in half the time of the TASPVSIPL FFTs for the sizes shown. Inall, 56 linear algebra functions inVSIPL and LAPACK/BLAS werewrapped in VSIPL/ERI. VSIPL/ERIwill be free to all U.S. sites(Government-purposes rights) andwill also be open source for the U.S.Government. MSTI will offer a freelicense for academic institutions. Acommercial implementation will alsobe available.

TTTThhhheeee UUUUppppddddaaaa tttteeeedddd NNNNAAAAVVVVOOOO MMMMSSSSRRRRCCCC UUUUsssseeee rrrr SSSSuuuuppppppppoooo rrrr tttt TTTTeeeeaaaammmm

FRANK GREEN

USER SUPPORT ANALYST

JARED BAROUSSE


SHEILA CARBONETTE

LEADUSER SUPPORT ANALYST

SHAWN RICHARDSON


Global Grid Forum 6October 15 l 17, 2002

Chicago, ILwww.gridforum.org

IBM SP SCICOMP 6August 19 l 23, 2002

San Francisco, CAwww.spcicomp.org

PACT 2002September 22 l 25, 2002

Charlottesville, VAwww.pactconf.org

Oceans 2002October 29 l 31, 2002

Biloxi, MSwww.oceans2002.com

UPCOMING EVENTSJUN

JUL

AUG

SEP

OCT

NOV

DEC

JAN

FEB

MAR

APR

MAY

SIGGRAPH 2002July 21 l 26, 2002

San Antonio, TXwww.siggraph.org/s2002

2002 USENIX AnnualTechnical Conference

June 9 l 14, 2002Monterey, CA

www.usenix.org

USENIX Securi ty

SymposiumAugust 5 l 9, 2002San Francisco, CAwww.usenix.org

APS DCIOMP CCP 2002

August 25 l 28, 2002San Diego, CA

www.aps.org/DCOMP/Meetings/meeting/meet

02/CPCP2002.htm

2002Western Paci f ic

Geophysics Meet ingJuly 9 l 12, 2002

Wellington, New Zealandwww.agu.org

GlobalGrid Forum 5

July 21 l 25, 2002Edinburgh, Scotlandwww.gridforum.org

the director’s corner · the director’s corner the next few months will be notable ones here at...

Documents