a debugging standard for high-performance...

95

A debugging standard for high-performancecomputing

Joan M. FrancioniComputer Science Department, Winona StateUniversity, Winona, MN 55987, USAE-mail: [email protected]

Cherri M. PancakeDepartment of Computer Science, Oregon StateUniversity, Corvalis, OR 97331, USAE-mail: [email protected]

Throughout 1998, the High Performance Debugging Forumworked on defining a base level standard for high performancedebuggers. The standard had to meet the sometimes con-flicting constraints of being useful to users, realistically im-plementable by developers, and architecturally independentacross multiple platforms. To meet criteria for timeliness, thestandard had to be defined in one year and in such a way that itcould be implemented within an additional year. The Forumwas successful, and in November 1998 released Version 1of the HPD Standard. Implementations of the standard arecurrently underway. This paper presents an overview of Ver-sion 1 of the standard and an analysis of the process by whichthe standard was developed. The status of implementationefforts and plans for follow-on efforts are discussed as well.

Keywords: Debuggers, debugging, high performance com-puting, parallel programming, parallel debuggers, standards

1. Introduction and background

The High Performance Debugging Forum (HPDF)was established in March of 1997, with the goal ofdefining standards relevant to debugging tools for high-performance computing (HPC) systems. By the fol-lowing January, the first draft of the standard was madeavailable to the HPC community for comment and eval-uation. This paper describes the process and the resultsof that standards effort.

Prior to HPDF, there were no published standardsor rigorous definitions of debugger interfaces or func-tionality. As might be expected, the lack of standards

meant that debugger implementations varied widely. Inthe serial world, this situation has not been entirely bad;variation has contributed to the overall development ofappropriate and effective debugger syntax/semantics.Since serial programmers are likely to continue work-ing on a system for extended periods of time, they canget used to a “favorite debugger” and not have to worryabout changing tools frequently. Some serial debug-gers are even supported on multiple platforms (e.g.,gdb [20] and dbx [21]), although their implementationsmay vary in subtle ways on each platform.

Unfortunately, the scenario is quite different for theparallel programming community. The nature of HPCis one of rapidly changing hardware and software en-vironments. At any one time, there are fewer than adozen companies producing HPC machines and theycompete within a relatively small market [16]. Fromthe computer manufacturer’s viewpoint, there has beenno real economic advantage to expending effort so thatdebuggers will be consistent with those on competitors’machines – or even those on future versions of theirown machines. Consequently, each new machine maypresent a totally new debugger, or one that has featuresincompatible with earlier or future version.

From the user’s perspective, it is not cost effective tohave to learn a new debugger for every new machine,especially when the lifespan of the tool or machine maybe only a couple of years. Moreover, most HPC userswork on different systems from one project to another,and sometimes for a single project. Ease-of-use andcross-platform compatibility have become key consid-erations in user decisions about whether to adopt newparallel tools [17]. To date, however, there is no par-allel debugger that behaves consistently across differ-ent architectures and operating systems, nor one thatis considered easy to learn and use. Even debuggersoriginally developed by independent software develop-ers may vary from one implementation to another, suchas the Totalview developed by Etnus (formerly Dol-phin Interconnect Solutions) [5], versus the Totalviewported by Cray to the T3E series).

Scientific Programming 8 (2000) 95–108ISSN 1058-9244 / $8.00 2000, IOS Press. All rights reserved

96 J.M. Francioni and C.M. Pancake / A debugging standard for high-performance computing

The HPC community has discussed the need for par-allel debugger standards at national and internationalworkshops over the years. In October of 1996, usersand debugger developers at the ARPA/NSF Workshopon Parallel Tools concluded that a community-widestandards effort should be started without delay. Anorganizational meeting was held at the SC96 confer-ence the following month, and the High PerformanceDebugging Forum was officially established in Marchof the following year. HPDF is a collaborative effortinvolving researchers in the area of parallel debugging,commercial parallel debugger developers, and repre-sentatives of HPC user organizations. It is sponsoredby the Parallel Tools Consortium (Ptools) [15]. Similarefforts to standardize parallel performance tools are un-derway by Ptools and the APART (Automatic Perfor-mance Analysis: Resources and Tools) Esprit WorkingGroup.

“HPDF” refers to the Forum that defined the stan-dard. For purposes of identification, the name ofthe standard itself is High Performance Debugger(HPD) Standard. In this paper, we explain the goalsand overall structure of Version 1 of the HPD Stan-dard. (Complete information, including definitionsof syntax and semantics, can be found on-line athttp://www.ptools.org/hpdf.) The limitations of thestandard, the processes employed in developing it, andthe current status of the standard are also discussed.

1.1. Goals of the HPD standard

The Forum established three general goals concern-ing parallel debuggers. First, parallel debuggers shouldsatisfy the basic requirements of persons who developapplications for HPC machines. That is, the target usershould be application developers, as opposed to com-piler writers, library developers, etc. Second, paralleldebuggers should be usable by those application devel-opers, in the sense of easy to learn and easy to use.Third, parallel debuggers should be consistent acrossplatforms, so that users of one standard-conformingdebugger can switch to another with little or no effort.

One important lesson learned from previous stan-dards efforts is that the window of opportunity forHPC standards is quite small [15]. The rate of changein HPC hardware and operating systems is extremelyrapid. Experience has shown that any standards effortwhich takes more than a year or two from start to finishwill be outdated before it can be implemented. Tak-ing this into account, the Forum decided that its initialstandard should be defined within a year and imple-mentable within another year. Specifically, the HPDStandard, Version 1 should:

– Capture the best-practice knowledge and experi-ence of parallel debugger implementors across theindustry.

– Establish a well-defined, testable, and minimalcore set of features that can be implemented on allHPC systems.

– Ensure that parallel debugger implementors pro-vide this set of features in a consistent way.

– Limit the core set in size so that initial commercialimplementations can be available within a year ofthe standard’s release.

To meet these goals, the Forum decided to subdi-vide the relevant issues into different versions, or lay-ers, such that each successive version would build uponprevious ones. Also, it was decided that the first versionshould define a standard command-based (i.e., non-graphical) interface for parallel debuggers. Issues suchas graphical interfaces and support for debugging opti-mized code would be deferred to future versions. Whilemuch of the standard was expected to be equally appli-cable to serial debuggers, attention would be focusedon those issues that arise when the program being de-bugged includes multiple threads, multiple processes,or collections of multi-threaded processes.

The resulting standard, HPD Version 1, defines thesyntax and semantics of commands supporting the mostneeded debugging functions. Individual implementa-tions are free to add other commands – indeed, this maybe necessary in order to support debugging of machine-specific features – but only the commands specifiedin the standard are required. A debugger that meetsall the requirements specified in the formal HPD stan-dards document [8], is said to be an HPD-conformingdebugger.

1.2. Architecture-independence of the standard

The HPD Standard attempts to be both hardware- andoperating-system-neutral, in the sense that it should bepossible to build a standard-conforming debugger on awide variety of different computing systems. This wasa realistic objective because a broad group of computerand software manufacturers participated in the Forum.In most cases, the group was able to reach consensus sothat required features of the standard could be system-independent. Some features were defined as extensionsto the standard, in recognition of the fact that not allsystems can support them at the present time. (A hand-ful of special features, known to be system-dependent,were explicitly called out with suggestions of how thevariation should be accommodated.)

J.M. Francioni and C.M. Pancake / A debugging standard for high-performance computing 97

Since the HPD Standard addresses the needs of HPCapplication developers, it assumes that programs needto be system-independent. In particular, target pro-grams are assumed to be written in one or more high-level languages, for execution on possibly many differ-ent computer systems, and execution performance ofthe program is an important consideration.

Explicit parallelism is assumed as the basic pro-gramming model, so the standard applies to bothshared-memory programming (multiple threads of ex-ecution in a single address space, such as those spec-ified by OpenMP and distributed-memory program-ming (multiple processes co-operating via message-passing libraries, such as PVM [6] or MPI [13]). HPD-conforming debuggers may also be useful for implic-itly parallel programs (e.g., auto-parallelized DO loopsor array operations), but the issues of how to map fromruntime or intermediate-level information to originaluser source code are not addressed in the initial versionof the standard.

The HPD Standard recognizes the functionalityneeded for three models of parallelism: processes-only,threads-only, and multilevel (both multi-process andmulti-thread). Where the constraints of these mod-els differ, the standard specifies how a debugger sup-porting each model will behave. The objective is tomake it possible for debuggers of all three types to pro-vide support that is as consistent as possible, given theconstraints imposed by the underlying models. Thus,if a user’s application is processes-only, the behaviorshould be consistent regardless of whether it is execut-ing under the control of a processes-only or a multileveldebugger.

The languages considered during development of thestandard were Fortran (F77 and F90), C, and to a lesserextent, C++. In principle, the standard could apply toother languages, but debugger syntax was not explicitlychecked for conflicts with expression syntax in otherlanguages. Moreover, the standard does not addressinterpreted languages, which typically are packagedwith a built-in debugger. A debugger is said to “supporta language” if its commands apply to programs in thatlanguage (e.g., gdb supports C). An HPD-conformingdebugger must provide the following functionality insupport of each target language:

– Accept source files written in that language, andexecutables generated from them.

– Navigate to source locations in procedures writtenin that language.

– Accept right-hand-side expressions in the syntaxof that language, evaluate them to object valuesusing the semantics of the language, and displaythose values.

– Accept left-hand-side expressions in the syntax ofthat language, evaluate them to object referencesusing the semantics of the language, and assignvalues to those objects.

The HPD Standard gives specific directions for ap-plying the above functionality in C and Fortran becausethey are the most common HPC languages.

1.3. Outline of this paper

An overview of the HPD Version 1 standard is pre-sented in Section 2. The general conceptual modelis described along with definitions for the general ter-minology used throughout. In Section 3, the processof developing the standard is presented and analyzed.Section 4 includes a discussion of the next phase ofdevelopment for the HPD Standard. An Appendix liststhe command set for Version 1 of the standard.

2. Conceptual model of parallel debugging

A debugger is a tool that gives a user visibility into,and control over, an executing program – the target pro-gram. A parallel debugger performs that same func-tion for a parallel program. In the world of parallelcomputing, an executing program consists of one ormore processes, each associated with a particular exe-cutable (and perhaps one or more shared libraries) andeach occupying a memory address space. Every pro-cess, in turn, has one or more threads, each with its ownregister set and its own stack. The target program isthus the complete set of threads and/or communicatingprocesses that make up a given execution of the user’sapplication, over the full course of program execution.

2.1. Interacting with the debugger

Although the debugger affects a target program, it isnot part of the target program’s execution. As shownin Fig. 1, the debugger can be thought of as runningin separate process(es). Thus, debugger semantics aredefined separately from the semantics of the target pro-gram language. (For an alternative way of defining de-bugger semantics, see [2].) The debugger communi-cates with the target program according to an execution-


consoleinput

consoleoutput

debuggerinput

debuggeroutputprogram

interface

execution-timeinterface

user interface

Debugger Program

Target Program

User

Fig. 1. Relationship between debugger and target program.

time interface that is usually operating system-specific(e.g., ptrace or /proc).

When the target program executes, it goes through aseries of program states. If we assume a deterministictarget program – as is the case with any serial program– it is possible to require that, as the debugger startsand stops program execution, it does not change thesequence of program states that would have occurredif the program were executing on its own. In a paral-lel program, however, the debugger’s intervention maychange the timing interrelationships among threads orprocesses; this in turn may change the overall behaviorof the target program. Although this is not the intent ofthe debugger, it can not be guaranteed not to occur.

Initiating Debugging. The standard requires that itbe possible to initiate debugging sessions in three ways.The debugger can be invoked from the command lineand the target program executed within the the debug-ger environment. Alternatively, the debugger may beattached to a program that is already executing in thenormal run-time environment. In either case, if the tar-get program consists of many processes, the debuggermay need to interact with the run-time system or thelibrary that is responsible for managing those processes(e.g., PVM [6] or MPI [13]). Third, the debugger maybe associated with a parallel program that terminatedabnormally, as long as an image of the running programwas captured and stored as a core file. Since there areno running process(es) associated with the target pro-gram in this case, only certain debugging operations areavailable for core-file debugging. In particular, while itis possible to examine the location of the program andthe values of its variables, it is not possible to modifyanything about the program image preserved in the corefile.

In all three cases, the debugger not only controlsthe executable(s) that constitute the target program, re-flected in the memory and register values of the exe-cuting program, but also utilizes debugging informa-tion associated with the source files of the executables.Such information provides a means for the debugger togive high-level output to the user, expressed in terms ofthe variables and procedures used in the source code. Italso enables the debugger to access smaller componentsof the program (e.g., source files), thereby eliminatingthe need for some assistance from the user. On most tar-get systems, debugging information is only generatedwhen the executable is compiled with special options(such as “-g”) in effect.

User Command Model. Debuggers are designed toreceive user input on what actions should be taken tocontrol the target program’s execution or to reveal infor-mation about it. For the purposes of the HPD Standard,this input is via a command-language (non-graphical)interface. The specifics of the command language aregiven in the formal standards document; here we dis-cuss the general command model.

When a debugger command is executed, one or moreof the following occurs:

– A change takes place in the current state of thetarget program.

– A change takes place in the information that thedebugger maintains about the target program.

– Information about the target program or the de-bugging session is displayed to the user.

A prompt is issued when the command has com-pleted and the debugger is ready to accept another one.

The command model is sequential in the sense thatonly one command is being processed at any giventime. For instance, if the user issues a command to


print out an array, the debugger does not prompt againuntil all elements of the array have been displayed.The standard does not define any kind of “backgroundcommand execution” capability analogous to the useof an ampersand within a UNIX shell. (It is, of course,fine if a particular debugger implementation adds thisfeature.)

Because the command model is sequential, it is im-portant that the description of each command in thestandard defines explicitly when that command is con-sidered “complete”. For some commands, this is ob-vious; e.g., the command to print an array is completewhen all the values have been displayed. For others,such as the command to step the execution of eachprocess through its “next” statement, the moment ofcompletion is not as intuitively obvious. One processmight execute a receive operation which forces it topause until the message arrives from some other pro-cess; does this mean that the step can’t complete? Stillother commands, like the setting of a breakpoint, maycause deferred actions (stopping at the breakpoint) tooccur later in execution, but the command “completes”much earlier (once the breakpoint has been registeredby the debugger).

The sequential nature of commands applies to de-bugger processing of the commands, not to the rela-tionship of debugger execution with program execu-tion. The standard’s command model does not requirethat the target program be stopped when the debuggerprompts for and performs commands. It dictates onlythat one command be complete before another can beissued. Some processes may continue to execute whilethe debugger is performing commands. On the otherhand, some commands cannot be performed by the de-bugger if the target program is still active. For example,it may not be possible to print the values of an arrayif one or more processes is still executing (and hence,might change those values). Therefore, each commanddescription explicitly states when the command can beissued and what type of error ensues if the command isattempted at other times.

Debugger Output. A debugger typically issues avariety of messages in response to user input. Some ofthese confirm that an operation completed successfullyor indicate that a problem occurred. Others providein-depth detail about what the debugger or the targetprogram is doing. The HPD Standard specifies whatoutput is required for each particular command and alsodefines a general mechanism for controlling the amountof warning and diagnostic messages.

Debuggers differ from other interactive tools in thatsome output may appear on the screen, not at the time

a command is issued but at some later time. This is thecase for commands related to actionpoints, i.e., thosepoints where the flow of program execution suspendsunder user request. For example, a debugger typicallygenerates no immediate output when a breakpoint isset, but instead prints an informational message eachtime a process or thread arrives at the breakpoint.

2.2. Effects of parallelism on debugger behavior

Execution control of a target program is relativelysimple in a serial debugging environment, since theprogram is always either stopped or running. When itis running and its operations cause some event to occur(e.g., arriving at a breakpoint), this is said to triggerthe event and the debugger stops the program. Theuser can later continue program execution, effectivelyundoing the effect of triggering the breakpoint. Parallelprogram execution is more complex, however, as eachthread has an individual execution state. When a threadtriggers a breakpoint, the question arises as to what, ifanything, should be done about the other threads andprocesses.

In this section we discuss how thread sets make itpossible for the user to determine which threads areaffected by debugger comments. This provides a basisfor describing how parallel execution can be managed.We define actionpoints as a means for causing programexecution to stop, then present the stopping and startingmechanisms adopted for the HPD Standard and discusshow these are used to control program execution.

Thread sets. The concept of thread sets providesthe foundation for extending the semantics of serial de-bugger operations to a form suitable for parallel pro-grams. This concept allows a debugger command to beapplied to a whole collection of processes or threads,rather than to just one at a time. Process sets are thestandard practice of existing parallel debuggers that ac-commodate multiple processes (cf. Cdbx [3], code-view [19], CXdb [1], HP/DDE [7], ipd [11], Lade-bug [4], MPPE [12], ndb [18], P2D2 [9], pdbx [10],Prism [22], TotalView [5]).

The HPD Standard extends this notion to multi-threaded programs as well. It defines a thread set to bea set of threads drawn from all threads in all processesof the target program. Unlike a serial debugger, whereeach command clearly applies to the single thread ofexecution control, parallel debuggers can have poten-tially many distinct threads of control and potentiallymany different locations corresponding to a programsymbol (e.g., a variable). The concept of a target thread


set is thus used to restrict a debugger command so thatit applies to one, many, or all threads of control.

Commands must always apply to some thread(s),referred to as the target set for the command. Whileit is possible for the user to explicitly type the targetset as part of each command, the debugger maintainsan implicit target set, referred to as the current set.When the debugger initiates, the current set includesall threads in the target program, but the user is freeto change this at any time. (Additional information onthread sets is provided in Section 2.3.)

Actionpoints. By setting up actionpoints, the userrequests in advance that target program execution stopunder certain conditions. Three types of actionpointsare supported in the standard. Each allows the user toindicate that execution should continue until some spe-cific type of program event occurs. A breakpoint spec-ifies that the execution of a process should stop when-ever it reaches a given location relative to the sourcecode. A watchpoint provides analogous control on thebasis of data storage, stopping whenever the value of avariable is updated. A barrier, as its name suggests, ef-fectively prevents processes from proceeding forwardbeyond a certain point in the source code until otherprocesses have also arrived, providing a mechanism forsynchronizing the activities of processes. (Barriers canonly be applied to entire processes, not to individualthreads.)

Each actionpoint is associated with a trigger set,or set of threads for which the actionpoint has beendefined. There is also a stop set that establishes whichthreads should be halted as a result of the actionpoint.When any member of the trigger set causes the programevent to trigger, the debugger intervenes, issues sometype of message to indicate that the event has occurred,and stops all members of the stop set as well as thetriggering thread. The user can then examine the valueof program and debugging variables, and make changesaccordingly.

Thread State. As it executes, the target program goesthrough a series of states. We can think of executioncontrol as a way for the user to request that the debug-ger allow the target program to advance, stopping atsome future program state. Although the term “stop”is intuitive, what we really mean is “pause executionand allow the user to examine the state of the programbefore continuing”.

In fact, at any point the target program can be saidto have a composite state, reflecting the states of allthe individual threads involved in its execution. Eachthread that is actively executing the target program

is, at any given time, in one of three possible exe-cution states: running, stopped/runnable, orstopped/held.

The running state is defined from the perspectiveof the debugger. That is, an execution command suchas go has been issued, the debugger has passed theappropriate request to the underlying run-time system,and no subsequent program event has triggered. Fromthe debugger’s point of view, such a thread is “run-ning”, and debugger commands that attempt to exam-ine or change that thread’s program information, willnot be possible. (From the perspective of the under-lying run-time environment, such a thread may makemany transitions between being ready to run and actu-ally running, but typically these lower-level transitionsare invisible to the debugger and the debugger user.)

A thread enters the stopped/runnable state un-der several circumstances:

– when the executable is first loaded or the debuggerfirst attaches to an existing process

– when the user explicitly asks the debugger to stopthe thread

– when the thread’s execution triggers a programevent

– when some other thread’s execution triggers a pro-gram event that affects this thread

Once the thread has stopped, debugger commandsto examine or change the state of the thread becomeavailable. In addition, the thread becomes eligible asa target for any command which causes it to resumeexecution. (With debuggers that do not choose to im-plement the extensions for controlling threads individ-ually, all threads within a given process will be stoppedwhen any one of them triggers an event, and all willbe returned to the running state when execution isresumed.)

The stopped/held state is similar to the stop-ped/runnable state, except that a thread in this statewill not respond to resume commands. A thread entersthis state as a result of triggering a barrier. The thread’sstate must first be changed to stopped/runnable– which happens when the barrier has been satisfiedor by explicit user command – before it is eligible forresuming. Only a few debugger commands for exam-ining and changing state are available when a thread isin the stopped/held state.

Advancing Program Execution. The advantage ofinteractive debuggers is that they allow the programmerto exert control over the execution of the target program.That is, rather than simply launching the program to


run until it terminates (or pauses for input), the usercan choose to step execution through one statement at atime, run the program until he/she decides to stop it, orallow it to run freely until some actionpoint is reached.

For most serial debuggers, the program can bestopped between the execution of any two consecutivestatements. The situation is not so simple for paralleldebuggers, which have to manage the execution of mul-tiple threads. The stopping model employed by paralleldebuggers that support multithreaded target programs(e.g. [20,21]) is referred to as the “stop-the-world”model. In this model, the debugger automatically stopsall threads of execution whenever any thread triggersan actionpoint. This has the advantage of providing atarget program that is quiescent while the user exam-ines program state. However, it has the disadvantageof seriously disrupting program execution.

An alternative stopping model is to simply leaveother threads of execution unaffected when one (ormore) triggers an actionpoint. This is the modeladopted by most multiprocess parallel debuggers(e.g. [1,10,11,18]). For example, in a client-server ap-plication, it is more natural when a client triggers abreakpoint that the server not be stopped as well.

The HPD Standard adopts a dual stopping model:stop-the-world for threads and leave-others-alone forprocesses. That is, any processes that have not trig-gered an actionpoint will be unaffected, but all threadsin each process that have done so will be stopped to-gether. This approach provides a common denomina-tor that can be implemented on all HPC systems. Thealternative – to stop only the individual threads thattriggered an actionpoint – is specified as an extension tothe standard; syntax and semantics are defined for anyimplementations that wish to implement the extension.Figure 2 summarizes this concept.

Parallel debuggers must also define a starting model.This applies to the so-called resume commands, thosewhich re-start execution after one or more threads havestopped. The issue here is how many threads to re-start– all of them which were stopped, or just those selectedby the user. In HPD Version 1, the default startingmodel is the mirror image of the stopping model, so thata resume command un-does the effect of triggering anactionpoint. The ability to start just individual threadswithin a process is defined as an extension. This startingmodel is depicted in Fig. 3.

2.3. Other features of the HPD standard

The notion of symbols from traditional serial debug-ging has also been extended to accommodate features

unique to parallel programs. This section describesthe special features that an HPD-conformant debuggeruses to manage the complex namespace associated withparallel execution.

Named Thread Sets. As discussed in the last section,thread sets constitute the primary mechanism for con-trolling which elements of the executing program areaffected by each debugger command. Because of theirimportance, the HPD Standard includes mechanismsfor associating logical names with these sets. That is,in addition to relying on the default set or overridingit with an explicit specification on the command line,the user can refer to symbolic names associated withparticular sets of threads. The user can use one of thesenames to refer to the group, rather than having to typea list of the processes/threads belonging to it.

Six debugger-defined sets are created and maintainedautomatically by the debugger:

– all: the set of all threads associated with thetarget program

– running: all currently executing threads (i.e., inthe running state)

– stopped: all threads that are not currentlyrunning (i.e., in the stopped/runnable orstopped/held states)

– runnable: all threads capable of running (i.e.,in the stopped/runnable state)

– held: all threads that cannot run until some se-ries of events occurs (i.e., in the stopped/heldstate)

– exec(executable): all threads associatedwith a particular executable

Membership in a debugger-defined set changes overthe course of the program. That is, the members towhich one of these set names refers will depend onwhen the command is issued.

In addition, the user is free to create any number ofuser-defined sets, reflecting the logical organization ofthe program. For example, sets might be establishedfor client versus server processes, a master thread anda collection of worker threads, or just those processescomputing the values of boundary elements in an ar-ray. The set names can then be used in any commandsand can be used in combination with debugger-definedsets. For example, a command could refer to just thoseprocesses in the set named FirstTen that are currentlystopped.

Consider the situation where a user defines the setBoundaries to include the processes responsible forcalculations of ghost array elements, corresponding to


Stop all threads ofprocesses that trigger

Stop only the threadsthat trigger

Stop all threadsin all processes

HPD standard Not included Optional extension

Stopping Model

Fig. 2. Execution control stopping model.

Start all threads ofprocesses in current set

Start only the threadsin current set

Start all threadsin all processes

Not included HPD standard Optional extension

Starting Model

Fig. 3. Execution control starting model.

processes [8–10,19,21,22]. (Note that these are logicalnumbers assigned by the debugger; the operating sys-tem and the parallel runtime system may assign othernumbers to the process/thread. The appropriate map-pings may be retrieved via other debugger commands.)If Boundaries is then assigned as the current set,subsequent commands would apply to just those pro-cesses. This mechanism can also be used in combina-tion with debugger-defined sets. To continue the ex-ample, if the user then asked to see a list of stoppedprocesses, the list would show just those processes inBoundaries that are currently stopped.

The membership of any named set (debugger- oruser-defined) is determined at the time it is evaluatedfor use in a particular operation. There are two distincttimes when evaluation can take place: the point ofdefinition and the point of use. The first occurs whenthe name of the set is first defined, while the pointof use occurs whenever the set name is referenced orwhen the debugger applies some membership test (e.g.,to determine which thread modified the value beingmonitored by a watchpoint).

Consider the example of Boundaries. At thepoint of definition, the set would include any of theboundary processes that happen to exist at that time,say [8–10,19]. By the time Boundaries is actuallyused on a debugger command, however, some of thoseprocesses may no longer exist and new ones may havecome into existence. Suppose [8,19] have terminatedwhile processes [20,21] have been created by the timethe point of use occurs. In that case, Boundarieswould refer to the set [9,10,20,21] at the point of use.

Given the potential variation in membership of anamed set, it must be possible to distinguish betweenthe two points of evaluation. Dynamic sets are evalu-

ated at the point of use, while static sets are evaluatedat the point of definition. Because dynamic sets willbe the most common, all named sets are dynamic bydefault. There is a special syntax for indicating that setmembership should be evaluated statically.

Machine State, Program State, and Debugger State.A central concept of debugging is that program exe-cution causes a series of orderly transitions from onestate to another. In parallel debugging, these states areactually collections of state information about all pro-cesses and threads in the parallel program. State canbe reported in terms of the machine-level operationsthat actually execute the program, the higher-level op-erations that the programmer specified in writing thesource code, or the additional information maintainedby the debugger in order to control program execution.

Machine state is expressed in machine-level termssuch as addresses and registers, and includes the follow-ing for each process: process memory; each thread’sregister set; each thread’s stack content; and eachthread’s execution state (running, stopped/runnable, orstopped/held). A debugger has the ability to examinemachine state, make modifications to it (e.g., changingthe contents of a memory location or register), and inferrelationships.

Rather than assisting the user in accessing machinestate directly, HPD-conforming debuggers provide ahigher level interface to the machine via program state.Program state is an interpretation of machine state interms of the high-level language of the source code.Specifically, code addresses are related back to sourceprogram statements, data addresses are related backto source program variables, and the contents of stackaddresses are expressed in terms of a call stacks, re-flecting the source program’s invocation of functions


and subroutines. Note that there will be a separate callstack for each process or thread that is executing theprogram.

While program state describes the actual state of theexecuting target program, debugger state refers to theinformation that the debugger maintains in order to in-terpret and respond to user commands. The informa-tion in debugger state is defined by the user directly(e.g., defining aliases), modified as a result of a moregeneral command (e.g., setting a breakpoint), modifiedas a result of a program event (e.g., triggering a break-point), or determined by examining program state (e.g.,evaluating the current state of a thread).

The HPD Standard provides commands for examin-ing many features of program and debugger state. Itdoes not allow the user to interact with the target pro-gram at the level of machine state (so-called instruction-level debugging).

Managing Symbols and Expressions. Many com-mands refer to one or more program objects by usingsymbol names as arguments. In addition, some com-mands take expressions as arguments, where the ex-pression may contain symbol names representing pro-gram variables. Thus, the rules for forming and in-terpreting symbol names affect much of a debugger’sfunctionality. The debugger learns about a program’ssymbols and their relationships by reading the debug-ging information that was generated during programcompilation. The information includes a mapping fromsymbol names to descriptions of objects and, for eachsymbol, details about what kind of symbol it is (e.g.,a function), where it is located in memory once theexecutable has been loaded, and other associated fea-tures (e.g., number and data types of any function ar-guments).

Two distinct concepts are relevant in discussing theinterpretation of symbol names: context and scope.Context is dynamic and implies where the program’spoint of execution is currently located. In serial debug-ging, this can be specified by naming a line-numberfrom the source code and the contents of the call stackcorresponding to the single thread of execution. Sincean HPD debugger must keep track of potentially verymany threads, its execution context is a 4-tuple withthe components: {process, thread, frame, active-line}.Each thread has its own execution context, which mayor may not be the same as that of other threads. Theprocess and thread components uniquely identify thethread. The frame and line components indicate wherethe location is with respect to the thread’s call stack.

With respect to symbols, scope refers to a region ofa program’s source code that has a specific set of sym-

bols associated with it. Thus, while context reflectsprogram dynamics, scope is a static concept. When-ever a thread’s execution context lies in a given scope,the associated symbols may legitimately be referenced.Every piece of debuggable code has a single such scopeassociated with it.

The algorithm for symbol lookup is language depen-dent. For most languages with which we are concerned,however, the general process is similar. The currentsymbol’s execution context determines a target scope,whose associated symbols are searched for a matchingname. If the name is not found, the symbols for thescope that contains the target scope are searched; thisoccurs recursively, so the symbols for the containingscopes are searched outwardly, in order of the scopes’nesting. Symbols for any scope that does not con-tain the original scope are not searched; these symbolsare out of scope (i.e., cannot be matched). Additionallanguage-independent rules often apply. For instance,if lookup is performed on behalf of a breakpoint com-mand, only a symbol corresponding to a procedure willbe matched; for a watchpoint, only one representing avariable; and so forth.

Symbol names are used in expressions, which arean important part of the data display and manipulationfunctions of the debugger. In effect, expressions iden-tify the data to be processed. In debuggers, expressionsappear in two contexts: expressions that are evaluatedto yield a result value (e.g., arguments to print),and expressions representing memory locations to bechanged. Some languages allow very complex expres-sions in one or both contexts. For example, Fortran 90is famous for its array sub-language. Although HPDFencourages implementors to provide support for thelargest possible sub-language, it does not require allimplementors to do so. Only a subset of expressionsand operators are required by the standard.

Note that when multiple threads execute the sameportion of source code, a single scope can be reflectedin dozens (or hundreds) of execution contexts. Thus,when a user asks the debugger to display or change thecurrent value of a symbol, the question arises of whichone(s) of potentially many values is desired. This isanother case where the thread set is used to managedebugger processes; the command is interpreted as re-ferring to just those instances of the symbol that cor-respond to threads in the current set (or the explicitlyspecified set).

Managing Command Output from Multiple Threads.Another area where the HPD Standard had to extendserial debugging mechanisms was the display of out-


put. If the target thread set for a command includesvery many members, there could be a potentially largeamount of output,much of which may be repetitive. Forexample, consider the command to display the valuesof an 100-member array. If multiple threads each havetheir own copies of the array, the user could quickly beinundated by the output.

The HPD Standard introduces the concept of outputaggregation to reduce output volume and repetition.As a convenience to the user, when the output acrossmultiple processes and/or threads is identical, the de-bugger “collapses” it, displaying just a single copy ofthe values. Suppose, for example, that a user wishesto display the contents of the 100-member array, whichhas just been initialized and copied to all threads. Evena small number of threads would result in a lengthylisting that would make it difficult to detect whether allarrays do or do not have the same contents. Aggregatedoutput provides a means of summarizing the contentsof all threads which have the same values. If the arrayis the same on all threads, only one copy is displayed.If one thread has different values, these are displayedseparately.

3. Analysis of the standard

Version 1 of the HPD Standard was designed to meetthe specific goals described in Section 1. As such, itspecifies a command language that can be used to debugparallel programs. The language provides a core setof commands in the sense that a user can accomplishthe basic functions necessary for parallel debuggingusing only these commands. In particular, it meets theBaseline Development Environment requirements fordebuggers defined by two national HPC task forces [14,15].

On the other hand, the standard is not exhaustive byany means. A number of commands that users haveidentified as desirable are not yet included in the stan-dard, primarily due to the time and scope constraintsestablished by the Forum. Indeed, it is expected thatHPD-conforming debuggers will also include featuresthat are not covered by the standard.

In this age of graphical user interfaces (GUIs), it mayseem odd and even futile to have spent so much energydefining a command-line interface. Surely debuggerimplementors can and must deliver attractive, GUI-based debuggers to be competitive. Nevertheless, thechoice to restrict the first standard to a command-lineinterface was important for four major reasons.

1. Surveyed users were clear that a textual interfaceis important, for dial-in access to remote systems,for supporting “scripts” of frequently-used com-mand sequences, and for some, because of per-sonal preference in terms of interface style.

2. The standards group was able to focus on the basicfunctionality of debugger operations, rather thanon the presentation of the commands, streamlin-ing the process of standards definition.

3. A standard command interface will provide a ba-sic infrastructure on which tool developers canlayer machine-independent interfaces, includingGUIs – something which has not been possible todate.

4. The fact that company-specific GUI guidelinesneed not be adhered to made it possible to reachconsensus sooner.

Thus, defining the functionality and terminology of acore set of commands not only yields something that isuseful in its own right, but establishes a solid foundationfor subsequent efforts to define standard GUI elementsfor debuggers and related tools.

3.1. Conformance to the standard

HPDF determined that a debugger will conform tothe standard if it implements all commands requiredby the standards document [8], preserving the speci-fied syntax and semantics. In a few cases, the require-ments vary according to whether the debugger supportsthreads-only, processes-only, or multilevel target pro-grams. While such variations are permitted, each de-bugger implementation is expected to conform to ex-actly one of the three models.

That does not mean that all HPD-conformant debug-gers will be identical. It is expected that individualimplementations will provide additional functionalityand commands. Conformance will hold, however, aslong as such additions do not conflict with the HPDdefinitions.

A significant amount of the group’s time was spentexploring issues of implementability. Compiler and op-erating system support for debugging vary widely fromone parallel computing platform to another. Therefore,it may be that a particular implementation is unable tosupport some features in a standard-conforming way.It is expected that a document will be provided indicat-ing where and how the implementation varies from thestandard.

In addition to specifying requirements, the HPDStandard presents a number of recommendations about


how features might be implemented most effectively.Consider the example of output from the command toprint the current value of a scalar. The standard re-quires that if the values are identical for consecutivelynumbered threads or processes, the lines must be ag-gregated for easier viewing. The recommendation isthat a more sophisticated algorithm be used, so that ifthe values from threads 0–5 and 7–10 are identical, butthread 6 differs, only two lines of output will be gen-erated (rather than three, which would conform to thestandard).

In addition, a number of extensions provide prelim-inary information about anticipated future versions ofthe standards or features that are considered importantby users but cannot be required at this point. For exam-ple, a high priority among users is the ability to store acheckpoint file at some intermediate point during pro-gram execution, and then later on load the file into thedebugger and continue execution from that point. Thiscould not be incorporated into the standard because theability to store and/or restart checkpoint files is depen-dent on operating system support that is not availableon most current machines. The extension to the HPDstandard specifies what the syntax should look like ifsuch a feature can be implemented, however. Exten-sions are based on the group’s discussions about partic-ular debugger operations and reflect the direction futurestandards are likely to take. Thus, they are meant togive guidance to developers who are currently workingon such functions.

3.2. Process used to develop the standard

The process used to develop the HPD Standard, Ver-sion 1 was successful in producing a standard withinthe time frame established as a goal. Looking backon the process reveals that there were five factors inparticular contributing to this success as follows:

– timing– participation of key players– limited duration of effort– limited scope of effort– structure of the Forum

The timing of the effort was close to ideal. At thestart of the Forum, most of the HPC vendors were en-gaged in the early development phases of new or re-vised debuggers for their systems. They were inter-ested in settling on the key issues in a timely fashion.Furthermore, the participants were already dealing withthe relevant issues, so that the standards effort was not

completely orthogonal to their work responsibilities.They also were able to consider the issues being dis-cussed with an awareness of how things should work intheir own debuggers without being entirely constrainedby how things were already implemented.

Another facet of the timing factor was that the needfor a standard was already agreed upon by the HPCcommunity. Moreover, the members of the Forum col-lectively had enough experience with other, previousstandards efforts to know what perils to watch out forand how much compromise would actually be neces-sary. Thus, the group began this effort with a solidagreement to make it happen and a good understandingof what was necessary to do this.

The Forum had the participation of key players fromindustry, debugger research, and the HPC user commu-nity, with a relatively constant group of people presentat all meetings. These two factors greatly contributedto the success of the standard’s effort. Table 1 liststhe HPC companies which participated in the Forum.The debugging research community was representedby members from both academia and federal researchlabs. The user community participated in two ways: in-dividuals who attended the regular meetings (includingrepresentation from the EuroTools group), plus mem-bers of the larger Ptools community who provided for-mal feedback on the decisions being made. There wasalso a significant amount of input from the generalHPC community, which was facilitated by making de-tailed minutes of every meeting, email archives fromthe working groups, and all working documents for thestandard available on the Web in a timely fashion.

The Forum had the benefit of reviewing previousstandards groups dealing with HPC issues. Their ex-periences made it clear that the duration of the effortneeded to be relatively short. It was decided early onthat the target time-line was one year to develop thefirst phase of the standard and one year for implemen-tation of that standard. By clarifying this up front, thegroup as a whole knew that discussions could only goon so long before decisions needed to be made. Inaddition, industry representatives knew that resolutionof the issues would happen within a finite time so thatthey could plan their development work accordingly.

In order to keep to the time-line of one year fordefinition plus one for implementation, the Forum wasforced to limit the scope of the project to a manageableload. For example, even though we knew that userswanted checkpoint capabilities, it turned out to be toocomplicated an issue to work out all the specifics in thetime we had and it was also determined to be infeasible


Table 1Industry participants in HPDF

Company Product

Cygnus Solutions software (gdb debugger)Dolphin Interconnect Solutions, Inc. (now Etnus, Inc.) software (Totalview debugger)Digital Equipment Corporation (now Compaq) hardware and softwareHewlett-Packard/Convex Division hardware and softwareHewlett-Packard hardware and softwareIBM hardware and softwareNumerex software (debugger not yet released)The Portland Group softwareSequent Computer Systems hardware and softwareSGI/Cray Research hardware and softwareSun Microsystems hardware and softwareTera Computer Company hardware and software

for most of the companies to implement in the nearfuture. Therefore, it was left as an extension to HPDVersion 1, rather than a requirement. Such decisionshad to be made frequently throughout the year and therewere definitely times when it was hard to contain thescope of the project. On the whole, however, the groupwas able stick with the time-line and let things go whennecessary in order to move forward.

The structure of the Forum consisted of three co-chairs, a number of working groups, and the group asa whole. The entire group met five times throughoutthe year for formal two- and three-day meetings. Theco-chairs were responsible for organizing these meet-ings, directing the discussions during the meetings, andposting the minutes of the meetings on the Web. Theworking groups met at the formal meetings but alsoworked via electronic discussion. In addition, the largergroup did a significant amount of work between meet-ings in order to work out details and draft the standarddocument in an appropriate way. Pancake served aslibrarian for the Standard document. Clearly, the targettime-line could not have been met without a high levelof work outside of the meetings.

3.3. Moving forward with the standard

Three companies, whose debuggers account for asignificant majority of current usage, have stated infor-mally that they intend to implement the standard. At theParallel Tools Consortium Annual Meetings which tookplace in April 1999 and May 2000, Etnus (formerlyDolphin Interconnect Solutions) and Cygnus both in-dicated that they anticipated availability of debuggersimplementing at least part of the standard within thenear future. In addition, two university research groupsthat are producing Java debuggers have indicated thatthey will base the interface on the HPD standard.

The University of Tennessee, Oregon State Univer-sity, and NASA Ames Research Center collaborated todevelop a reference implementation in order ot demon-strate feasibility and uncover any development prob-lems associated with the standard. They were success-ful in developing a front end to parse command in-put and pass it to a back-end based on the p2d2 dis-tributed debugger that was developed a few years agoat NASA/Ames. Due to funding limitations, however,only some of the HPD functionality was actually im-plemented and tested.

Since the standards group met, several debugger de-velopment efforts at specific companies have been dis-continued. This appears to have been due to a con-junction of two factors: decrease in funding levels forparallel tools at HPC hardware companies, and increas-ing reliance on the availability of debuggers furnishedby third-party software companies. Two companieshave indicated informally that if they do fund debug-ger efforts in the future, they intend to implement thestandard.

Finally, the companies involved in HPDF plus threenon-U.S. companies (NEC, Fujitsu, and Pallas) partic-ipated in a 1998–99 task force to establish guidelinesfor specifying software requirements on procurementsof HPC machines [14]. This effort involved representa-tives from a variety of user sites, charged with identify-ing key software that should be available in a consistentway across individual vendor platforms. The result-ing guidelines, which represent consensus of the usersand vendor representatives, included an HPD confor-mant debugger as a key requirement. There is evidencethat HPD debugger is already being included on newsoftware purchase requests.

The implementation process is obviously critical indetermining the suitability and efficacy of any stan-dard. Etnus is the first company to release an HPD-conforming debugger (the product was announced in


Spring of 2000). Indeed, their progress on implemen-tation efforts has suggested a number of refinementsand clarifications to the standard. These are currentlyunder discussion and should be submitted for formalHPDF approval by early 2001. It is hoped that theinitial implementation of a Java debugger conformingto the HPD command-line interface will be availablefrom one of the university efforts shortly thereafter.

Finally, while the availability of a standard command-level debugging interface is clearly an improvement, itis not the only support needed for debugging HPC ap-plications. The Forum plans to initiate a second phaseof development, which will extend efforts into one ofthe areas that had to be ignored to to the factors out-lined in the introduction. Among the possible topics tobe addressed by version 2.0 of the HPD standard, thefollowing are certain to be central:

1. graphical displays of array data2. specific features to support debugging of message-

passing programs, and3. at least some level of standardization for GUI

implementations

The High Performance Debugging Forum hasdemonstrated that it is possible for tool users and de-velopers to collaborate successfully. The participationof both groups was central in arriving at a definitionof the software support needed for debugging HPC ap-plications. Further, the fact that so many companiesparticipated was important in establishing models thatwould be feasible across a broad variety of platforms.While the group was not able to arrive at a “silver bul-let” solution for all problems associated with paralleldebugging, it was highly successful in circumscribinga scope that both addressed critical needs and could bedealt with in a timely fashion. The real outcome, ofcourse, will not be proven until multiple vendor prod-ucts using the standard have been put into productionuse.

Acknowledgements

Special acknowledgement must be made to theWorking Group leaders, who probably ended up do-ing much more work than they originally anticipated!These included Richard Title (HP), Charles Koelbel(Rice University), Gail Alverson (Tera), Robert Hood(NASA Ames Research Center), Janis Johnson (Se-quent), and Shirley Browne (University of Tennessee).

The Parallel Tools Consortium formally sponsoredthe program, providing its Web site and access to its

user community. Sessions of its annual meetings in1997 and 1998 were devoted to HPDF progress, and theSteering Committee assisted in identifying appropriatemembers from the industry and research sectors.

The HPC Modernization Program of the US Depart-ment of Defense helped defray the costs of meetingsand of maintaining the shared Web services. This wasmade possible through an agreement with NorthrupGrumman Corporation, on behalf of the NAVO MajorShared Resource Center, to Oregon State University.Meeting planning and coordination services were pro-vided by Oregon State, with assistance from variousmembers of the Forum.

Appendix. Commands required by HPD Version 1

– General Debugger Interface

∗ alias – Create or view user-defined com-mand(s)

∗ unalias – Remove previously defined com-mand

∗ history – Reference the session commandhistory

∗ set – Change or view the value(s) of a debuggerstate variable(s)

∗ unset – Restore default setting for a debuggerstate variable(s)

∗ log – Start or stop the logging of debuggerinput/output

∗ input – Read and execute commands stored ina file

∗ info – Display debugger environment infor-mation

∗ help – Display help information

– Process/Thread Sets

∗ focus – Change the current process/thread set∗ defset – Assign a set name to a group of

processes/threads∗ undefset – Undefine a previously defined

process/thread set∗ viewset – List the members of a pro-

cess/thread set∗ whichsets – List all sets to which a pro-

cess/thread belongs

– Debugger Initialization/Termination

∗ load – Load debugging information about tar-get program and prepare for execution


∗ run – Start or re-start execution of target pro-cess(es)

∗ attach – Bring currently executing pro-cess(es) under control of the debugger

∗ detach – Detach debugger from target pro-cess(es), leaving target process(es) executing

∗ kill – Terminate execution of target pro-cess(es)

∗ core – Load core-file image of process(es) forexamination

∗ status – Show current status of processes andthreads

∗ quit – Terminate the debugging session

– Program Information

∗ list – Display source code lines∗ where – Display the current execution location

and call stack∗ up – Move up one or more levels in the call

stack∗ down – Move down one or more levels in the

call stack

– Data Display and Manipulation

∗ print – Evaluate and display the value of aprogram variable or expression

∗ assign – Change the value of a scalar programvariable

– Execution Control

∗ go – Resume execution of target process(es)∗ step – Execute statement(s) by a specific pro-

cess/thread∗ halt – Suspend execution of target process(es)∗ wait – Block command input until target pro-

cess(es) stop

– Actionpoints

∗ break – Define a breakpoint∗ barrier – Define a barrier point∗ watch – Define an unconditional watchpoint∗ actions – Display a list of actionpoints

References

[1] Convex Computer Corporation, Convex CXdb User’s Guide,Second Edition, October 1993, DSW-473.

[2] R.H. Crawford, R.A. Olsson, W.W. Ho and C.E. Wee, Seman-tic Issues in the Design of Languages for Debugging, Com-puter Languages 21(1) (April 1995), pp. 17–37.

[3] Cray Research, Inc., UNICOS Symbolic Debugger ReferenceManual, June 1991, SR-2091 6.1.

[4] Digital Equipment Corporation, Ladebug Debugger Manual,Version 4.0, March 1996, AA-11 yPZ7EE-TE.

[5] Dolphin Interconnect Solutions, Inc. (now Etnus, Inc.), To-talView Multiprocess Debugger User’s Guide, Version 3.7.7,September 1997.

[6] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchekand V. Sunderam, PVM3 User’s Guide and Reference Manual,Oak Ridge National Laboratory, Oak Ridge, TN, September1994, ORNL/TM-12187.

[7] Hewlett Packard Company, HP/DDE Debugger User’s Guide,First Edition, July 1996, B3476-90015.

[8] High Performance Debugging Forum, HPD Version 1 Stan-dard: Command Interface for Parallel Debuggers, C. Pancakeand J. Francioni, eds, Technical Report CSTR-97, Dept. ofComputer Science, Oregon State University, 1997. Also avail-able online at http://www.ptools.org/hpdf/draft.

[9] R. Hood, The P2D2 Project: Building a Portable DistributedDebugger, in: Proceedings of the SIGMETRICS Symposiumon Parallel and Distributed Tools, Philadelphia, May 1996.

[10] IBM Corporation, IBM AIX Parallel Parallel Environment:Programming Primer, Release 2.0, June 1994, SH26-7223.

[11] Intel Corporation, Paragon Interactive Parallel Debugger Ref-erence Manual, October 1993, 312547-002.

[12] MasPar Inc., MasPar Programming Environment (MPPE)User Guide, Version 2.1, July 1991, 9305-0000.

[13] Message Passing Interface Forum, MPI: A Message PassingInterface Standard, International Journal of SupercomputingApplications 8(3/4), 1994.

[14] C.M. Pancake and C. McDonald, eds., Task Force on Re-quirements for HPC Software and Tools: Guidelines for Spec-ifying HPC Software, Technical Report 99-80-01, Dept. ofComputer Science, Oregon State University, March 1999.Also available online at http://www.nacse.org/distribution/HPCreqts/report.

[15] C.M. Pancake, Establishing Standards for HPC SystemSoftware and Tools, NHSE Review 2(1) (Nov. 1997). On-line journal available at http://nhse.cs.rice.edu/NHSEreview/97-1.html.

[16] C.M. Pancake, Collaborative Efforts to Develop User-OrientedParallel Tools, in: Debugging & Performance Tuning for Par-allel Computing Systems, M. Simmons and D. Reed, IEEEComputer Society Press, 1996, pp. 355–366.

[17] C.M. Pancake, Software Support for Parallel Computing:Where Are We Headed? Communications of the ACM 34(11)(1991), 52–64.

[18] Parasoft Corporation, nCUBE 2 Programmer’s Guide,Rev. 2.0, December 1990, pp. 102294.

[19] Silicon Graphics, Inc., CASEVision/Workshop User’s Guide,(Vols I and II), April 1992, 007-1523-020 and 007-1524-020.

[20] R. Stallman and Cygnus Support, Debugging with GDB,Cygnus Solutions, Inc., 1994.

[21] SunSoft, Inc., Solaris Application Developer’s Guide, 1997,ISBN 0-13-205097-8.

[22] Thinking Machines Corporation, Prism User’s Guide, Ver-sion 1.2, March 1993.

Submit your manuscripts athttp://www.hindawi.com

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


Distributed Sensor Networks


Advances in

FuzzySystems

Hindawi Publishing Corporationhttp://www.hindawi.com

Volume 2014


ReconfigurableComputing

Hindawi Publishing Corporation http://www.hindawi.com Volume 2014


Applied Computational Intelligence and Soft Computing

Advances in

Artificial Intelligence


Advances inSoftware EngineeringHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications


Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014

Advances in

Multimedia


Biomedical Imaging


ArtificialNeural Systems

Advances in


RoboticsJournal of



Computational Intelligence and Neuroscience

Industrial EngineeringJournal of


Modelling & Simulation in EngineeringHindawi Publishing Corporation http://www.hindawi.com Volume 2014

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014


Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in


a debugging standard for high-performance...

Documents