information technology measurement and testing activities at nist

Volume 106, Number 1, January–February 2001Journal of Research of the National Institute of Standards and Technology

[J. Res. Natl. Inst. Stand. Technol. 106, 341–370 (2001)]

Information Technology Measurement andTesting Activities at NIST

Volume 106 Number 1 January–February 2001

Michael D. Hogan, Lisa J. Carnahan,Robert J. Carpenter, David W. Flater,James E. Fowler, Simon P. Frechette,Martha M. Gray, L. Arnold Johnson,R. Michael McCabe, DouglasMontgomery, Shirley M. Radack,Robert Rosenthal, and Craig M.Shakarji

National Institute of Standards andTechnology,Gaithersburg, MD 20899-0001

[email protected]@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]

Our high technology society continues torely more and more upon sophisticatedmeasurements, technical standards, and as-sociated testing activities. This was truefor the industrial society of the 20th cen-tury and remains true for the informationsociety of the 21st century. Over the lasthalf of the 20th century, informationtechnology (IT) has been a powerful agentof change in almost every sector of theeconomy. The complexity and rapidlychanging nature of IT have presentedunique technical challenges to the NationalInstitute of Standards and Technology(NIST) and to the scientific measurementcommunity in developing a sound mea-surement and testing infrastructure for IT.This measurement and testing infrastruc-ture for the important non-physical andnon-chemical properties associated withcomplex IT systems is still in an early stage

of development. This paper explains keyterms and concepts of IT metrology, brieflyreviews the history of the National Bu-reau of Standards/National Institute of Stan-dards and Technology (NBS/NIST) in thefield of IT, and reviews NIST’s current ca-pabilities and work in measurement andtesting for IT. It concludes with a look atwhat is likely to occur in the field of ITover the next ten years and what metrologyroles NIST is likely to play.

Key words: conformance; hardware; infor-mation technology; interoperability; mea-surement; performance; reference data;software; testing.

Available online: http://www.nist.gov/jres

Glossary

ADP Automatic Data ProcessingAES Advanced Encryption StandardAFIS Automatic Fingerprint Identification

SystemAH Authentication HeaderAIAG Automotive Industry Action GroupANSI American National Standards InstituteANSIDT American National Standard Dictionary

of Information TechnologyAPs Application Protocols

ARPA Advanced Research Projects AgencyATEP-CMS Algorithm Testing and Evaluation

Program for Coordinate MeasuringSystems

ATSs Abstract Test SuitesBASIC Beginner’s All-Purpose Symbolic

Instruction CodeCAD Computer-Aided DesignCATIA Computer Assisted Three-dimensional

Interactive Application

341


CC Common CriteriaCD-ROM Compact Disk- Read Only MemoryCEC ERIM Center for Electronic Commerce at the

Environmental Research Institute ofMichigan

CIS Component Interaction SpecificationCPU Central Processing UnitCMSs Coordinate Measuring SystemsCMT Cryptographic Module TestingCMVP Cryptographic Module Validation

ProgramCOBOL Common Business Oriented LanguageCORBA Common Object Request Broker

ArchitectureCSE Communications Security EstablishmentCSMA/CD Carrier Sense Multiple Access with

Collision DetectionDARPA Defense Advanced Research Projects

AgencyDAT Digital Audio TapeDES Data Encryption StandardDoD Department of DefenseDOM Document Object ModelDSL Digital Subscriber LinesDSS Digital Signature StandardESP Encapsulating Security PayloadFBI Federal Bureau of InvestigationFDT Formal Description TechniqueFIPS Federal Information Processing StandardGOSET French Operational Group for the

Standard for Exchange and TransferGPS Global Positioning SystemGSA General Services AdministrationHTML Hypertext Markup LanguageIEC International Electrotechnical

CommissionIEEE Institute of Electrical and Electronics

EngineersIETF Internet Engineering Task ForceIKE Internet Key ExchangeIP Internet ProtocolIPS Internet Protocol SuiteIpsec-WIT IPsec- Web-based Interoperability TesterISO International Organization for

StandardizationIT Information TechnologyITL Information Technology LaboratoryIUT Implementation Under TestLAN Local Area NetworkMCITT Manufacturer’s CORBA Interface

Testing ToolkitMEL Manufacturing Engineering LaboratoryMOSIS Metal Oxide Semiconductor

Implementation System

NBS National Bureau of StandardsNBSNET NBS NETwork-deployed in 1979NIAP National Information Assurance

PartnershipNIST National Institute of Standards and

TechnologyNIST Net NIST Network Emulation ToolNISTIR National Institute of Standards and

Technology Internal/InteragencyReport

NSA National Security AgencyNVLAP National Voluntary Laboratory

Accreditation ProgramOEMs Original-Equipment ManufacturersOMB Office of Management and BudgetOMG Object Management GroupPCM Pulse Code ModulationPDM Product Data ManagementPITAC President’s IT Advisory CommitteePKI Public Key InfrastructureSAs Security AssociationsSEAC Standards Electronic/Eastern Automatic

ComputerSHS Secure Hash StandardSI International System of UnitsSQL Structured Query LanguageSTEP Standard for the Exchange of Product

Model DataSWAC Standards Western Automatic ComputerTIMS Testability of Interaction-driven

Manufacturing SystemsTRAMS TRAce Measurement SystemTREC Text Retrieval ConferencesTS Technology ServicesUSPRO U.S. Product Data AssociationVHS Video Home SystemVIM The International Vocabulary of Basic

and General Terms in MetrologyVLSI Very Large Scale IntegrationW3C World Wide Web ConsortiumWSQ Wavelet Scalar QuantizationWWW World Wide WebXML Extensible Markup LanguagexDSL (generic) Digital Subscriber Line

1. Introduction

Our high technology society and economy rely moreand more upon sophisticated measurements, technicalstandards, and associated testing activities. This wastrue for the industrial society of the 20th century andremains true for the information society of the 21stcentury. The contributions of NBS/ NIST to the develop-

342


ment of measurements for physical and chemical prop-erties have been widely accepted from the earliest daysof the establishment of the institution. NBS/NIST mea-surement work is based on recognized scientific princi-ples and techniques, and, over the years, it has supportedthe evolving needs of industries such as steel, machinetools, automobiles, and chemical processing for accu-rate and precise measurements.

Over the last half of the 20th century, InformationTechnology (IT) became a powerful agent of change inalmost every sector of the economy. Today, IT continuesto play a significant role in improving products andservices, making people more productive, and support-ing a strong economy. The complexity and rapidlychanging nature of information technology have pre-sented unique technical challenges to NIST and to thescientific measurement community in developing asound measurement and testing infrastructure. Thismeasurement and testing infrastructure for the impor-tant non-physical and non-chemical properties associ-ated with complex information technology systems isstill in an early stage of development.

The American National Standard Dictionary of Infor-mation Technology (ANSDIT) [1] defines the term, in-formation technology , as follows:

The art and applied sciences that deal with data andinformation. Examples are capture, representation,processing, security, transfer, interchange, presenta-tion, management, organization, storage, and retrievalof data and information.

IT systems, which are typically a blend of hardwareand software, can be characterized as becoming moredigital. The hardware can be described as becomingever more complex and difficult to manufacture. Thesoftware can be characterized as becoming more com-plex and difficult to develop while easy to replicate. ITsystems, including computers, computer networks, tele-phones, telephone networks, televisions, and cable net-works, are ubiquitous, impacting all industries (manu-facturing, health care, education, etc.), and all aspects ofthe economy. The ongoing challenge is to develop ITmeasurement and testing methods that will ensure thatthese complex IT systems are usable, scalable, secure,and interoperable. With the exception of the measure-ment of time, measurement and testing of attributes ofdigital IT systems do not rely upon measuring with theInternational System of Units (SI). Rather IT measure-ment involves ascertaining the correctness of informa-tion or data when stored, processed, transferred, dis-played, managed, organized, retrieved, etc.

Definitions for several terms already mentioned in thecontext of IT will help to further conceptualize thescope of this paper.

The American National Standard Dictionary of Infor-mation Technology (ANSDIT) defines the terms, dataand information , as follows:

data (ANSDIT)Any representation subject to interpretation (such asthrough analysis or pattern matching) or to whichmeaning may be assigned, such as by applying socialconventions or special agreed upon codes. Data canbe processed by humans or by automatic means.

information (ANSDIT)(1) The meaning that is currently assigned to data bymeans of the conventions applied to these data. (2) Ininformation processing, any fact, concept, or mean-ing derived from data and associated context or se-lected from knowledge.

Figure 1, also from the American National StandardDictionary of Information Technology (ANSDIT), illus-trates the concepts of data and information processing.

The American National Standard Dictionary of Infor-mation Technology (ANSDIT) also gives the followingdefinitions for hardware and software :

hardware (ANSDIT)Any physical component capable of data processing,for example, computers, peripheral equipment.

software (ANSDIT)All or part of the programs, procedures, rules, andassociated documentation of a data processing systemor an information processing system. Software is anintellectual creation that is independent of themedium on which it is recorded.

From these definitions, it is apparent that hardware hasboth physical and other data processing attributes. Incontrast, software is composed of intellectual or logicalattributes, and, with the exception of time, physical at-tributes are not essential.

It is interesting to note that the terms, measurementand testing , are not defined in the same vocabularydocument. The International Vocabulary of Basic andGeneral Terms in Metrology (the VIM) [2] defines mea-surement but not test or testing . ISO/IEC Guide 2: 1996,Standardization and related activities—General vocab-ulary [3] defines test and testing but not measurement .

343


Fig. 1. Concepts of data and information processing.

measurement (VIM)Set of operations having the object of determining avalue of a quantity.

test (ISO/IEC Guide 2)Technical operation that consists of the determinationof one or more characteristics of a given product,process or service according to a specified procedure.

testing (ISO/IEC Guide 2)Action of carrying out one or more tests.

Measurement and testing appear to be defined so thatthese terms are either conceptually equivalent or, atleast, closely related. Therefore, the terms, measurementand testing, are often combined in this paper because oftheir rough equivalence or overlap. A distinction be-tween these terms is also sometimes made by consider-ing testing to be a measurement or measurements to-gether with a comparison to a specification.

2. Historical Development of ITMeasurement and Testing Activitiesat NBS/NIST

NBS/NIST activities that support the developmentand measurement of digital systems have their origins inthe work that the organization did during World War II.In the late 1940s NBS was invited to be the technical

consultant for the procurement of digital computers bythe Bureau of the Census, the Office of Naval Research,and the Department of the Army. When the computersthat were ordered from commercial sources were notready in time to carry out the needed functions, NBSmade plans to build a small “interim” computer thatwould provide the needed computing capabilities. Thisproject to build a computer was adopted partly becauseof the delay in delivery of the commercial computersand partly to enable NBS to gain experience in machineconstruction and design [4]. NBS was in a position totake on this task because it had developed significantexpertise in electronics during World War II, particu-larly in the production of electronic components anddevelopment of printed circuits.

NBS began construction of the Standards ElectronicAutomatic Computer (SEAC) in 1948 and completedthe project in 1950. SEAC was later renamed the Stan-dards Eastern Automatic Computer when a companioncomputer was built in California in 1956 and named theStandards Western Automatic Computer (SWAC).SEAC was the only stored program machine in theUnited States, and was the fastest such machine in theworld at the time. It had 512 words of memory (44 bitsper word), a cycle time of one megahertz, and punchedpaper tape input and output. SWAC was a parallel ma-chine with magnetic drum memory. SEAC was used towork on many problems including solving partial differ-ential equations by Monte Carlo methods, generatingoptimum sampling plans for the Census Bureau, calcu-

344


lating transient stresses on aircraft structures, and devel-oping accounting procedures for the Social SecurityAdministration. SEAC was retired in 1964, and its partsare at the Smithsonian Institution.

After the development of SEAC and SWAC by NBS inthe late 1940s and 1950s, computers became commer-cial realities. In the 1950s, the computer industry beganto grow despite often quoted pessimistic estimates of thenumber of computers that would be needed in the world.As computers were adopted by the public and privatesectors, Federal agencies no longer needed a govern-ment agency to develop computers for them, but neededinstead a base of government competency in computerhardware and software technology to assist them inmanaging and using the new electronic devices.

The technical expertise that NBS had gained in com-puter technology at that time was recognized when Pub-lic Law 89-306 (79Stat.1127) was enacted on October30, 1965, as an amendment to Title I of the FederalProperty and Administrative Services Act of 1949(63Stat.377). The purpose of this legislation, known asthe Brooks Act, was to “provide for the economic andefficient purchase, lease, maintenance, operation, andutilization of automatic data processing equipment byFederal departments and agencies.”

Under the Brooks Act, three agencies were givencentral management responsibilities for automatic dataprocessing (ADP) activities. The General Services Ad-ministration (GSA) was directed to coordinate andprovide for the economic and efficient purchase, lease,and maintenance of ADP equipment by Federal agen-cies. The Secretary of Commerce was authorized toprovide scientific and technological advisory services toFederal agencies and to GSA, to make recommenda-tions to the President relating to the establishment ofuniform Federal ADP standards, and to undertake nec-essary research in computer sciences and technology.The Office of Management and Budget (OMB) wasassigned responsibility for exercising fiscal and policycontrol over the executive functions assigned to GSAand the Secretary of Commerce.

The authorities assigned to the Secretary of Com-merce for standards development, technical assistanceand research were subsequently delegated to the Na-tional Bureau of Standards. Executive Order 11717,dated May 9, 1973, transferred to the Secretary of Com-merce all functions being performed by the Office ofManagement and Budget relating to the establishment ofGovernment-wide automatic data processing (ADP)standards, including the function of approving standardson behalf of the President.

As the government continued to expand its use ofcomputers in the 1960s, users experienced problems ofincompatibilities among hardware, software, and com-

puter-generated data. Standards were recognized as theprincipal tool for solving this problem. To carry out itsresponsibilities under the Brooks Act, NBS establishedthe Federal Information Processing Standards (FIPS)program to develop standards for data, programs, andcomponents, data communications, computer perfor-mance, applications and data, personnel and environ-ment and acquisition and reassignment of ADP prod-ucts. The Federal government supported thedevelopment of standards as an effective way to facili-tate the interchange and sharing of data, programs, andequipment by Federal agencies. This focus on standardshelped to increase awareness by government and indus-try of the need for compatibility and more effectiveutilization of ADP products and services.

During the 1970s and 1980s NBS/NIST worked withprivate sector standards organizations to develop techni-cal standards that were used by both public and privatesectors. At the time, NBS was one of few governmentorganizations with the technical expertise needed tosupport the development of both voluntary industrystandards and government-developed standards that metunique government needs. These standards were devel-oped by NBS, both in conjunction with standards orga-nizations and with other government agencies. Stan-dards that NBS thought suitable for use by othergovernment agencies went through an open, public re-view process before they were submitted to the Secre-tary of Commerce for approval. Approved standardswere issued as Federal Information Processing Stan-dards (FIPS) and specified for government-wide use.

A major augmentation of NBS’s responsibilities todevelop standards and guidelines for Federal computersystems occurred in 1987 when the Computer SecurityAct was passed. Managers at all levels in Federal orga-nizations were confronting difficult challenges in pro-tecting the information processed by their organiza-tions’ computer systems. IT was becoming an essentialcomponent of many government activities; many orga-nizations and individuals were communicating andstarting to do business via computer networks. Thesesystems and networks were vulnerable to threats such asintruders, hardware and software failures, outages, andviruses that cost users millions of dollars in lost time andproductivity. Federal managers needed ways to assurethat the confidentiality, integrity, reliability, andavailability of their information resources was safe-guarded.

Under the Computer Security Act of 1987, NBS wasdesignated as responsible for developing technical, man-agement, physical security, and administrative standardsand guidelines for the cost-effective security and privacyof sensitive unclassified information processed in Fed-eral computers. NBS/NIST worked with the private sec-

345


tor and with government agencies to address the com-mon needs of both groups of users for improved secu-rity. NBS/NIST carried out developmental projects insecurity awareness, user training, authentication tech-nology, contingency planning, network security, and en-cryption, and issued recommendations of good prac-tices, standards, and guidance.

In 1996, the Brooks Act was replaced by the Informa-tion Technology Management Reform Act (Public Law104-106). This legislation reenacted NIST’s responsibil-ities to develop standards and guidelines for Federalcomputer systems. The Secretary of Commerce was di-rectly authorized to approved standards and guidelinesdeveloped by NIST. Between 1968 and 1996, about 190of these standards and guidelines were issued as FederalInformation Processing Standards (FIPS) for use gov-ernment-wide. The FIPS series included standards forsecurity, interoperability, and interchange of data, manyof which adopted for government-wide use the volun-tary industry standards, that NIST/NBS staff membershelped to develop.

In 1996, the policy of issuing FIPS that adopted vol-untary industry standards was modified as NISTchanged its focus to concentrate on developing tests,measurements, proofs of concept, reference data andother technical tools to support the development of piv-otal, forward-looking technology. NIST determined thatFIPS were to be developed when there were compellingFederal government requirements such as for securityand interoperability and there were no acceptable indus-try standards or solutions.

This policy was in accordance with the NationalTechnology Transfer and Advancement Act of 1995(Public Law 104-113) and Administration policies,which supported the development of voluntary industrystandards both nationally and internationally as the pre-ferred source of standards to be used by the Federalgovernment. As a long time supporter of voluntary in-dustry standards, the Federal government prefers to usethese industry standards, rather than creating its ownstandards. As a result, the government is able to relyupon the private sector to supply it with goods andservices. NIST continues to collaborate with nationaland international standards committees, users, industrygroups, consortia, and research and trade organizations,to get needed standards developed, but focuses mainlyon issues of information security for new and revisedFIPS.

The need for tests, measurements, proofs of concept,reference data and other technical tools to support thedevelopment of new technology became more critical asinformation technology systems became more complex.NIST’s capabilities to measure and test the attributes ofdigital IT systems had their beginnings in the expertise

gained with the building of the digital electronic com-puter, SEAC, in the 1940s and 1950s and with the devel-opment of Federal Information Processing Standards(FIPS) in the 1960s. Some of the NBS scientists andengineers who worked on SEAC subsequently becameinvolved in measurements for magnetic computer stor-age media. Many of the NBS staff developing FIPS alsodeveloped testing methodologies and tests for the soft-ware that implemented the complex technical specifica-tions in many of the FIPS.

Four examples of past projects that helped to advancetesting and measurements of IT systems include mea-surements of the electrical and magnetic properties ofcomputer storage media, development of conformancetests for programming languages, development of inter-operability tests for local area networks, and perfor-mance measurements for parallel processors.

2.1 Measurements of the Electrical and MagneticProperties of Computer Storage Media

The ability to interchange data stored on digital mag-netic media was a new and important operational re-quirement of computer systems of the 1960s to 1970s.However, interchange between tapes produced by differ-ent manufacturers was not easily achieved because ofdifferences in dynamic signal amplitude response. Intesting tapes, NBS scientists found that there was a dy-namic signal amplitude response spread of almost 50 %on a comprehensive sample of manufacturers’ produc-tion control tapes. The off-the-shelf tapes produced bythese manufacturers would have had an even greaterspread in signal values, making it difficult to inter-change data from tapes produced by different manufac-turers.

The NBS Magnetic Media Storage Group addressedthis problem by developing standard reference materialsand calibration services for computer storage media.This enabled manufacturers of different types of storagemedia to design their products on the basis of a com-parison to a known and generally accepted standardreference tape or disk.

The standard reference materials and calibration ser-vices developed by NBS were used by government andindustry media testing laboratories and by manufactur-ers of the media and associated data storage equipment.NBS’s work led to improvements in product qualitycontrol and more efficient data processing operations ingovernment and industry. Tests made after the referencematerials and services had been available for only ashort time showed a marked improvement in the unifor-mity of values for dynamic signal amplitude response.This work was unique since there were no alternativeindustrial expertise and sources for these materials andservices in the 1970s and 1980s.

346


2.2 Conformance Testing for ProgrammingLanguages

Variations in the compilers that the Federal govern-ment acquired for the Common Business oriented Lan-guage (COBOL) programming language led to the needfor better quality control, and to NBS activities to de-velop conformance tests for programming languages.Compilers often implemented the features of the lan-guage that the implementers chose or more importantlyimplemented features incorrectly, resulting in difficultytransferring programs among different compilers. TheDepartment of Defense (DoD) started the developmentof tests for the COBOL [5] and Fortran programminglanguages. NBS also developed COBOL and Fortrantests that were added to enhance the earlier DoD tests.Later this work was transferred to NBS where test de-velopment was continued [6] [7]. This early work inprogramming language conformance test developmentset the stage for conformance test development by pri-vate sector and international companies in C andCOBOL respectively with technical guidance fromNBS/NIST and standards committees. In addition, testswere developed for other languages such as Beginner’sAll-Purpose Symbolic Instruction Code (BASIC) [8] ,Ada, and Structured Query Language (SQL) [9]. Thetests captured the technical description of standardsspecifications, and were used to measure whether prod-ucts implemented the specifications correctly.

The development of conformance tests comple-mented NBS/NIST participation in the development ofindustry technical standards, and helped support theimplementation and use of these standards. At the timethat the programming language standards were underdevelopment, NBS/NIST staff members had the scarcetechnical skills and experience that were needed to sus-tain the standards work in national and internationalstandards organizations. With the support of the mem-bers of the standards committees, NBS/NIST operatedtesting services that applied the test suites to products.Certificates of conformance were issued as appropriate.Experience in using the test suites helped to make thewriting of standards more precise because the specifica-tions became more definitive and contained fewer areasthat were left to definition by the implementers.

The primary purpose of these early IT measurementefforts was to help federal agencies buy quality prod-ucts, but the efforts also helped industry by advancingthe application of quality processes to compiler develop-ment. The use of conformance tests in Government op-erated testing services had a significant impact on thequality of products and their compliance to FIPS stan-dards [10]. By providing neutral ways to measure prod-ucts, the conformance tests helped buyers and users to

select standards-conforming products. At the same time,the tests helped developers find errors in their imple-mentations and thus to improve the quality of their pro-gramming language products.

2.3 Interoperability Testing for Local AreaNetworks

During the 1980s, NBS staff members played a piv-otal role in the development of local area networkingtechnology. During the early and mid 1970s experimentswere conducted on the Advanced Research ProjectsAgency (ARPA) Network, the predecessor to today’sInternet. The NBS Experimental Computer Facilityhosted the 6th ARPA Net node—a terminal interfacemessage processor. By the end of the 1970s the geo-graphically distributed resource sharing ARPA Networkcomputers sometimes stopped functioning, but the tech-nology was demonstrable. In contrast to these wide areanetworks, the new challenge was to understanding howto provide rich connectivity for a campus, office com-plex or factory floor where all the resources includingthe physical cable were locally owned, operated andadministered. This new approach was called a local areanetwork or LAN.

In May of 1979 NBS and the Mitre Corporationteamed up to cosponsor the first Local Area Communi-cations Network Symposium [11]. While significant ad-vances in broadband cable LAN technology were re-ported, the symposium attendees were particularlyenthusiastic about the emergence of a new technologycalled Carrier Sense Multiple Access with Collision De-tection or CSMA/CD. Digital Equipment Corporation,Intel Corporation and the Xerox Corporation produced aspecification for a three megabit per second coaxialcable based CSMA/CD system to meet the needs of the“office automation” community. A report was given onthe NBS work on a campus-wide CSMA/CD networkcalled NBSNET. From those experiences and with thehelp of many others, NBS started a voluntary interna-tional standards effort known as the Institute of Electri-cal and Electronics Engineers (IEEE) Project 802.Building on the good ideas of over 100 experts fromcompanies all over the world, a family of LAN standardsemerged.

NBS built a LAN protocol testing laboratory to studyand measure implementations of these new ideas in anattempt to help industry better understand interoperabil-ity and performance issues. Experiences with NBSNETand other commercial implementations of CSMA/CDled to the publication of Federal Information ProcessingStandard 107 [12] to foster interoperability of the myr-iad products offered to solve office automation prob-lems.

347


The manufacturing automation community and theprocess control industry needed a different LAN stan-dard and focused on Broadband Token Passing Bus.This technology eventually became another member ofthe Project 802 family of standards.

At the request of industry, NBS established a Manu-facturing Automation Protocol Testing Laboratory. Thegoal was to help end users and original equipment man-ufactures link automated machinery and smaller miniand micro systems and controllers from different manu-facturers, and enable the systems to work together on thefactory floor. Users wanted cost effective, off-the-shelfsolutions for distributed computer applications. Todemonstrate Token Passing Bus interoperability, a testmethodology was designed by NBS scientists and engi-neers and tested by equipment vendors working closelywith NBS.

This work resulted in a deeper understanding of theinteroperability issues affecting Broadband Token Pass-ing Bus protocols. By demonstrating specific interoper-ability problems in repeatable laboratory experimentsusing state-of-the-art measurements, NBS helped toachieve evolutionary changes in the emerging voluntarystandards that ultimately lead to better network prod-ucts.

2.4 Performance Measurement of ParallelProcessors

NBS/NIST developed Multikron, a series of VeryLarge Scale Integration (VLSI) instrumentation chipsand interface boards that capture performance data topromote both high-performance computing and flexiblescalable systems. The system measures the performanceof parallel processors and workstations on high-speednetworks by recording events triggered either by soft-ware memory writes or the transition of hardware sig-nals. The chips can either timestamp the captured dataand send it over a collection network or use it to controlcounters and clocks aboard the chip. The resulting accu-rate measurements permit researchers to understand thesource of performance bottlenecks and therefore learnhow to scale their system designs upwards without sig-nificantly perturbing the system under measurement.

Development of concepts and hardware to enable lowperturbation and high accuracy performance measure-ment of computer systems and networks began in themid-1980s at NBS. It was recognized that traditionalapproaches using software capture of a computer’s timeclock were inadequate: the many software instructionsentailed high overhead; computers usually had poor res-olution from their standard clocks. A substantial im-provement in time resolution and a great reduction ininstrumentation perturbation would occur if timestamps

and other data could be captured by the initiation of asimple “write” instruction to a memory-mapped device.Any timing clock and its reading were the responsibilityof the special hardware—the only system perturbationwas the “write” instruction that signaled the event. Thisapproach was one of two initially pursued in the courseof developing the NBS/NIST TRAce Measurement Sys-tem (TRAMS) and the later MultiKron devices. Earlyin the effort, inexpensive fabrication of small customintegrated circuits became available through the DefenseAdvanced Research Projects Agency (DARPA) MetalOxide Semiconductor Implementation System (MOSIS)service. This allowed much of the instrumentation hard-ware to be contained in custom integrated circuits de-signed expressly by NIST.

A second, parallel measurement approach within theproject consisted of a NIST-developed printed circuitboard which contained a pattern matcher, usuallymatching processor address lines, and a 36 bit times-tamp counter which was incremented at a 10 MHz rate[13]. Events were defined by the pattern matcher andcaused no perturbation in the computer’s sequence ofinstructions. When a pattern match was triggered, itcaused transfer of the current value of the timestampcounter and other identifying data to a first-in, first-out(FIFO) data memory. All this was outside of the mea-sured system. Event data was time-stamped to a 100 nsresolution. The event trigger could also cause the cap-ture of data in one or more NIST custom instrumentationchips, which each contained four counters that couldaccumulate information on the utilization of variousportions of the system under test.

While passive, pattern-matching triggering avoidedintroducing program perturbations, a need was recog-nized for considerably more trigger points than werepractical in such a matching system. Furthermore, exist-ing key pattern trigger features, such as virtual ad-dresses, would soon be unavailable when embeddedwithin a microprocessor chip. Events would be hard todefine and to observe, so for each event, it was decidedto accept the perturbation of the single computer“write” instruction. Explicit “write” instructions in thecode under test would define measurement events. Dataassociated with each “write” instruction could identifywhich active software trigger caused the data capture. Incommitting to this view, the number of available definedevents became far larger than with passive pattern trig-gering. Furthermore, events corresponded directly todetails within the code being investigated. Programmersliked this feature, which was necessary for unambiguousmeasurement values.

Fabrication improved rapidly. Larger-scale custom in-tegrated circuits were practical for small-scale experi-menters, so essentially all of the second-generation mea-

348


surement-assist hardware became contained in a singleMultiKron chip [14]. MultiKron has a 56 bit times-tamp counter and 16 performance counters (32 bits percounter). All the counters can be incremented at fre-quencies up to 50 MHz, for a 20 ns resolution. Multi-Kron chips have been attached to each processor inmultiprocessor computer systems, with all the Multi-Krons driven by the same clock frequency and withtheir timestamps synchronized. This allows a time-cor-related comparison of the actions on the various proces-sors in the system. In addition, hardware has been devel-oped that uses signals from the Global PositioningSystem (GPS) satellites to allow synchronization ofMultiKron chips at locations separated by thousands ofkilometers. This capability is important for measure-ment of computer and communication networks. [15,16].

3. Present IT Measurement and TestingActivities at NIST

In 1996, NIST management established an inter-labo-ratory study group to discuss and review the status,needs, and challenges of providing measurements forinformation technology (IT). The study group, com-posed of representatives from the Manufacturing Engi-neering Laboratory (MEL), the Information TechnologyLaboratory (ITL), and Technology Services (TS), had awide spectrum of experiences and perspectives on test-ing and measuring physical and IT quantities. The groupdeveloped a white paper (NISTIR 6025), Metrology forInformation Technology [17], which became the startingpoint for discussions with industry and with the scien-tific measurement community to address the overallsubject of metrology for IT. NISTIR 6025 suggested aconceptual basis for metrology, and reviewed IT testingmethods, the status of IT metrology, and opportunitiesfor advancing IT metrology.

Metrology is defined in the International Vocabularyof Basic and General Terms in Metrology (the VIM) asthe science of measurement. The VIM notes that metrol-ogy includes all aspects both theoretical and practicalwith reference to measurements, whatever their uncer-tainty, and in whatever fields of science or technologythey occur.

NISTIR 6025 concluded that IT metrology is a validbranch of metrology. IT metrology differs from physicalmetrology in several ways including: the SI dimension-ing system is not as relevant; less analytical methodsexist to quantify uncertainty: and the area is relativelynew compared to physical metrology. As a result, ITmetrology has its own unique set of challenges, oppor-tunities, and priorities.

IT system hardware can be very complex but rela-tively easy to measure because it relies upon mature andsophisticated physical and chemical measurement sci-ence. Software, on the other hand, is increasingly com-plex and difficult to measure due to the limitations ofpresent software metrics [18].

The concept of calibration is well understood in thephysical metrology community. Calibration means thatthe measurement of the value of properties is related tomeasurements on primary standards usually providedby the primary national laboratory. The relationship iscalled traceability. The purpose of calibration and trace-ability is to ensure that all measurements are made withthe same sized units of measurement to the appropriatelevel of uncertainty so that the results are reliably com-parable from time to time and place to place.

Traceability is the ability to relate individual mea-surement results through an unbroken chain of compari-sons leading to one or more of the following sources:national primary standards, intrinsic standards, com-mercial standards, ratios, and comparison to a widelyused standard which is clearly specified and mutuallyagreeable to all parties concerned.

The concept of calibration may not be directly rele-vant for software. An international report of the opensystems community, ISO/IEC TR13233: 1995 Informa-tion technology—Interpretation of accreditation require-ments in ISO/IEC Guide 25—Accreditation of Informa-tion Technology and Telecommunications testinglaboratories for software and protocol testing services[19], substituted the terms “validation and traceability”for the terms “measurement traceability and calibra-tion.” The report concluded that validation is to softwareand protocol test tools as calibration is to measurementequipment.

NISTIR 6025 identified measurement and test areasthat needed additional research and development to ad-vance the state of IT metrology. Some of these areas arebeing explored, while others have not been studied ordeveloped extensively.

Level of confidence in test results . Currently the qual-ity of an information technology product or compo-nent is assured without rigorous metrics for confi-dence in test results. Commercial producers ofsoftware use a combination of factors such as codeanalysis, beta testing and defect analysis to determinethat a product is “good enough” to release. Variousanalyses of code can be a step toward a more rigorousdefinition of certainty of a product’s quality, but moreneeds to be done to define the mathematical founda-tions and methods for assessing the uncertainty inquality determinations. IT metrology needs a set ofconcepts equivalent to calibration, traceability, and

349


uncertainty, which are important in physical metrol-ogy. If uncertainty could be calculated by statisticalmethods as for physical test results, the level of confi-dence in software could be calculated.

Interoperability testing . Work is needed to investigatethe implications of combinations of interoperabilitytesting. For example, if implementation A and imple-mentation B interwork and if implementation B andimplementation C interwork, what are the prospectsof implementations A and C interworking?

Automatic generation of test code . Developing testcode for IT conformance testing can be more timeconsuming and more expensive than developing thestandard or a product which implements the standard.Efforts are needed to specify more formally the stan-dard or specification and to generate test code fromthis formalization.

IT dimensioning or description system(s) . NISTIR6025 discussed general concepts of fundamental andderived units for IT metrology. These concepts couldbe expanded, and a general vocabulary could be de-veloped to describe the components, which compriseinformation systems. This would entail developing arich, standardized terminology to capture the func-tionality and capabilities of a software component, inaddition to the interface specifications. The definitionof these formal specifications in a standardized, rigor-ous way would enable designers and systems integra-tors to select software components with confidenceregarding the component’s capabilities and how itwill integrate into the system being built. Further-more, automated composition of systems based onspecifications will be possible once these types ofdefinitions exist and are widely deployed in a certifi-able way.

Software metrics . Software metrics are needed tomore rigorously measure and test increasingly com-plex software as it is being developed.

Algorithm testing . As researchers develop new al-gorithms, some means of measuring the performanceof these algorithms for comparison purposes isneeded. The requirement is for measures of perfor-mance such as benchmarking programs, which areused to measure specific aspects of a computer’s ca-pabilities. There are challenges: determination of atheoretical foundation for measuring the performanceof algorithms, and means of ensuring that implemen-tation-dependent performance results are meaningful.

NIST has been performing numerous activities to ad-vance IT measurement and testing and to assist industryin the development and application of measurements andtests for IT. Following are descriptions and summariesof a few selected NIST projects in the areas of confor-mance testing, interoperability testing, performancetesting, and reference data. These projects describepresent needs in IT measurement and testing and NIST’scapabilities to address such needs.

3.1 Conformance Testing

The concept of conformance testing has been appliedto IT for about thirty years, a relatively long period oftime for IT. From ISO/IEC Guide 2: 1996, Standardiza-tion and related activities—General vocabulary , a defi-nition can be derived as:

conformance testingSystematic examination of the extent to which aproduct, process or service fulfills specified require-ments by means of testing.

Conformance testing methodology is well developedand widely used. Testing methodologies have been de-veloped for operating system interfaces, computergraphics, document interchange formats, computer net-works, and programming language processors.

IT standards are almost always developed and speci-fied in a natural language, English, which is inherentlyambiguous. Sometimes the specifications are originallydeveloped or translated into a less ambiguous languagecalled a formal description technique (FDT). Because ofthe complexity and ambiguity, most testing methodol-ogy documents require the development of a set of testcase scenarios (e.g., abstract test suites, test assertions,test cases) which must be tested.

The standards developing activity usually developsthe standard, the FDT specification, the testing method-ology, and the test case scenarios. Executable test codewhich tests the test case scenarios is developed by one ormore organizations, often resulting in more than oneconformance testing product being available. However,if a rigorous testing methodology document has beenadhered to, it should be possible to establish whethereach product for testing conformance is a qualityproduct and an equivalent product. In some cases, anexecutable test code and the particular hardware/soft-ware platform it runs on become accepted as a referenceimplementation for conformance testing. Occasionally, awidely successful commercial IT product becomes boththe de facto standard and the reference implementationagainst which other commercial products are measured.

350


NIST has been a leader in developing conformancetesting methods and products for standards or specifica-tions to promote the development of testable standardsand specifications, the interoperability of products, andthe development of conforming products. Currentprojects focus on testing and measurement to expeditethe research, development, standardization, and com-mercialization of new technologies. The emphasis isplaced on developing tests early in the technology cycleto help IT developers improve the design of their prod-ucts.

3.1.1 Standard for the Exchange of Product ModelData

NIST/MEL has developed web-accessible confor-mance testing capabilities for use by software vendorsseeking to implement ISO 10303, Standard for the Ex-change of Product Model Data . ISO 10303 is the inter-national standard for the exchange of product modeldata. This standard, commonly known as “STEP,” wasdeveloped to facilitate electronic communication amongthe engineering and manufacturing software systemsused throughout the aerospace, automotive, and elec-tronics supply chains [20]. NIST has played a significantrole in industry’s development and adoption of Standard

for the Exchange of Product Model Data (STEP) [21].STEP is not a single standard but rather a complex suiteof standards aimed at different data exchange scenarios.The architecture of STEP includes a language for infor-mation modeling, methods for mapping informationrepresentations to exchange structures, integrated re-sources with which to construct application-specificspecifications, and the actual specifications that soft-ware vendors implement—known as Application Proto-cols (APs). STEP developers recognized the need toprovide conformance specifications in conjunction withthe APs early on. Conformance classes are specified aspart of each AP, and Abstract Test Suites (ATSs) arecalled for as well. The conformance testing process asenvisioned in the STEP standards is illustrated in Fig. 2.

Hence the standards community provided the basicnotion with which to test the conformance of softwareimplementing STEP. However, as a voluntary standardsorganization, the technical committees could notprovide the remainder of the infrastructure necessary toconduct conformance testing. Specifically there was theneed for unique software tools to facilitate STEP confor-mance testing, a laboratory to conduct the tests, and anorganization to ratify the results of the tests. In fact,NIST, in conjunction with the Center for ElectronicCommerce at the Environmental Research Institute of

Fig. 2. Conformance assessment process.

351


Michigan (CEC ERIM), had already developed many ofthe specialized tools necessary to conduct conformancetesting through its earlier work in STEP interoperabilitytesting. Industrial representatives from Boeing, GeneralMotors, and others provided the motivation to initiate aconformance testing service through their participationin the U.S. Product Data Association (USPRO). Con-current with the establishment of the STEP conformancetesting service in USPRO, the French OperationalGroup for the Standard for Exchange and Transfer(GOSET) was also in the process of establishing a con-formance testing service.

The USPRO STEP Certification service was formallyannounced in June 1998. Fundamental to the establish-ment of the service was the formation of a certificationcontrol board. The board was made up of representativesfrom the STEP user community (e.g., manufacturerssuch as Boeing and General Motors), vendors of STEPproducts (e.g., Parametric Technology Corp. and Uni-graphics Solutions) and the standards community (i.e.,the ISO subcommittee responsible for STEP, NIST). Therole of the board was to set policy with regard to testingrequirements, procedures, and to provide a forum for

resolution of certification issues. NIST provided guid-ance in the establishment of the board and acted asliaison to the American National Standards Institute(ANSI) and the National Voluntary Laboratory Accredi-tation Program for accreditation questions.

The certification process employed (shown in Fig. 3)is innovative in the sense that it is conducted via theInternet. Rather than having representatives come to avendor’s physical facility to conduct tests, vendorsdownload test cases from the test laboratory’s website,create the specified product representations in their sys-tems, and upload their results to the test laboratory’swebsite where they are analyzed by the test laboratorypersonnel. These transactions are conducted in a secureenvironment and privacy is assured. The internet-en-abled testing process also allows vendors to pre-test anddebug their products in advance of the formal test. Atthe testing laboratory, CEC ERIM, a suite of uniquesoftware tools, is employed both to generate test dataaccording to the conformance classes and test suitesspecified in the STEP standard as well as to analyze theresults vendors submit. The tools and processes utilizedat the testing laboratory are illustrated in Fig. 4.

Fig. 3. ISO 10303 certification process.

352


Fig. 4. STEP conformance testing tools.

The first USPRO certification was awarded in April1999 to Dassault Systemes for their Computer AssistedThree-dimensional Interactive Application (CATIA)computer-aided design product.1 Since that time threeother vendors’ products have been certified and thereare additional certifications in progress. At the sametime NIST has been working with GOSET to harmonizeU.S. certification requirements and procedures withthose offered in France. The expected result is that therewill be mutual recognition of STEP certifications inter-nationally.

3.1.2 Cryptographic and Other Security Standards

Security standards have special requirements for test-ing to assure that implementations do not contain errorsor flaws that could be used to attack systems. NIST/ITLhas developed conformance tests for cryptographic stan-dards that have been issued as Federal Information Pro-cessing Standards (FIPS). The Data Encryption Stan-dard (DES) [22] was the first cryptographic standard for

1 Certain commercial equipment, instruments, or materials are identi-fied in this paper to foster understanding. Such identification does notimply recommendation for endorsement by the National Institute ofStandards and Technology, nor does it imply that the materials orequipment identified are necessarily the best available for the purpose.

which ITL developed conformance tests. A testbed wasdesigned to validate hardware implementations of theDES. An implementation was verified by ITL if it cor-rectly performed a set of 291 test cases that had beendefined to exercise every basic element of the DES.

Currently, ITL coordinates validation programs forthe Federal Information Processing Standards (FIPS)that specify cryptographic functions. These standardsinclude the Data Encryption Standard (DES), DigitalSignature Standard (DSS) [23], Secure Hash Standard(SHS) [24], and Security Requirements for Crypto-graphic Modules [25]. The latter standard specifies theoverall requirements for the modules that implementcryptographic algorithms and methods, and is theframework for all of the cryptographic standards. Beforethis standard was in place, security implementations of-ten failed despite the use of strong encryption methods.NIST realized that standards for encryption algorithmswere not enough, and that there was a need for a methodto specify the characteristics of the cryptographic mod-ule. Cryptographic modules can be standalone equip-ment, add-on encryption boards embedded into a com-puter system, or software applications such as digitalsignature operations. When cryptographic logic is im-plemented in software, the processor that executes thesoftware is also part of the cryptographic module.

353


NIST/ITL established the Cryptographic Module Val-idation Program (CMVP) to manage the validation test-ing of cryptographic modules and algorithms that arespecified by FIPS. The CMVP has been carried out incooperation with the Communications Security Estab-lishment (CSE) of the Government of Canada since July1995. The tests are conducted by third-party laborato-ries, which have been accredited as Cryptographic Mod-ule Testing (CMT) laboratories by the National Volun-tary Laboratory Accreditation Program (NVLAP).Information on tested products is available to all. Prod-ucts validated as conforming to standards are placed ona Validated Products List, and are available for use bythe Federal agencies of the United States and Canada forthe protection of sensitive information. The CMVP pro-motes the use of validated products and provides Fed-eral agencies and the private sector with a security met-ric to use in procuring equipment containingcryptographic modules. Users selecting equipment onthe Validated Products List can have confidence that theproducts meet their requirements for security. The pro-gram validates a wide variety of modules including Pub-lic/Private Key encryption products, secure radios, andtokens.

NIST and the National Security Agency (NSA) ini-tiated a major effort to promote an internationally ac-cepted testing program for security products through theNational Information Assurance Partnership (NIAP).This joint activity is directed toward meeting the needsof information technology producers and users for testedsecurity products. The products are tested against theCommon Criteria (CC) for Information Technology (IT)Security Evaluation , an international standard (ISO/IEC15408) [26] for specifying and evaluating the securityfeatures of computer products and systems. The Com-mon Criteria (CC) can be applied to a wide range oftechnologies such as operating systems, database man-agement systems, Public Key Infrastructure (PKI), fire-walls, smart-cards, telecommunications switches, net-work devices, middleware, and applications. NIST,NSA, and the security organizations of Canada, France,Germany, the Netherlands, and the United Kingdomcontributed to the development of the CC to replace themany different security evaluation methods that hadbeen used in North America and Europe.

NIAP establishes objective test methods and testingprocedures for the cost-effective evaluation of securityproducts. Security evaluations are formalized testingand analytic processes that use the CC to determinewhether IT products have been developed correctly ac-cording to their specifications and whether they are ef-fective in countering stated security problems. The test-ing is done by accredited, independent testinglaboratories. The goal is to enable product developers to

get their products tested easily, and for users to haveaccess to information about tested products. The evalua-tions help users to determine whether IT products andsystems meet their security requirements.

3.1.3 Extensible Markup Language (XML)

NIST/ITL is developing conformance tests for Exten-sible Markup Language/Document Object Model(XML/DOM) [27]. The success of the World Wide Webis due in a large part to the success of the HypertextMarkup Language (HTML), and its universal methodsfor displaying data. XML affords a standards-based ap-proach to defining, interacting with, and exchangingdata for a variety of domain-specific applications. DOMdefines ECMAScript and Java bindings for interactingwith both XML and HTML data, permitting dynamiccreation and manipulation of web pages defined usingthese metalanguages.

Although the XML specification [28] was released in1999 by the World Wide Web Consortium (W3C), thereare already millions of XML pages appearing on theWeb. Virtually all application domains are looking touse XML to exchange structured information. In addi-tion, XML processors are beginning to appear in betaversions of popular web browsers and associated appli-cations. As such, conformance of these XML proces-sors to the W3C Recommendation will permit robust,interoperable solutions. NIST/ITL, in conjunction withindustry partners, has developed a robust test suite thatwill be used as a metrology tool for testing XML proces-sors.

3.2 Interoperability Testing

A generic definition for interoperability testing is:

interoperability testing (NISTIR 6025)The testing of one implementation (product, system)with another to establish that they can work togetherproperly.

Interoperability testing methodologies are not as wellestablished as conformance testing methodologies. In-teroperability testing usually takes one of three ap-proaches to ascertaining the interoperability of imple-mentations (i.e., commercial products). The first is totest all pairs of products. Typically an IT market can bevery competitive with many products and it can quicklybecome too time consuming and expensive to test all ofthe combinations. This leads to the second approach oftesting only part of the combinations and assuming theuntested combinations will also work together. The thirdapproach is to establish a reference implementation andtest all products against the reference implementation.

354


NIST develops methodologies and tools to facilitateinteroperability testing and pilot deployment of emerg-ing technologies. Interoperability testing is typically fo-cused on the early stages of product research and devel-opment. NIST helps industry conduct interoperabilitytesting by providing reference implementations and test-ing tools; and by often facilitating multi-vendor interop-erability testing events and testbeds.

3.2.1 IPsec Web-based Interoperability Tester

NIST/ITL developed the Internet Protocol securityWeb-based Interoperability Tester (IPsec-WIT) in re-sponse to the need of the Internet Engineering TaskForce (IETF) for an interoperability test system for In-ternet security protocols. The IPsec-WIT supports thecurrent Internet protocol suite as well as the future addi-tion of new functionality to the Internet Protocol suite(IPS) as the use of the Internet continues to increase.IPsec was developed around the Cerberus prototype.Cerberus is a reference implementation of security pro-tocols developed by the Internet Engineering Task Force(IETF), and used by many companies as a basis forproduct development and testing. Implemented as a ker-nel module for the Linux operating system, the Cerberusprototype supports IPsec authentication and encryption

services in both tunnel (i.e., firewall) and transport (i.e.,host) mode.

IPsec-WIT provides an innovative method [29] thatallows implementers and system integrators to remotelycontrol and execute a series of interoperability tests any-time and anywhere without requiring any distribution oftest system software, relocation of the systems undertest, or coordination of access to the reference imple-mentation or tester. Users include security researchers,product implementers and highly technical users such asnetwork engineers and administrators. This user com-munity is primarily interested in easily available andflexible test systems that provide detailed technical diag-nostics as systems and new applications are in the devel-opment phase.

IPsec-WIT is a novel use of World Wide Web(WWW) technology for the purposes of protocol inter-operability testing (see Fig. 5). The Web, or WWW,enables universal browsing among data repositories viathe Internet. The Web has been used as a testing tool totest other WWW related technology [30], and as a sim-ple configuration and control interface for remote proto-col implementations. Using a common web browser, anIPsec-WIT user can easily navigate through directoriesof pre-defined test cases, configure the test system, exe-cute tests, and view detailed test results.

Fig. 5. Ipsec-WIT architecture.

355


The test system is based upon NIST developed, pub-licly available reference implementations of IPsec (Cer-berus ) and Internet Key Exchange (IKE) (PlutoPlus ).Cerberus [31] is a Linux kernel reference implementa-tion of the IPsec [32] protocol, including both the En-capsulating Security Payload (ESP) [33] and Authenti-cation Header (AH) [34]. ESP and AH are encapsulationprotocols that provide cryptographic confidentiality(ESP), and authentication (AH), to individual IP pack-ets. PlutoPlus [35] is a Linux reference implementationof IKE [36], a dynamic key management protocol usedto negotiate security associations (SAs) and keying ma-terial between two systems. Fundamentally, the IPsec-WIT system is a wrapper and front end that drives theCerberus and PlutoPlus reference implementations andanalyzes their output.

Cerberus and PlutoPlus reference implementationswere designed to be completely operational stand-aloneprototypes. They were extended with additional controland diagnostic capabilities specifically for use in IPsec.PlutoPlus is a complex, stateful key management dae-mon. This complexity allows for interoperability prob-lems to occur at many different places in the protocol.Testing diagnostic statements were added extensively tothe PlutoPlus implementation. To allow the test systemto be used simultaneously by multiple users, PlutoPlushad to be modified to operate in a single negotiationmode in addition to its normal daemon mode. In singlenegotiation mode, the PlutoPlus process exits after eachIKE negotiation (i.e., a single test). This allows theIPsec-WIT test system to independently control IKEbehavior and correlate outputs for each user of the sys-tem.

The main engine of the IPsec-WIT system is a Perlcgi-bin script that ties the user interface HTML forms,directories of test cases, and prototype reference imple-mentations into a working interoperability test system.Interoperability testing scenarios usually involve an iter-ative series of quick tests and long debugging periods.To allow users to retain state regarding test configura-tions and progression through test directories betweendebugging sessions, simple state files were developed tocontain control parameters (e.g., tester configurationvariables, Implementation Under Test (IUT) addresses,current test case, etc.). These files are indexed by a userprovided unique user-id / password.

Test cases were customized with the user specificcontrol information in the state files. The test cases arethe primary component of this system. To deal withprotocols with many optional features, it was necessaryto organize the test cases into context relevant cate-gories. For IPsec, the majority of interoperability prob-lems are the result of non-interoperable or incorrectapplication of the cryptographic algorithms. Because of

this, the IPsec test cases were organized into test directo-ries based upon cryptographic algorithm (e.g., DES,3DES, etc.). Organized in this way, users can easily testvariations of a particular algorithm before moving on tothe next algorithm.

Interoperability issues with IKE were explored. Inany IKE negotiation there are several message ex-changes involving multiple cryptographic algorithms. Afailure at any point in the exchange will cause interoper-ability problems. IKE test directories were developedbased on the different cryptographic algorithms, and theability to make additional configuration changes to agiven test was provided. This allows the users the flexi-bility to test for a given negotiated algorithm and thenmake subtle changes to the test configuration to test formore IKE specific problems.

The test cases serve as both documentation and exe-cutable code. The test cases were written in HTML toeasily convey to the user the test purpose, how to config-ure the IUT for the test, and exactly how the test will beexecuted. The test cases were enriched with specialtagged variables that the Perl script replaces with userspecific control information, creating a test case tailoredfor a particular user.

The test system evolved from an IPsec test systemwith a handful of test cases to a robust test system thatincludes hundreds of IPsec and IKE test cases. Thisevolution resulted from the continued development ofthe NIST prototype implementations, feedback fromIPsec-WIT users, changes and clarifications in the IPsecand IKE specifications, observations on how IPsec-WITwas being used, and new ideas about the test system.

Multi-vendor interoperability has been the cornerstone for the explosive growth of the Internet. The in-creasing complexity of new protocols, such as thatfound in Internet security protocols, makes developinginteroperable products very difficult. IPsec-WIT helpsimplementers and standards organizations work towardthe development of interoperable products that are effec-tive and cost-effective. More than 350 implementers andusers have used IPsec-WIT to improve their formal test-ing procedures. The use of this system paves the waytoward the development of testing tools for other proto-cols as well.

3.2.2 STEP Interoperability Testing

From 1995 through 1997 NIST participated in theAutoSTEP project with the Automotive Industry ActionGroup (AIAG). The purpose of the AutoSTEP projectwas to introduce STEP into automotive industry usage,ensure that STEP met the needs of the industry, and todetermine the benefits the industry could derive fromusing the standard [37]. The scope of the project was the

356


exchange of product data for mechanical design usingthe STEP Application Protocol for configuration con-trolled 3D designs of mechanical parts and assemblies[38] for the purpose of packaging analysis. Participantsin the project included original-equipment manufactur-ers (OEMS), first and second tier suppliers, softwarevendors and consultants. Figure 6 illustrates the tier rela-tionships among the manufacturers involved along withthe computer-aided design (CAD) systems utilized byeach participant. The fundamental issue that had to beaddressed by the AutoSTEP project was whether or notSTEP would be an improvement over the data exchangemethods then employed by the participants. That is,would it result in a higher rate of successful file trans-fers? Would it be able to transfer data that other methodscould not? If STEP could be shown to reduce the costsof data exchange, save time, or overcome technical bar-riers in supply chain relationships, there would then bea business case for STEP adoption in this industry seg-ment.

To assess the efficacy of STEP for the project, partic-ipants defined meaningful metrics for OEM-supplierdata exchange scenarios (see Fig. 7). These metricsincluded simple metrics that could be computed directlyfrom the model exchanged as well as process metricsthat are determined by comparing simple metrics atdifferent stages in the exchange process. Some of thesimple metrics could be determined using functionalityavailable in the CAD systems being employed. Otherrequired unique software tools developed by CEC ERIMin conjunction with NIST specifically targeted testing ofSTEP implementations. Among the metrics utilized inthe project were:

• counts of different kinds of entities transferred ineach exchange;

• properties of the model transferred such as its cen-ter of gravity, volume, and mass;

• counts of violations of constraints specified in theSTEP standard;

• file size;• translation time;• the time required to make a model imported via

STEP into a valid model in the receiving CADsystem;

• the ratio of the time required to make an importedmodel valid versus the time required to recreate theoriginal model in the receiving CAD system.

Over the course of the project the rate of successfulfile exchanges using STEP improved dramatically, ex-ceeding 80 %. As the success rate began to plateau,factors not directly related to STEP began to emerge asimpediments to successful exchange. Chief among thesewere the issues of CAD system accuracy and bad mod-els. The accuracy issue was due to the mismatch innumerical accuracy maintained within different ven-dors’ products. Hence a model that is inherently accu-rate in one system (e.g., a model that forms a closedvolume) because of that vendor’s maintenance of, say,10�6 numerical accuracy may not form a closed volumein a system that only maintains accuracy of 10�3. Re-lated to this issue was that of bad models. In one systema model may be deemed valid despite seemingly minorrepresentational idiosyncrasies. Those same subtletiescould cause the receiving system to declare the model asinvalid. Errors of this type were initially attributed to

Fig. 6. STEP interoperability testing.

357


Fig. 7. Model exchange scenario used in the AutoSTEP project.

problems with STEP translation but further investigationrevealed their true pathologies. In the end, the Au-toSTEP achieved its goals of demonstrating the efficacyof STEP for OEM-supplier data exchanges and this hasresulted in use of the standard for actual commercialmanufacturing.

3.3 Performance Testing

The IEEE Standard Glossary of Software Engineer-ing Terminology , IEEE 610.12-1990 [39], defines per-formance testing as:

performance testingTesting conducted to evaluate the compliance of asystem or component with specified performance re-quirements.

Tools and techniques have been developed by NIST toassist in the performance measurement and characteri-zation of networking technologies, middleware, dis-tributed systems, and hardware components. The focusis on the research and development of the measurementmethodologies and the characterization of generic tech-nologies rather than the benchmarking of specific prod-ucts or implementations.

3.3.1 Manufacturer’s CORBA Interface TestingToolkit

NIST/MEL is developing a publicly available soft-ware toolkit that supports several kinds of testing (inte-gration, conformance, performance) of manufacturingsystem software components whose interfaces use theObject Management Group’s specification, the Com-mon Object Request Broker Architecture (CORBA).

As part of NIST’s laboratory efforts in conjunctionwith the Advanced Technology Program’s AdvancedProcess Control Framework Initiative [40] and the Na-tional Advanced Manufacturing Testbed Frameworkproject [41] a software toolkit was developed to addresstesting issues associated with distributed object archi-tectures. Specifically these projects were employing theCommon Object Request Broker Architecture(CORBA) [42] as the infrastructure upon which soft-ware components for manufacturing control were beingbuilt. While the CORBA infrastructure provides manyadvantages for implementation of distributed systems,testing the systems produced is difficult. Because com-ponents in such systems have complex interdependen-cies, testing the components in isolation is impossible.Hence NIST developed the Manufacturer’s CORBA In-terface Testing Toolkit (MCITT, pronounced “M-kit”)[43]. MCITT minimizes the effort needed to produce

358


simple emulations of servers, or dummy componentsthat can be used to replace actual servers in a testingscenario. The toolkit is available from NIST [44].

Two complementary mechanisms are provided in thetoolkit for defining the behaviors of dummy compo-nents. Each was developed specifically for use withMCITT and allow the toolkit user to specify CORBAserver, client, or overall system interactions. The Inter-face Testing Language is a procedural language forspecifying and testing the behavior of CORBA clientsand servers. Interface Testing Language allows for run-time assertions, script-like behavior specifications, con-formance testing commands, timed loops, as well assimple commands for creating and binding to CORBAobjects. The second mechanism provided is the Compo-nent Interaction Specification (CIS). A CIS is a textualspecification of an interaction scenario for an entiredistributed system. Unlike an Interface Testing Lan-guage file, a CIS can describe nearly all of the interac-tions between all of the components in the system. Onecaveat regarding CIS capabilities is that it does not sup-port the generation of client requests, though those canbe added manually using Interface Testing Language.Together these two mechanisms provide a powerful ar-senal with which to conduct a variety of tests in dis-tributed systems based on CORBA. Unit testing, inte-gration testing, system testing, performance testing,conformance testing, and stress testing are all possibleusing MCITT.

3.3.2 NIST Network Emulation Tool

The NIST Network Emulation Tool (NIST Net) is ageneral-purpose tool for emulating the performance dy-namics in IP networks. NIST Net was designed to allowcontrolled, reproducible experiments with network per-formance sensitive applications and protocols in a sim-ple laboratory setting. By operating at the Internet Pro-tocol (IP) level, NIST Net can emulate the criticalend-to-end performance characteristics imposed by var-ious wide area network situations (e.g., congestion loss)or by various underlying subnetwork technologies (e.g.,asymmetric bandwidth situations involving (generic)Digital Subscriber Line (xDSL) and cable modems).

NIST Net is implemented as a kernel module exten-sion to the Linux operating system and an X WindowSystem-based user interface application. In use, the toolallows an inexpensive PC-based router to emulate nu-merous complex performance scenarios, including: tun-able packet delay distributions; congestion and back-ground loss; bandwidth limitation; and packetreordering /duplication. The X interface allows the userto select and monitor specific traffic streams passingthrough the router and to apply selected performance“effects” to the IP packets of the stream. In addition to

the interactive interface, NIST Net can be driven bytraces produced from measurements of actual networkconditions. NIST Net also provides support for userdefined packet handlers to be added to the system. Ex-amples of the use of such packet handlers include: timestamping / data collection, interception and diversion ofselected flows, generation of protocol responses fromemulated clients.

3.3.3 Exchange of Fingerprint Images

Since the late 1960s, the Federal Bureau of Investiga-tion (FBI) and NBS/NIST have participated in a collab-orative effort to automate the Federal Bureau of Investi-gation’s (FBI) fingerprint identification processing andto develop standards for the exchange of fingerprintinformation [45]. Until 1990 the standard fingerprintcard was the primary source and repository for all fin-gerprint images. To automate the process for transmis-sion, search, and identification of digital fingerprint im-ages, the FBI decided to place their existing andprojected master file of 400 million plus fingerprintimages online. The typical size of a file needed to con-tain a single fingerprint image was estimated to be ap-proximately 600 000 bytes. To make the use of thesefingerprint images practical, each image file had to beconsiderably reduced in size to minimize storage spaceand transmission time resource demands. The WaveletScalar Quantization (WSQ) algorithm was developed asthe compression technique to be used. This lossy com-pression approach easily produces a 15:1 size reductionwith minimal loss in image fidelity.

The WSQ specification [46] outlines a comprehen-sive and flexible framework for compressing gray-scalefingerprint images using the wavelet scalar quantizationtechnique. Figure 8 is an overview of the operation of theWSQ encoder and decoder. The encoder uses the sourceimage as input. It then performs a decomposition of thefingerprint image into a number of subbands, each ofwhich represents information in a particular frequencyband. This decomposition is accomplished by a discretewavelet transformation of the fingerprint image. Each ofthe subbands is then quantized using values from aquantization table where the “lossy” attribute of thealgorithm and resulting data compression is introduced.These quantized coefficients are then passed to a Huff-man encoding procedure that produces the final com-pressed image file. The decoder portion of the al-gorithm essentially operates in reverse. A compressedimage is the input to the decoder. The Huffman, quan-tization, and transform tables are extracted from thecompressed file and used in the entropy decoder, de-quantizer, and transform steps to produce a recon-structed image.

359


Fig. 8. WSQ encoder/decoder overview.

To ensure adherence to the specification, NIST/ITLestablished a compliance testing program for vendorsand researchers to follow. An encoder implementationmust produce a syntactically and numerically correctcompressed file in accordance with the WSQ algorithm.The compressed file must be capable of being properlydecoded without any errors by an independent decoderimplementation. A decoder implementation must be ca-pable of correctly reconstructing an image file from asyntactically correct WSQ compressed file. When com-pared to the original image file, the reconstructed imagefile should be numerically accurate to within specifiedtolerances.

Validation for a specific WSQ implementation isbased on compressing a reference set of test images inaccordance with the WSQ specification. The tests speci-fied for encoders and decoders are not exhaustive testsof interoperability, numerical precision, or image fi-delity, but tests of correctness of the images tested. It isnot possible to test all images of all arbitrary sizes, usingall the features contained in the specification. The ITLapproach tests only part of all the possible combinationsthat can be used and assumes that untested combinationswill also function properly.

Compliance with the WSQ specification is deter-mined by comparing the output from the candidate im-plementation with the output from a double precisionreference implementation developed by NIST/ITL.Products are evaluated using this same reference imple-

mentation. The criteria for product validation use thetest results from candidate WSQ implementations todetermine if acceptable tolerance limits have been met.Tolerance limits were derived by comparing the resultsfrom the ITL reference implementation with resultsfrom an implementation developed by the Los AlamosNational Laboratory. The two implementations were de-veloped independently but in the same time frame. Inter-mediate results on 1800 processed fingerprint imagesfrom Special Database 9 [47] were compared betweenthe two implementations and the results were used todetermine the acceptable tolerances.

To have a WSQ encoder/decoder implementation val-idated, the product developer downloads the ITL refer-ence test set from the Internet. This test set consists of17 fingerprint images. Each image was scanned at aresolution of 19.69 pixels per millimeter (500 pixels perinch) and quantitized to 8 bits of gray-scale (256 levels).Dimensions of the image sizes ranged from 375 � 526pixels to 832 � 768 pixels. In addition to the rawscanned image file, a WSQ-encoded image file, and areconstructed image file is provided for each of theoriginal fingerprint image files. The developer returnstwo files for each of the 17 image files. Output data setsgenerated by a candidate implementation must matchthe output data sets from the ITL reference implementa-tion to within the stated accuracy requirements (see Fig.9).

360


Fig. 9. WSQ encoder and decoder compliance tests.

All federal, state, and local agencies doing businesswith the FBI are required to transmit and receive finger-prints as compressed fingerprint image files. The imple-mentation validation program aims to ensure that inter-operability, proper numerical precision, and imagefidelity are consistent among different implementations.When the compressed image files created for exchangesare in accordance with the WSQ Gray-scale, fingerprintimages can be exchanged between federal, state andlocal agencies. These organizations require that vendorscomply with the Fingerprint Image Compression Speci-fication and vendors obtain validation for their imple-mentations of the WSQ algorithm. To date, over onehundred implementations have been submitted andtested for compliance to the WSQ algorithm. Compa-nies who have participated in this program includeAware, Cogent, Harris, IBM, PRC, Printrak, andSagem-Morpho. As a result of this program, testing forproper performance of fingerprint image systems hasachieved reliable interoperability between the systems ofFBI and of different agencies using dissimilar Auto-mated Fingerprint Identification Systems (AFIS).

3.4 Reference Data

The use of reference data is very important in bothphysical and IT metrology. But the term has different

definitions for reference data as applied to physical andIT metrology:

reference data (NISTIR 6025)In physical metrology, reference data is quantitativeinformation, related to a measurable physical orchemical property of a substance or system of sub-stances of known composition and structure, which iscritically evaluated as to its reliability.

In information technology, reference data is any dataused as a standard of evaluation for various attributesof performance.

Reference data provide critical information for testingthe performance of complex IT systems. For many kindsof IT systems, NIST has played an important role incollecting and disseminating unbiased data that has beenused by and relied upon by developers to test the perfor-mance of their systems.

3.4.1 Algorithm Testing and Evaluation Programfor Coordinate Measuring Systems

NIST/MEL is identifying reference data to be used inthe testing of coordinate measuring machine algorithmsand to compare the performance of a coordinate mea-

361


suring machine with the performance of other machinesin a similar category. MEL has developed a softwaresystem, which can be used to assess the uncertainty incoordinate measuring machine algorithms.

Coordinate measuring systems (CMSs) are machinesused by manufacturers to determine dimensional char-acteristics of mechanical parts by analyzing 3-dimen-sional point data acquired on part surfaces [48]. CMSsemploy data analysis software to reduce measured pointcoordinates to curve and surface geometries. In turn,these geometries are compared to the tolerance limitsfor the part. It is the uncertainty of the geometriescomputed by the CMS data analysis software that deter-mines the quality of the measurement. In the pastdecade mounting evidence showed that data analysissoftware had significantly contributed to measurementerrors in CMSs. To help address the issue of data anal-ysis software quality, NIST developed an AlgorithmTesting System for CMS software and provides a SpecialTest Service through the NIST Calibration Program toassess the uncertainty of CMS software. The test ser-vice is known as the Algorithm Testing and EvaluationProgram for Coordinate Measuring Systems (ATEP-CMS).

The architecture of the NIST Algorithm Testing Sys-tem is illustrated in Fig. 10.

The data generator is driven by a test descriptioncontaining a high level description of the tests to beperformed. The generated data is then provided both toNIST’s highly accurate internal algorithms [49] and tothe CMS software under test. NIST’s algorithms yieldreference results to which the test fits returned by thesoftware under test are compared. The analysis com-pares geometric differences among fits and detects codeerrors and systematic bias in the CMS software. Testdescriptions can be tailored to estimate different aspectsof CMS software errors.

The ATEP-CMS is the first calibration service ever tobe offered by NIST for software. NIST’s experiencewith ATEP-CMS is assisting standards committee in the

American Society of Mechanical Engineers (B89.4.10,Methods for Performance Evaluation of CoordinateMeasuring System Software) and in the InternationalOrganization for Standardization (ISO) (TC213 WG10,Dimensional and geometrical product specificationsand verification) on development of CMS test proce-dures and test data sets. In addition, NIST is currentlyworking to make the ATEP-CMS a web-accessible cali-bration service.

3.4.2 Speech Corpora

For more than a decade, NIST/ITL has been involvedin developing reference data of standardized speech andnatural language to help the research community, indus-try, and universities measure and improve speech recog-nition algorithms and to facilitate error analysis anddiagnostics [50]. Initially, the collections of speech datawere recorded on 9-track magnetic tapes, which oftenproved unreadable. Later experiments used a hybrid sys-tem involving Beta and Video Home System (VHS)format video tapes on which Pulse Code Modulation(PCM) files were recorded. This was replaced by DigitalAudio Tape (DAT) and then by Compact Disk—ReadOnly Memory (CD-ROM).

The reference data that has been recorded and dis-tributed on CD-ROMs contains speech databases, re-ferred to as “corpora.” These corpora are collections ofdigitized waveforms, annotations, transcriptions, and re-lated data, and include “found speech” from radiobroadcasts as well as “read speech,” readings of pre-pared text. This work has been done in cooperation withthe Defense Advanced Research Projects Agency(DARPA).

Many developers of speech recognition and speechunderstanding applications use the speech corpora torefine their systems. In cooperation with developers andresearch institutions, ITL has prepared development andevaluation test sets for speech technology applications.The technology developers apply the tests and then

Fig. 10. Architecture of the NIST algorithm testing system.

362


provide ITL with the results of the tests that they haveimplemented. ITL analyzes the data, and prepares uni-formly-scored tabulated results, including the results ofnumerous paired-comparison statistical significancetests and other analyses. These results assist developersin improving the quality of their systems.

3.4.3 Text Retrieval Conferences (TREC)

Text Retrieval Conferences (TREC), sponsored byNIST and the Defense Advanced Research ProjectsAgency (DARPA), were started in 1992 to support re-search within the information retrieval community [51].The conferences provide the infrastructure necessaryfor large-scale evaluation of text retrieval methodolo-gies. The TREC conferences encourage research in in-formation retrieval based on large test collections, andhelp to increase communication among industry,academia, and government by creating an open forumfor the exchange of research ideas. As a result, technol-ogy can be transferred more quickly from research labsinto commercial products. The conferences have helpedto demonstrate substantial improvements in retrievalmethodologies on real-world problems and have in-creased the availability of appropriate evaluation tech-niques for use by industry and academia.

TREC is overseen by a program committee, which iscomposed of representatives from government, industry,and academia. NIST provides a test set of documentsand questions. Participants run their own retrieval sys-tems on the data, and return to NIST a list of the re-trieved top-ranked documents. NIST pools the individ-ual results, judges the retrieved documents forcorrectness, and evaluates the results. The TREC cycleends with a workshop that is a forum for participants toshare their experiences.

This evaluation effort has grown in both the numberof participating groups and the number of tasks eachyear. The conference held in 1999 included 66 groupsrepresenting 16 countries. The TREC test collectionsand evaluation software are available to the retrievalresearch community at large, so organizations can eval-uate their own retrieval systems at any time. TREC hassuccessfully met its dual goals of improving the state-of-the-art in information retrieval and of facilitatingtechnology transfer. Retrieval system effectiveness hasapproximately doubled in the 7 years since the firstconference. TREC has also sponsored the first large-scale evaluations of the retrieval of non-English(Spanish and Chinese) documents, retrieval of record-ings of speech, and retrieval across multiple languages.The TREC test collections are large enough so that theyrealistically model operational settings, and most of to-day’s commercial search engines include technologyfirst developed in TREC.

4. Research and Development for ITMeasurement and Testing at NIST

Work on advanced methodologies and techniques isdirected toward improving the quality of testing andmeasurement in areas such as software testing and de-vising new approaches and tools to test technologiessuch as collaborative systems.

4.1 Advanced Encryption Standard

Strong, reliable cryptographic algorithms are essen-tial for protecting sensitive data and electronic transac-tions in the 21st century. To replace the cryptographicalgorithms that have been available to protect informa-tion throughout the 1980s and 1990s, NIST/ITL invitedvoluntary standards organizations, the cryptographiccommunity, and industry to help define requirementsfor, to develop, and to evaluate candidates for the newcryptographic algorithm. This unique participatorypublic process started in 1997 is expected to result in anAdvanced Encryption Standard (AES) that will be com-pleted in 2001. The AES will specify an unclassified,publicly disclosed encryption algorithm capable of pro-tecting sensitive government information well into the21st century.

In 1998, 15 candidate algorithms were accepted foranalysis for suitability for the AES. The assistance of thecryptographic research community was solicited in ana-lyzing the candidates. The review of each algorithmincluded a methodical evaluation of the following char-acteristics: security (including any known attacks orweaknesses); efficiency (both speed and memoryusage); flexibility (implementation on low- and high-end smart cards; support of additional key and blocksizes, including whether the reference code actuallysupported the additional key sizes; suitability for use asa pseudo-random number generator, hashing algorithm,etc.; and whether or not encryption and decryption werethe same procedure); algorithm simplicity; and otherissues raised by the public discussion.

Security was the most important factor in the evalua-tion, and it encompassed features such as: resistance ofthe algorithm to cryptanalysis, soundness of its mathe-matical basis, randomness of the algorithm output, andrelative security as compared to other candidates.

Cost was the second important area of evaluation,encompassing licensing requirements, computationalefficiency (speed) on various platforms, and memoryrequirements. The speed of the algorithms on a widevariety of platforms was considered. The focus wasprimarily on the speed associated with 128 bit keys.Additionally, memory requirements and constraints forsoftware implementations of the candidates were con-sidered.

363


The third area of evaluation included characteristicssuch as flexibility, hardware and software suitability,and algorithm simplicity. Flexibility included the abilityof an algorithm to handle key and block sizes beyond theminimum that must be supported; to be implementedsecurely and efficiently in many different types of envi-ronments; and to be implemented as a stream cipher,hashing algorithm, and to provide additional crypto-graphic services. It must be feasible to implement analgorithm in both hardware and software, and efficientfirmware implementations are advantageous. The rela-tive simplicity of an algorithm’s design was also anevaluation factor.

Tests were devised to measure the characteristics andto analyze the algorithms in these areas of evaluation. Inaddition to its measurements and evaluations, NIST con-sidered all comments, papers, studies, reports and pro-posed modifications. Each candidate was reviewed rela-tive to the announced evaluation criteria and otherpertinent criteria suggested by the public. The results ofthe research were reviewed with the cryptographic com-munity and five algorithms were selected for furtherstudy. After the five candidate algorithms were selected,the results of all of the analyses were made public, andfurther tests were devised to select the algorithm oralgorithms to be considered for the AES. On October 2,2000, NIST proposed the Rijndael data encryption al-gorithm as the AES. NIST will prepare the draft AESFIPS for public review and comment. After the close ofthe public comment period, the standard will be revisedby NIST, as appropriate, in response to those comments.A review, approval, and promulgation process will thenfollow. If all steps of the AES development processproceed as planned, it is anticipated that the standardwill be completed by the summer of 2001 [52].

4.2 Random Number Generation

The need for random and pseudorandom numbersarises in many cryptographic applications. For example,common cryptosystems employ keys that must be gener-ated in a random fashion. Many cryptographic protocolsalso require random or pseudorandom inputs at variouspoints, e.g., for auxiliary quantities used in generatingdigital signatures, or for generating challenges in au-thentication protocols. A study of random number gen-eration was undertaken to investigate whether statisticaltesting could be applied to cryptographic testing. Thestudy was published as NIST Special Publication 800-22A Statistical Test Suite for Random and PseudorandomNumber Generators for Cryptographic Applications[53], and discusses some aspects of selecting and testingrandom and pseudorandom number generators.

Random number generators used in cryptographicapplications may need to meet stronger requirementsthan for use in other applications. In particular, theiroutputs may need to be unpredictable in the absence ofknowledge of the inputs. The criteria for characterizingand selecting appropriate generators were examined aswell as the possible use of statistical testing instead ofcryptanalysis. Although statistical tests were found to beuseful as a first step in determining whether or not agenerator would be suitable for a particular crypto-graphic application, no set of statistical tests could abso-lutely certify a generator as appropriate for use in aparticular application. Therefore, it was concluded thatstatistical testing is not an adequate substitute for crypt-analysis.

4.3 Software Testing Methods

NIST/ITL is investigating methods to produce soft-ware tests, including conformance tests, which arederived from formal specifications. Specification basedtesting methods are not well developed, but theypromise significantly more economical means for test-ing software than currently available, and reduced time-to-market for companies producing software products.Most software testing has focused on structural testing,that is, testing based on execution paths within the codethat implements a specified function. There are so manypossible combinations that testing all of them is notfeasible. Further, structural testing is not possible withmany systems, as there is no access to source code.Deriving tests from specifications is particularly usefulfor bodies developing software standards, e.g., the WorldWide Web Consortium (W3C), the Object ManagementGroup (OMG), the Institute of Electrical and Electron-ics Engineers (IEEE). The goal of these groups is speci-fication development, while the goal of the participatingcompanies is software implementation. Companies im-plementing software to standard specifications will beable to improve software quality by automatically gener-ating software tests and demonstrating conformance tothem. As a result, software products with fewer failuresand closer adherence to standards can come to the mar-ket earlier and at less cost.

Another approach to improving software system qual-ity is through the collection and analysis of error, fault,and failure data. The error, fault, and failure data arebeing used to develop profiles for industry use and forimproving statistical analysis methods. A handbook offaults with detection and prevention methods is beingdeveloped. Software tools are being developed to enablesoftware developers to collect and analyze data. Thesedata collections help to fill a need for reference data on

364


software failures to protect against the release of soft-ware systems with faults and to help assess softwaresystem quality.

Further work to improve the development and mea-surement of software testing is through the applicationof statistical methods to derive quantitative measures ofsoftware correctness or quality. Under investigation anddevelopment are new methods for software testing basedon stochastic processes and statistical measures toprovide qualitative measures for determining the proba-bility that software correctly adheres to its specification.This includes both white-box testing and black-box test-ing to determine if a program conforms to its functionalspecifications. White-box testing assumes that programsource code is available for inspection and metering.Black-box testing allows access to a program onlythrough its defined interface. Functional specificationsdeveloped under conventional techniques usuallyspecify formal syntax with semantic rules written innatural language. Functional specifications developedthrough formal methods require that both syntax andsemantics be specified using mathematically rigoroustechniques. The goal is to integrate statistical techniquesinto software testing to ensure software quality and toprovide quantitative measures of stability, reliability,and conformance to specifications.

4.4 Testability of Interaction-drivenManufacturing Systems (TIMS)

NIST/MEL is characterizing the performance of soft-ware systems (e.g., CORBA) in the context of real-timemanufacturing requirements. This project focuses onmethods for testing multiple software components inter-acting with each other according to specified behaviors.This includes investigating mechanisms to build testabil-ity into software behavior specifications.

An interaction-driven manufacturing system is onecomposed of multiple software components in whichthe interactions between those components are auto-mated. Automation is typically achieved through thedefinition of interfaces, both standard and proprietary,which allow the components to directly interact withoutthe need for human intervention at every step along theway [54]. Such a system is typically implemented in adistributed fashion and its primary function is organizedaround the interactions of complex, loosely coupledcomponents. The components may be large and complexsystems themselves, they may be proprietary commer-cial off-the-shelf systems whose inner workings are notknowable, and they may purport to support standardinterfaces. Since interaction-driven systems had begunto appear frequently in the manufacturing context, aproject to respond to the growing need for relevant test-

ing expertise was begun in 1998. The mission of theTIMS project is to close the gap between testing capa-bilities and the testability of interaction-driven manufac-turing systems. The project is striving to enhance testingcapabilities with improved test methods, by definingdesign-for-testability criteria for specification writers,and by applying tools and techniques to improve thequality of system-level specifications [55].

Thus far the project has investigated the characteris-tics of interaction-driven manufacturing systems, inves-tigated mechanisms for specifying interactions [56],conducted experimental testing of systems, and identi-fied testability issues as a result of those experiments. Asignificant amount of work has been spent developingintegrated testing capabilities for software implementingthe Object Management Group’s Product Data Manage-ment Enablers specification [57] or the related ProductData Management (PDM) Schema being developed forthe international STEP standard [58] [59]. In essencethese emerging standards are duals of each other andshare a common semantic notion of PDM data. Thesestandards would be implemented by PDM software ven-dors thereby making their PDM systems available ascomponents of larger software systems. In developingtest capabilities, the TIMS project uncovered numerousissues with respect to the testability of the specificationsdeveloped by the standards community [60], and thisexperience will help drive anticipated work on designfor testability. In addition, the project has also conductedexperiments in identifying faults that are introduced byimperfect integration of legacy components for controlof a manufacturing shop floor and in testing of agent-based systems in a manufacturing scenario. Project re-sults and the status of on-going efforts are kept up-to-date on the project’s web pages.

5. The Future of IT Metrology

At the start of the 20thcentury, the industrial age waswell underway. Electricity and the electric light, theautomobile, the telephone, the railroad and the telegraphlines connecting the Nation characterized the industrialinfrastructure. The establishment of the National Bureauof Standards in 1901 was a consequence of the realiza-tion that scientific and industrial progress in the newcentury would require better measurements in the labo-ratory and the factory. Now, at the start of the 21stcentury, we are in the midst of an information age.Therefore, it certainly seems most appropriate to con-sider where IT will be in 10 years and how that mightdrive requirements for IT measurements by 2010.

365


5.1 The Forecasting Dilemma

However it is daunting to forecast 10 years ahead forIT because of the ongoing pace of innovation in IT andthe rates of deployment of new IT. Software productlifecycles are now measured in web years (1 webyear = 90 calendar days) so looking ahead 10 calendaryears is actually looking ahead 40 web years, a trickybusiness at best. A decade ago, no one foresaw theexplosion of the deployment of the Internet and thecreation of the World Wide Web. Presently, traffic on theInternet doubles every 100 days. The next 10 years in ITare most likely to continue to be characterized by un-foreseen innovation, explosive growth, and amazingprice/performance improvements.

There is one performance forecast that has held up formore than 30 years, Moore’s Law. The observation,made in 1965 by Intel co-founder Gordon Moore, wasthat each new memory integrated circuit containedroughly twice as much capacity as its predecessor, andeach chip was released within 18 to 24 months of theprevious chip. This trend of computing power risingexponentially with time has held up well. This law hasnow been valid for more than three decades, as shown inFig. 11, and it appears likely to be valid for at leastseveral more device generations.

While computational power has been increasing ex-ponentially, communication speeds have not. This hasbeen due to business factors rather than the lack oftechnological innovation in tethered and untethered

communications. With the recent deregulation of thetelecommunications industry, there should be a quantumleap in available communication speeds over the nextfew years. Competition will drive the increase in speed.New digital wireless systems will lead to the deploymentof new mobility products and services. At the end of2000, about 44.4 million households should be online,compared to about 12.7 million households online in1995, an increase of nearly 250 % over 5 years. Most ofthese households will still be connected to the Internetby relatively slow analog modems. Broadband connec-tions such as digital subscriber lines (DSL) and cablemodems are now providing speeds that are 10 to 100times faster than analog modems. At the end of 2000,about 2 million U.S. homes will be connected to theInternet via cable modems and another 1.1 million resi-dential customers will have DSL. The race to providemore economical broadband connections to the house-hold and business is just now starting. The next 10 yearsshould transform our communications infrastructure aswe find new ways to exploit advances in computationalpower.

Requirements for security and interoperability will betop priorities in the information age of the 21st century.As the Internet continues to grow, it will be vulnerableto many threats that can inflict serious damage andcause serious losses. Almost all government and busi-ness organizations are dependent on information tech-nology to carry out their activities, from the simplest tothe most complex.

Fig. 11. Moore’s Law.

366


Information systems are increasingly open, intercon-nected and essential to our nation’s commerce, trans-portation, financial services, communications, and gov-ernment. These systems can be vulnerable to threatssuch as accidents, human error, and deliberate attacksunless careful steps are taken to protect them. Thethreats persist despite efforts to make our informationsystems more secure and trustworthy and to improveawareness of the serious consequences of poor security.

The use of computers and networks enables U.S.companies to increase productivity and adapt to chang-ing markets. Organizations can provide an integratedcollection of low-cost, reliable services and handletremendous volumes of business and technical transac-tions. The ability to amass, analyze, and control largequantities of data enables costs to be reduced. Whileefficiency and accuracy are improved, new ways of do-ing business and new forms of economic activities willnot occur in the absence of continuing improvements insecurity management and technical methods.

Interoperability will be essential to capture the bene-fits of new devices that incorporate small computers andsensors. Devices such as televison sets, handheld com-puters, digital cameras, and laptop computers make itpossible to have instant access to information anywhereand at anytime. Forecasts predict that use of these infor-mation appliances will outstrip the use of the personalcomputer by 2002. New applications will be possibleonly if the devices can interchange information accu-rately and reliably.

5.2 Software

So with the advent of the 21st century, what will beneeded in the next 10 years in the way of better measure-ments for the information age? What is the equivalent tobetter measurements for electricity for the industrial ageof 1900? Specific forecasting for IT is certainly not easyand probably not needed. With increases in computa-tional power, communication speed, connectivity, andmultimedia content, IT systems will become increas-ingly complex. This IT system complexity will be in thehardware, but more critically in the software. When itcomes to IT measurements in the next 10 years, measur-ing software will be the challenge.

The President’s IT Advisory Committee (PITAC) Re-port, Investing in Our Future [61], listed software as itsfirst concern for research:

“Software—The demand for software has grownfar faster than our ability to produce it. Further-more, the Nation needs software that is far moreusable, reliable, and powerful than what is beingproduced today. We have become dangerously de-

pendent on large software systems whose behavioris not well understood and which often fail in un-predicted ways. Therefore, increases in researchon software should be given a high priority. Spe-cial emphasis should be placed on developing soft-ware for managing large amounts of information,for making computers easier to use, for makingsoftware easier to create and maintain, and forimproving the ways humans interact with comput-ers.”

The report also pointed out the contrasts betweensoftware and hardware. During the past 40 years, com-puting hardware has increased at least eight orders ofmagnitude in performance. But the ability to developsoftware has not kept pace. Software is often fragile,unreliable, and extremely difficult and labor-intensiveto develop, test, and evolve. Some of the difficultiesinherent in software systems pertain to the lack ofmeaningful and standardized behavioral specificationsthat would make it possible to test the properties of thesystems. Too often software flaws are observable onlywhen the program is run, and even then flaws can beinvisible. Often there is no correspondence between aspecification and the software itself. Large softwaresystems are complex, unpredictable in performance,and difficult to describe precisely and to test.

IT measurement and testing for software will play afundamental role in ensuring that complex IT systemsare scalable, usable, reliable, secure, and interoperable.

5.3 Possible Roles for NIST

After 100 years of service, it is very apparent whatessential roles NIST now fulfills in physical and chem-ical metrology. The importance of those physical andchemical metrology roles is well understood by themetrology community and certainly appreciated by ourindustry customers. After about 30 years, the rolesNIST plays in IT metrology are still evolving.Presently, NIST is valued for specific IT conformancetests, interoperability tests, performance tests, and refer-ence data developed by NIST at the request of govern-ment and industry. Over the next 10 years, this role forNIST can be expected to continue. However, advancingthe science and technology for software development,especially better methods for developing software tests,may be the most critical need and daunting challengefacing the IT metrology community today.

After a few decades of research and practice, weappear to have achieved a plateau for the state-of-the-artin software testing, a plateau which will not suffice forthe large and complex systems of IT systems of nextthe 10 years. A recent review of the state of software

367


metrics [18] indicated a lack of rigor compared to phys-ical metrology. The physical metrology principles ofunits, scale, and uncertainty presently have no counter-part in software metrics. At the June 1999 InternationalConference on Testing Computer Software, the keynoteaddress by Elaine J. Weyuker, What Has Software Test-ing Research Done for Me Lately, and What Will It DoFor Me in the Future , reviewed the commercial rele-vance of recent research. This review included a surveyof the four most important journals over five years, from1994 to 1998. The survey revealed a total of 819 papers,of which only 49 primarily discussed software testingissues. Much of the reported research in those 49 paperswas judged by the speaker as not useful to the tester ofcomplex commercial software systems. While softwaretesting as a profession has certainly progressed over thelast decade, the software testing researcher has beenunable to fulfill the present needs of the software testingpractitioner.

What are the consequences of this shortfall in soft-ware testing research and consequently software testingpractice? NIST is presently attempting to quantify theeconomic consequences. In 1999, NIST initiated a studyon the Economic Impacts of an Inadequate Infrastruc-ture for Software Testing . The scope of this study wasexpanded in 2000 and the final report should be avail-able sometime in 2001. Identifying economic impactsshould help to identify and to focus research priorities.As a national metrology institution with expertise insoftware measurement and testing practice and research,NIST has an excellent window of opportunity over thenext ten years to continue to pursue research and devel-opment that could significantly advance the state of soft-ware testing.

This also may be the opportune time to develop com-prehensive reports on current best practices in IT con-formance, interoperability, and performance testing.Widely accepted and recognized best practices wouldassist current IT testing developers and practitioners.Widely accepted and recognized best practices for today,and their limitations, would also help to identify andfocus on the research priorities needed to get to the bestpractices that software testing practitioners will need in2010. Again, based upon NIST’s IT measuring and test-ing expertise and role as a national metrology institution,NIST is well positioned to serve as a catalyst and leaderin working with industry, government, and academia onthese issues.

6. Conclusions

Information technology (IT) has transformed the waythat people work, learn, and communicate, and con-

tributed significantly to the U.S. economic growth in the20th century. IT has been the engine for improved pro-ductivity, technological innovation, and new opportuni-ties. Today IT is the foundation for many sectors of theAmerican economy, including the transportation, elec-trical power, banking, water supply, and telecommuni-cations systems that support the basic infrastructures inthe United States.

As the use of IT continues to increase, advances in ITmetrology will be critical to support the development ofsystems that are scalable, usable, secure, and interopera-ble. Users of cell phones, hand-held personal digitalassistants, laptop computers and other wireless deviceswill need devices that are interoperable and capable ofsecurely and reliably exchanging data. People are usingmobile cell phones to access the Internet, shop, paybills, trade stocks, and a host of other applications. Theuse of mobile cell phones for these electronic commerceapplications is expected to rise from about 1 millionusers in 2000 to about 45 million users in 2004.

People rely more and more on their computers andtheir wireless devices that provide computing functions.Additionally, information security requirements con-tinue to escalate. New needs are appearing every day. Inthe 21st century, almost everyone will be dependent onIT systems to perform essential functions. As the tech-nology changes to provide new capabilities and features,new methods for measuring and testing IT will be essen-tial to continued growth and development.

7. References

[1] American National Standard Dictionary of Information Tech-nology (ANSDIT), ANSI X3.172-1996, American NationalStandards Institute, New York.

[2] International Vocabulary of Basic and General Terms in Metrol-ogy (the VIM), ISO, Geneva, Switzerland (1993) 59 pp.

[3] ISO/IEC Guide 2: 1996, Standardization and related activities—General vocabulary, ISO/IEC, Geneva, Switzerland, 81 pp.

[4] E. Passaglia, with K.A. Beal, A Unique Institution, The NationalBureau of Standards 1950-1969, U.S. Government Printing Of-fice, Washington, DC 20402 (1999).

[5] G. N. Baird, The DoD COBOL Compiler Validation System,Fall Joint Computer Conference (1972) pp. 819-827.

[6] C. Montanez-Rivera and L. A. Johnson, Conformance TestSpecifications for COBOL Intrinsic Function Module, NIST SP500-203, July 1992.

[7] NIST Software Standards Validation Group, 1978 Fortran Com-piler Validation System User’s Guide, Version 2.1, August1993.

[8] J. V. Cugini, J. S. Bowden, and M. W. Skall, NBS MinimalBASIC Test Program- Version 2, User Manual, (Volume 1,Documentation and Volume 2, Source Listings and Sample Out-put), NBS SP 500-70/1 and NBS SP 500-70/2, November 1980.

[9] D. Flater, L. Gallagher, S. Hurwitz, and J. Sullivan, User’sGuide for the SQL Test Suite Version 6.0, NISTIR 5998, De-cember 1996.

368


[10] G. N. Baird and L. A. Johnson, Compiler Validation—An As-sessment, National Computer Conference (1981) pp. 361-366.

[11] R. Rosenthal and N. B. Meisner, (eds), Proceeding of the LocalArea Communications Network Symposium—Update: LocalArea Networks, Boston, May 1979.

[12] R. Rosenthal, Local Area Networks: Baseband Carrier SenseMultiple Access with Collision Detection Access Method andPhysical Layer Specification and Link Layer Protocol, NBS Fed-eral Information Processing Standard FIPS 107, November 1984.

[13] A. Mink, R. Carpenter, G. Nacht, and J. Roberts, MultiprocessorPerformance-Measurement Instrumentation, IEEE Computer(1990) pp. 63-75.

[14] A. Mink, W. Salamon, J. Hollingsworth, and R. Arunachalam,Performance Measurement using Low Perturbation and HighPrecision Hardware Assists, Proc. of the Proceedings of Rtss’98,1 Dec 1998, IEEE, New York, pp. 379-388.

[15] C. Martin, A. Mink, W. Salamon, M. Indovina, and M. Couson,Performance Measurement of Remote ATM Cluster, ATM99Developments Conf, Mar-Apr 1999, Rennes, France.

[16] A. Mink, R. Carpenter, and M. Courson, Time SynchronizedMeasurements in Cluster Computing Systems, 13th Intl. Confer-ence on Parallel and Distributed Computing, PDSC2000, Aug1-20, 2000, Las Vegas, NV.

[17] M. D. Hogan, et al., Metrology for Information Technology,NISTIR 6025 (1997) 28 pp. (URL: http://www.nist.gov/itl/lab/nistirs/ir6025.htm).

[18] M. Gray, Applicability of Metrology to Information Technology,J. Res. Natl. Inst. Stand. Technol. 104 (6), 567-578 (1999).

[19] ISO/IEC TR13233: 1995 Information technology—Interpreta-tion of accreditation requirements in ISO/IEC Guide 25—Ac-creditation of Information Technology and Telecommunicationstesting laboratories for software and protocol testing services,ISO/IEC, Geneva, Switzerland.

[20] ISO 10303-1:1994. Industrial automation systems and integra-tion—Product data representation and exchange—Part 1:Overview and fundamental principles; ISO, Geneva, Switzer-land. December 1994.

[21] S. Kemmerer, ed., STEP The Grand Experience, NIST SP 939,July 1999.

[22] NIST, Federal Information Processing Standard 46, Data En-cryption Standard, 1999.

[23] NIST, Federal Information Processing Standard 186, Digital Sig-nature Standard, 2000.

[24] NIST, Federal Information Processing Standard 180, SecureHash Algorithm, 1995.

[25] NIST, Federal Information Processing Standard 140, SecurityRequirements for Cryptographic Modules, 1994.

[26] ISO/IEC 15408-3, Information Technology—Security Tech-niques—Evaluation criteria for IT Security, Parts 1-3, ISO/IEC,Geneva, Switzerland, 1999.

[27] Document Object Model (DOM) level 2 Core Specification Ver-sion 1.0 W3C Proposed Recommendation, (URL: http://www.w3.org/TR/2000/PR-DOM-Level-2-Core-20000927/)September 2000.

[28] Extensible Markup Language (XML) 1.0 (Second Edition),W3C Recommendation, (URL: http://www.w3.org/TR/2000/RED-xml-20001006) October 2000.

[29] R. J. Linn, Conformance Testing for OSI Protocols, ComputerNetworks and ISDN Systems, Vol. 18, 1990.

[30] L. Rosenthal, M. Skall, M. Brady, M. Kass, and C. Montanez-Rivera, Web-Based Conformance Testing For VRML, Standard-View, Vol. 5, No. 3, September 1997.

[31] NIST Cerberus, An IPsec Reference Implementation for Linux,(URL: http://www.antd.nist.gov/ipsec/).

[32] S. Kent and R. Atkinson, Security Architecture for the InternetProtocol, RFC 2401, ISOC, November 1998.

[33] S. Kent and R. Atkinson, IP Encapsulating Security Payload,RFC 2406, ISOC, November 1998.

[34] S. Kent and R. Atkinson, IP Authentication Header, RFC 2402,ISOC, November 1998.

[35] NIST PlutoPlus: An IKE Reference Implementation for Linux,(URL: http://is2.antd.nist.gov/plutoplus/).

[36] D. Harkins and D. Carrel, The Internet Key Exchange (IKE),RFC 2409, November 1998.

[37] S. P. Frechette, STEP Implementation: Solid Model ExchangeResults in the AutoSTEP Project, J. Agility Global Competition1 (3), 1-12 (1997).

[38] ISO 10303-203:1994. Industrial automation systems and integra-tion—Product data representation and exchange—Part 203: Ap-plication protocol: Configuration controlled 3D designs of me-chanical parts and assemblies ISO, Geneva, Switzerland.December 1994.

[39] IEEE Standard Glossary of Software Engineering Terminology,IEEE 610.12-1990, IEEE, New York.

[40] Project Brief: Advanced Process Control Framework Initia-tive, (URL: http://jazz.nist.gov/atpcf/prjbriefs/prjbrief.cfm?Pro-jectNumber=95-12-0027), NIST, 1996.

[41] H. M. Bloom, N. Christopher, A Framework for Distributed andVirtual Discrete Part Manufacturing, Proceedings of the CALSEXPO ’96, Long Beach, CA (1996).

[42] CORBA Basics, (URL: http://sisyphus.omg.org/gettingstarted/corbafaq.htm), Object Management Group.

[43] D. Flater, Manufacturer’s CORBA Interface Testing Toolkit:Overview, J. Res. Natl. Inst. Stand. Technol. 104 (2), 193-200(1999).

[44] MCITT Web Site, (URL: http://www.mel.nist.gov/msidstaff/flater/mcitt/), NIST, 1999.

[45] American National Standard for Information Systems, Data For-mat for the Interchange of Fingerprint, Facial, and Scar Markand Tattoo (SMT) Information, NIST SP 500-245, September2000, 81 pp.

[46] IAFIS-IC-0110 (V�) WSQ Gray-scale Fingerprint Image Com-pression Specification; Available from Criminal Justice Informa-tion Services Division, Federal Bureau of Investigation, 935 PAAve, NW, Washington, DC 20535.

[47] NIST Special Database 9—NIST 8-bit Gray Scale Images ofMated Fingerprint Card Pairs available from NIST SRD pro-gram, (URL: http://www.nist.gov/srd/nistsd9.htm).

[48] C. Diaz and T. H. Hopp, Evaluation of Software for CoordinateMeasuring Systems, Proceedings of the SM 1995 Clinic & Inter-face ’95 Symposium, June 1995.

[49] C. M. Shakarji, Least-Squares Fitting Algorithms of the NISTAlgorithm Testing System, J. Res. Natl. Inst. Stand. Technol. 103(6), 633-641 (1998).

[50] A. Martin, M. Przybocki, The NIST 1999 Speaker RecognitionEvaluation—An Overview, Digital Signal Processing. Vol. 10,2000, pp. 1-18.

[51] E. M. Voorhees, D. K. Harman, The Eighth Text Retrieval Con-ference (TREC-8), NIST SP 500-246, September 2000.

[52] J. Nechvatal et al., Report on the Development of the AdvancedEncryption Standard (AES), (URL: http://csrc.nist.gov/encryp-tion/aes/round2/r2report.pdf) National Institute of Standards andTechnology, October 2000, 116 pp.

[53] A. Rukhin et al., A Statistical Test Suite for Random and Pseu-dorandom Number Generators for Cryptographic Applications,NIST SP 800-22, November 2000, 162 pp.

[54] K. C. Morris, D. Flater, Standards-based Software Testing in aNet-Centric World, in Proceedings of the Ninth International

369


Workshop on Software Technology and Engineering Practice(STEP ’99), Pittsburgh, Pennsylvania, IEEE Computer Society,September 1999.

[55] D. Flater, TIMS Project Description, (URL: http://www.mel.nist.gov/msid/tims/), April 2000.

[56] D. Flater, Specification of Interactions in Integrated Manufactur-ing Systems, NISTIR 6484, March 2000.

[57] Object Management Group PDM Enablers, (URL: http://www.omg.org/cgi-bin/doc?mfg/98-02-02.pdf), February 1998.

[58] STEP PDM Schema Version 1.1, (URL: http://www.pdm-if.org/pdm_schema/), October 1999.

[59] D. Flater and K. C. Morris, Testability of Product Data Manage-ment Interfaces, submitted to the 34th Hawaii International Con-ference on System Sciences, January 2001.

[60] K. C. Morris and D. Flater, Design of a Flexible, IntegratedTesting System for STEP and OMG Standards, submitted to theJournal of Computer Standards and Interfaces, February 2000.

[61] President’s IT Advisory Committee (PITAC) Report, Investing inOur Future, February 1999.

About the authors: M. D. Hogan, L. J. Carnahan, R. J.Carpenter, M. M. Gray, L. A. Johnson, R. M. McCabe,D. Montgomery, S. M. Radack, and R. Rosenthal areengineers and computer scientists in the InformationTechnology Laboratory. D. W. Flater, J. E. Fowler, S.P. Frechette, and C. M. Shakarji are engineers, com-puter scientists, and mathematicians in the Manufactur-ing Engineering Laboratory. The National Institute ofStandards and Technology is an agency of the Technol-ogy Administration, U. S. Department of Commerce.

370

AT $31.00 A YEAR, CAN YOU AFFORD NOT TOKNOW WHAT’S GOING ON AT THE NATION’SMEASUREMENT SCIENCE LABORATORY?

YES NOMay we make your name/address available to other mailers? � �

The Journal of Research Brings You

Up-to-Date Scientific Articles and

Information on:

• Measurement Science and Technology

• Calibration Services

• Standard Reference Materials

• Cooperative Research Opportunities and Grants

• Conference Reports

AND MUCH MORE!

It’s All At Your Fingertips In the Journal of Research of theNATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

SUBSCRIBE TODAY !– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –

Superintendent of Documents Subscription Order FormJournal ofResearch of theNationalInstitute ofStandards andTechnology

�YES, send me subscriptionsto the JOURNAL OF RESEARCH OF THENATIONAL INSTITUTE OF STANDARDSAND TECHNOLOGY at $31 per subscrip-tion (6 times a year) so I can stay up to dateon the latest developments in measurementscience and technology.

2. The total cost of my order is $ . All prices includedomestic postage and handling. International customers pleaseadd 25 percent.

3. Please Choose Method of Payment:

� Check payable to the Superintendent of Documents

� GPO Deposit Account ��–�� VISA � MasterCard � Discover/NOVUS

��

Thank you for your order!(Credit Card Expiration Date)

(Purchase Order No.)

(Signature) (10-99)

1. Please Type or Print

(Company or personal name)

(Additional address/attention line)

(Street address)

(City, State, ZIP Code)( )

(Daytime phone including area code)

Order Processing Code

6596 ��

4. MAIL TO: New Orders, Superintendent of Documents, P.O. Box 371954, Pittsburgh, PA 15250–7954.

The International System of Units (SI)The Modern Metric System

Uncertain about the InternationalSystem of Units (universally abbreviatedSI), the modern metric system usedthroughout the world? Do you need toknow the proper way to express the resultsof measurements and the values of quanti-ties in units of the SI? Do you need to knowthe NIST policy on the use of the SI? Thenyou need the 1995 edition of the NationalInstitute of Standards and TechnologySpecial Publication 811, Guide for the Useof the International System of Units (SI) .

The 1995 edition of the National Institute of Standards and Technology Special Publication 811,Guide for the Use of the International System of Units (SI) , by Barry N. Taylor, is now available.

The 1995 edition of SP 811 corrects a number of misprints in the 1991 edition, incorporates asignificant amount of additional material intended to answer frequently asked questions concerningthe SI and SI usage, and updates the bibliography. The added material includes a check list forreviewing the consistency of written documents with the SI. Some changes in format have alsobeen made in an attempt to improve the ease of use of SP 811.

The topics covered by SP 811 include:• NIST policy on the use of the SI in NIST publications.• Classes of SI units, those SI derived units that have special names and symbols, and the SI

prefixes that are used to form decimal multiples and submultiples of SI units.• Those units outside the SI that may be used with the SI and those that may not.• Rules and style conventions for printing and using quantity symbols, unit symbols, and prefix

symbols, and for spelling unit names.• Rules and style conventions for expressing the results of measurements and the values of quan-

tities.• Definitions of the SI base units.• Conversion factors for converting values of quantities expressed in units that are mainly unac-

ceptable for use with the SI to values expressed mainly in units of the SI.• Rounding numbers and rounding converted numerical values of quantities.

Single copies of the 84-page SP 811 may be obtained from the NIST Calibration Program,100 Bureau Dr., Building 820, Room 236, Stop 2330, Gaithersburg, MD 20899-2330, telephone:301-975-2002, fax: 301-869-3548.

Special Publication 811

1995 Edition

Guide for the Use of the International

System of Units (SI)

Barry N. Taylor

United States Department of Commerce

Technology Administration

National Institute of Standards and Technology

NISTTechnical PublicationsPeriodical

Journal of Research of the National Institute of Standards and Technology—Reports NIST researchand development in those disciplines of the physical and engineering sciences in which the Institute isactive. These include physics, chemistry, engineering, mathematics, and computer sciences. Papers cover abroad range of subjects, with major emphasis on measurement methodology and the basic technologyunderlying standardization. Also included from time to time are survey articles on topics closely related tothe Institute’s technical and scientific programs. Issued six times a year.

Nonperiodicals

Monographs—Major contributions to the technical literature on various subjects related to theInstitute’s scientific and technical activities.Handbooks—Recommended codes of engineering and industrial practice (including safety codes) devel-oped in cooperation with interested industries, professional organizations, and regulatory bodies.Special Publications—Include proceedings of conferences sponsored by NIST, NIST annual reports, andother special publications appropriate to this grouping such as wall charts, pocket cards, and bibliographies.

National Standard Reference Data Series—Provides quantitative data on the physical and chemicalproperties of materials, compiled from the world’s literature and critically evaluated. Developed under aworldwide program coordinated by NIST under the authority of the National Standard Data Act (PublicLaw 90-396). NOTE: The Journal of Physical and Chemical Reference Data (JPCRD) is publishedbimonthly for NIST by the American Institute of Physics (AIP). Subscription orders and renewals areavailable from AIP, P.O. Box 503284, St. Louis, MO 63150-3284.Building Science Series—Disseminates technical information developed at the Institute on buildingmaterials, components, systems, and whole structures. The series presents research results, test methods, andperformance criteria related to the structural and environmental functions and the durability and safetycharacteristics of building elements and systems.Technical Notes—Studies or reports which are complete in themselves but restrictive in their treatment ofa subject. Analogous to monographs but not so comprehensive in scope or definitive in treatment of thesubject area. Often serve as a vehicle for final reports of work performed at NIST under the sponsorship ofother government agencies.Voluntary Product Standards—Developed under procedures published by the Department of Commercein Part 10, Title 15, of the Code of Federal Regulations. The standards establish nationally recognizedrequirements for products, and provide all concerned interests with a basis for common understanding ofthe characteristics of the products. NIST administers this program in support of the efforts of private-sectorstandardizing organizations.

Order the following NIST publications—FIPS and NISTIRs—from the National Technical InformationService, Springfield, VA 22161.Federal Information Processing Standards Publications (FIPS PUB)—Publications in this seriescollectively constitute the Federal Information Processing Standards Register. The Register serves as theofficial source of information in the Federal Government regarding standards issued by NIST pursuant tothe Federal Property and Administrative Services Act of 1949 as amended, Public Law 89-306 (79 Stat.1127), and as implemented by Executive Order 11717 (38 FR 12315, dated May 11, 1973) and Part 6 ofTitle 15 CFR (Code of Federal Regulations).NIST Interagency or Internal Reports (NISTIR)—The series includes interim or final reports on workperformed by NIST for outside sponsors (both government and nongovernment). In general, initialdistribution is handled by the sponsor; public distribution is handled by sales through the National TechnicalInformation Service, Springfield, VA 22161, in hard copy, electronic media, or microfiche form. NISTIR’smay also report results of NIST projects of transitory or limited interest, including those that will bepublished subsequently in more comprehensive form.

U.S. Department of CommerceNational Institute of Standards & TechnologyGaithersburg, MD 20899-0001

Official BusinessPenalty for Private Use $300

SPECIAL STANDARD MAILPOSTAGE & FEES PAID

NISTPERMIT NO. G195