techniques for methods of collection, database management

Agricultural and Forest Meteorology 103 (2000) 83–97

Techniques for methods of collection, database management anddistribution of agrometeorological data

P.C. Doraiswamya,∗, P.A. Pasterisb, K.C. Jonesb, R.P. Mothac, P. Nejedlikd

a United States Department of Agriculture, Agricultural Research Service, Beltsville, MD 2075, USAb United States Department of Agriculture, Natural Resources Conservation Service, National Water and Climate Center, Portland, OR, USA

c United States Department of Agriculture, World Agricultural Outlook Board, Joint Agricultural Weather Facility, Washington, DC, USAd Slovak Hydrometeorological Institute, Kosice, Slovak Republic

Received 8 October 1998; received in revised form 2 September 1999; accepted 7 September 1999

Abstract

The major concerns for the availability of climate and agrometeorological data as we move into the 21st century continue tobe in areas of data collection and data base management. New technologies have advanced our ability to address these issuesbut the solutions may require the commitment of resources that may yet be outside the reach of developing countries. Thispaper briefly reviews new and existing technologies in the areas of data collection methods with some emphasis on remotesensing methods. An example for a conceptual data base management system adapted by the US Department of Agricultureis presented as a framework for the acquisition, maintenance and distribution of climate and agrometeorological data for the21st century. The access of data to the national and international community can be resolved with the standardization of database management and electronic accessibility. Databases in the US have been upgraded for access through the World WideWeb, and include data archived from other countries that have co-operative programs with US institutions. The operationalneeds of national agricultural producers, researchers, and agricultural weather and crop forecasters are the focus of this paperalthough it also serves some of the needs of the international research community. © 2000 Published by Elsevier Science B.V.All rights reserved.

Keywords:Agrometeorological database; Climate data collection; Climate data management; UCAN; WMO

1. Introduction

The availability of a proper meteorological andagrometeorological data base is a major prerequisitefor studying and managing the processes of agricul-tural and forest production. In an examination of the‘Agrometeorological Needs and Perspectives in the21st Century’, the building of a database of mete-

∗ Corresponding author. Tel.:+1-909-680-1552;fax: +1-909-680-1510.E-mail address:[email protected] (P.C. Doraiswamy)

orological, phenological, soil and agronomic infor-mation is inevitably a major priority. The acquisitionof pertinent climate and agrometeorological data,processing, quality control, archive, timely accessand database management are important componentsthat will make the information valuable and used inagricultural research and operational programs.

Agrometeorological observations are focusedmostly to the important food-producing areas aroundthe world. Because of wide territorial scope, the basicnational databases that include meteorological, cli-matological and crop phenological data are typically

0168-1923/00/$ – see front matter © 2000 Published by Elsevier Science B.V. All rights reserved.PII: S0168-1923(00)00120-9

84 P.C. Doraiswamy et al. / Agricultural and Forest Meteorology 103 (2000) 83–97

kept and maintained by the National MeteorologicalServices. The building of these databases usually aredependant on the availability of technical personnel,and the hardware and software resources at the locallevel. The recent development in personal computertechnology, database software, and the adoption ofadvanced interpretative systems like Geographic In-formation System (GIS), have brought large datavolumes closer to the end users. As the use of agrom-eteorological data depends on the defined purposes,the accuracy and the management of the data can bedifferent and should be defined accordingly.

Great strides in the automation age have offerednew ways of making data products available to theuser community, particularly in industrialized coun-tries. The introduction of electronic transfer of datafiles via Internet using File Protocol Transfer (FTP)and the World Wide Web (WWW) has advancedthis information transfer process to a new level. TheWWW allows users to access text, images and evensound files that are linked together electronically. Theattributes of WWW include the flexible to handle awide range of data presentation methods and the pop-ularity to reach a large audience. The developmentand access to this type of electronic databases maycurrently be limited to industrialized countries.

The WWW interface handles the communicationbetween a remote server, where the data files reside,and the user’s display capability. This interface iscontrolled by a series of scripts and programs, whichexecute commands to read, process, and display dataproducts. The data can be stored in a self-describingfile format. The scripts and programs to generatemenus for the WWW interface can read the informa-tion that describes the content of the data file (i.e.,metadata). This modernization of data observing, pro-cessing, archiving, and display have made availableworldwide large quantities of environmental data.The increasing challenge is to manage these growingresources of data so that users can efficiently extractneeded information for their specific applications.

The global climate community and specifically theagricultural climate community has great interest inincorporating these new information technologies intoa systematic design for agrometeorological data man-agement. The goal is to ensure that the climate andagrometeorological data bases needed by the agricul-tural sector are collected, quality controlled, archived,

and disseminated in a timely manner. Any such de-sign needs to facilitate the collection, organization,analysis, and dissemination of data from a nationalagricultural weather and climate station data networkfor global, national, regional, state, and local pro-gram management and policy decisions and for dis-semination to the public for use in the preparation ofvalue-added services.

A recent attempt has been made in the US topromote the creation of such an agricultural climatedatabase system. The National Agricultural WeatherInformation System (NAWIS) was designed to bea partnership between National Weather Service(NWS), US Department of Agriculture (USDA), andother public and private sector users. The goal is tofacilitate the collection, organization, analysis, anddissemination of data from a national agriculturalweather and climate station data network. The princi-pal users are Federal, State, and local program man-agement, policy decision-makers, and the public foruse in the preparation of value-added services. Thegoals of the program are to provide (a). A nationallycoordinated agricultural weather information system,based on the participation of Federal and State agen-cies, colleges and universities, and the private sectorand aimed at meeting domestic agricultural weatherand climate information needs. (b). Data for researchand education, aimed at improving the quality andquantity of weather and climate information availableto agricultural producers, including research on dataanalysis of weather and climate, particularly as theyaffect ecosystems.

2. Conceptual database

The developing of a climate information deliverysystem requires a basic foundation with the ability toexpand and adapt with future needs. The cornerstonesand building blocks of a conceptual system are pre-sented. The concept, shown in Fig. 1, relies on a seriesof sequential and parallel operations, all leading to afree flowing system of climate information. The heartof the design is the database from which all informa-tion is obtained, while the body is the functions andprocesses, working and interacting to address the op-erational needs of agricultural producers, researchers,and agrometeorological forecasting. In this concept,

P.C. Doraiswamy et al. / Agricultural and Forest Meteorology 103 (2000) 83–97 85

Fig. 1. Climate information system — a conceptual database design.

the initial database is the essential building block for adiverse set of functions that depend on the user needs.There are a wide variety of users, some requiring onlybasic general information for their decisions, othersdemand more detailed and specific data products forfurther analysis.

The various elements in the conceptual designwill be investigated in some depth, drawing on theexperience and opportunities that now exist in thetechnologies of personal computers and the WWW.The intent of this paper is not to be a ‘how to’ cook-book, but rather a review of basic concepts of design.The concept of a database system for the 21st centuryis derived from USDA’s experiences in the set upof a Unified Climate Access Network (UCAN). TheUCAN consists of a consortium of State and Federalorganizations, which record, manage and archive me-teorological data (Pasteris et al., 1997). The primarygoal is to provide access to nationally, regionallyand locally-managed climate data to a clientele ofresearchers, decision makers, and other users world-wide via the Internet. Each participating organizationmaintains a computer that serves its climate data tothe UCAN. This ‘node’ functions in concert withother UCAN nodes to perform distributed searches,retrievals and analyses of requested sets of meteoro-logical observations. The various UCAN data serversinvolved in the process of searching for, assembling,and distributing climate data sets to the public are

coordinated through an efficient catalog of all me-teorological observing stations from which data arearchived and disseminated by the UCAN.

3. Data collection

3.1. Operational data needs of interest to agriculture

Agricultural weather and climate data systems arenecessary to expedite generation of products, analy-ses and forecasts that affect agricultural cropping andmanagement decisions, irrigation scheduling, com-modity trading and markets, fire weather managementand other preparedness for calamities, and ecosystemconservation and management. Basic data collectionactivities should ensure that information on weatherand climate, and their impact on crops, livestock, wa-ter and soil resources, and forest management can beprocessed, analyzed and distributed in the most effi-cient and timely manner. WMO publications 100 and134 (WMO, 1983, 1986) should serve as guides to thespatial and temporal array of parameters necessaryfor successful operations.

3.2. Methods of data observation

Agrometeorology as an interdisciplinary branchtakes data and provides the information of relatively


wide scope. Agrometeorological station networks aredesigned to observe the data of meteorological andbiological phenomena together with supplementarydata as disasters and crop damages occur. Meteoro-logical data can be considered as typically physicalelements that can be measured with relatively highaccuracy while other types of observation are inmany cases much more subjective and vary withthe observer. In collecting, managing and analyzingthe data for agrometeorological purposes, the sourceof data and the methods of observation define theircharacter and possible management. The number ofparameters observed differ with the method of datacollection.

The method of observation can be categorized intotwo major classes, manually observed and automateddata collection stations. A third source for agrome-teorological data that is gaining recognition for itscomplementary nature to the traditional methods issatellite remote sensing technology. An introductionto this topic is presented only as a source of dataand no attempts will be made to prescribe a databasesystem for remote sensing data.

3.2.1. Manually observed dataManually observed data mainly form the basis of

agrometeorological information not only in the plan-ning but also in the operative use. While the systemof manually observed meteorological data collectionis well established and standardized, the definitionsof agrometeorological data are not so specific andthe collection and control of these data are much lessunique. The typical observations are air temperature,relative humidity, sunshine duration, wind run, pre-cipitation and pan evaporation. Though the method-ologies of agronomic data observation are quite welldefined from the general point of view, the prac-tice varies greatly even on the regional level basedon variety and species. Observations are generallydates for planting, a few key stages of crop develop-ment and harvest. Plant height and crop water stressconditions may be observed for selective crops andsituations. Following are some basic considerationsfor agronomic measurements:• Observations are from a plant community and not

single plants.• There are much more descriptive elements in the

observation.

• The frequency of the observation is different andcan vary during the growing season.

• In Europe (WMO, 1988), up to 40% of collectedphenological data and used it operationally.

• 30% of countries collecting phenological data donot include it data in their database.The collection of manually observed data is highly

time consuming with transmission done by a varietyof methods and frequencies. But, as stated in WMO(1997), in many countries the main mode of transmit-ting data from observing sites to the national climatedata centers is by radiotelephones supplemented bypostal returns at designed time intervals. It is a slowsystem, but the indications are that it Will remain inplace for a long time to come. These data are thentypically entered into a computer data file. The im-plementation of the climate computing (CLICOM)(http://www.gsf.de/UNEP/wcdmp.html) project byWMO has afforded many countries a much higherlevel of data processing and data control, however, theCLICOM system has not been accepted as a univer-sal standard and many countries have adopted otherintegrated systems of data processing and archiving.The type and data collection method then, plays thedecisive role in data processing and management.

3.2.2. Automatic weather station data (AWS)AWS is becoming the ideal method for collection

of climate and agrometeorological data, and is a ne-cessity for many real-time operational programs. Thedata are now automatically collected from electronicmedia and automatically transmitted over the radio ortelephone systems. The initial cost is high to establishthe network but in the long run it is a more efficientmethod for simplified data processing and archive.This technology continues to evolve and the declinein costs makes it within reach especially in developedcountries. The maintenance of these stations requiresmore technical skills and can be centralized to mini-mize costs.

The AWS can collect data at a desired range of fre-quencies from minutes to 24 h averages, depending onthe need and data storage capacity. The data loggercan be programmed to provide daily summaries in ad-dition to the regular acquisition. The number of pa-rameters collected can be numerous, but the standardstation may collect air temperature, relative humid-ity, wind speed, barometric pressure, solar radiation,


rainfall, soil temperature (multiple depths), pan evap-oration, leaf wetness and soil moisture over a limitedrange. Manual and AWS data need to be archived atdesignated centers if possible. In the US, for example,data are archived at six regional data centers and atoffices of the state climatologists.

3.2.3. Remotely sensed dataRemotely sensed data and AWS systems provide

in many ways an enhanced and very feasible alterna-tive to manual observation for a very short time spanbetween data collection and transmission. In certaincountries where only few stations are in operation as innorthern Turkmenistan (Seitnazarov, 1999), remotelysensed data can improve information on crop condi-tions for an early warning system. We have over twodecades of experience in the use of data acquired fromgeostationary and polar orbiting satellites than anyother forms of remotely collected data. Geostationaryorbiting environmental satellites (GOES) can measurefrequent surface temperature, solar radiation (cloudcover) and relative rainfall. Access to the other geosta-tionary satellites spanning the globe should be possi-ble through international cooperation. Unlike the po-lar orbiting satellites, the GOES satellites can providecontinuous monitoring of the Earth’s atmosphere andsurface over a large region of the western hemispherein a geosynchronous orbit. These satellites monitor po-tential severe weather conditions, such as tornadoes,flash floods, hail storms, and hurricanes. The GOESImager instrument consisting of five channels rang-ing from the visible to the infrared wavelengths, hasa ground resolutions of 1 and 4 km at nadir.

The GOES sounder carries 18 thermal infraredchannels. Each of the GOES satellites scans pre-determined areas of the earth from the Mid Pacificregion to the eastern Atlantic region. During routinemode, observations are taken four times every hour,but when severe weather threatens, the GOES Imageris capable of 1 min interval scanning over a smallerarea. A variety of products from the sounder andimager are created operationally. These products arearchived at the National Climatic Data Center (NCDC)(http://www.ncdc.noaa.gov/ol/satellite/satellite prod-ucts.html)

Polar orbiting satellites such as NOAA’s advancedvery high resolution radiometer (AVHRR) sensorcan collect daily surface temperature and vegetative

growth and condition (relative crop water deficit). Theground resolution of observation is 1.1 km for NOAAAVHRR data covering most parts of the globe and iscomplete. Kogan (1994, 1995) developed the GlobalVegetation Index (GVI) product, which provides in-formation for the area between 75◦N and 55◦S andproduces weekly assessment of (16 km resolution)relative changes in seasonal land cover by compar-ing the data with normal dynamics of vegetation.The magnitude of the GVI is calibrated to delineatedrought areas when a major drought is in progress.

Doraiswamy et al. (1997) demonstrated the use ofthe Normalized Difference Vegetation Index (NDVI)to monitor biweekly changes in crop growth and de-velopment at 1 km spatial resolution. In an operationalmode, the spatial and temporal information derivedfrom NOAA AVHRR data can provide an early warn-ing to potential drought conditions. The integrationof remote sensing data with ground-based climatedata has been demonstrated for monitoring evapotran-spiration and vegetation growth at watershed scales(Kustas et al., 1994; Maas and Doraiswamy, 1996).Doraiswamy et al. (1998) integrated remote sensingdata with crop simulation models to monitor cropcondition and yields at regional scales in the US.Moussa (1999) discusses an approach where NDVIis integrated with yield monitoring and forecastingsystems in Niger.

There are other numerous remote sensing satellitesthat are in operation, but only systems that are directlypertinent and have been in use for agrometeorologi-cal observations are presented. These remote sensingmethods of data collection are based on measurementof reflected (short-wave) and emitted (thermal) elec-tromagnetic radiation from the surface of the earth.The data processing can be rigorous, however, mostprocessing software are standardized for retrieval ofbasic information that is useful for agrometeorolog-ical applications. Proper calibration and atmosphericcorrection of imagery data is still a major concern indata processing especially for application in retrievalof crop growth parameters.

Some developing countries are beginning to set uptheir own basic remote sensing centers that shouldbe able to handle the required level of processing.Regional centers are set up to service surroundingcountries such as in the African continent. The Foodand Agriculture Organization (FAO) has been very


instrumental in promoting and establishing these cen-ters in developing countries. Programs such as theFamine Early Warning System (FEWS) sponsoredby the US State Department rely on remotely senseddata as a surrogate for agrometeorological data whenno ground data is available.

Newer technologies on the horizon such as thoseon NASA’s Tropical Rainfall Measuring Mission(TRMM) satellite is being investigated for their ac-curacy in estimating the magnitude and spatial dis-tribution of rainfall. The TRMM is a joint spaceproject with Japan. TRMM is designed to mea-sure rainfall in the tropical belt. TRMM’s uniquecombination of sensor wavelengths, coverage, andresolving capabilities together with a low-altitude,non-sun-synchronous orbit provides a sampling capa-bility that should yield monthly precipitation amountsto a reasonable accuracy over a 500 km×500 km grid.The NASA Goddard Distributive Active ArchiveCenter (DAAC) has been archiving TRMM datasince the November 1997 launch has been releasedto the public. The data are useful for forecastmodel research, disaster mitigation, climatologicalstudies, agricultural predictions, and many otherapplications.

The need for supplementing the measurement ofsuch a highly spatial variable parameter as rainfallwas clearly recognized at the earliest conception ofTRMM (Thiele, 1987). A corollary objective involvesa substantial supporting research component to pur-sue process studies for better understanding and char-acterization of various rain systems and to improvesurface-based measurements of rainfall by advancingradar and rain gauge technology. Characteristics ofrainfall cover a wide range of temporal and spatialscales, affecting both in situ validation and remotesensing measurements.

Because of the high spatial and temporal vari-ability of tropical rainfall intensity, estimation ofrainfall patterns by in situ networks and remote sens-ing platforms require special attention to spatial andtemporal statistical properties of rainfall. One of thegoals of this mission is to measure rain rate inten-sity distributions, and spatial and temporal statisticsof rainfall variability. TRMM is effective only inthe tropical belt and may be the best solution inthe long run. These methods have been investigated(Rosenfeld et al., 1993, 1995) and will continue

to be evaluated by cooperators from around theworld.

4. Data transmission

New technologies allow many entities to collectclimate data, however, proper management, process-ing and database establishment of the informationare the keys to success. In many cases, the data mustbe shared to order to ensure a representative spatialdepiction of the climate for a specific area. Each net-work has internal protocols to observe and a databaseof the information collected. It is this processing ofthe information that requires significant coordinationto ensure that stations are correctly identified and thedata are properly associated with the station.

4.1. Manual data forwarding

The basic options for an observer to transmit climateinformation once it has been collected have expandeddramatically. The following techniques may be used,some very effectively, if funds are not available toinvest in an automated climate station.• Mail (postal service)• Telephone (voice w/recorder and subsequent tran-

scription)• Touch-tone electronic entry directly to a data

collection computer system• Internet browser entry directly to a data collection

computer systemThe Internet option can provide the observer with

the tools needed to interactively enter, quality controland view not only their observations, but from otherswho have reported their observations in both a textand graphic environment.

4.2. Automated data forwarding

Once data are collected by an automated climatestation, a mixture of both old and new technologiescan be accommodated for data transmission. The tech-nologies are summarized as follows:• Telephone (a climate station dials-out or is dialed-up

by a computer system)


• Radio; land-based line-of-sight repeaters directly toa computer system

• GOES downlink to a computer system• Meteor-burst communications (http://www.wcc.

nrcs.usda.gov/bbook/bb25 .html)Each method has its own strengths and weaknesses.

Each can be tailored to meet specific mission require-ments based on location, frequency of observation andcosts, both installation and maintenance. The informa-tion collected by these networks can be integrated withother databases once data protocols are established.

5. Data exchange formats

The electronic exchange of climate data has beenmade easier thanks to the high-speed computer net-works of the Internet. However, the need for commonexchange formats has grown more critical. A wide va-riety of exchange formats exists and has been adoptedfor use. This presentation focuses on a human andmachine-readable format in wide use within the USas an example.

5.1. SHEF — standard hydrometeorologicalexchange format

The standard hydrometeorological exchange for-mat (SHEF) (Pasteris et al., 1996) is a documented setof rules for coding of data in a form for both visualand computer recognition. It is designed specificallyfor real-time use and is not designed for historicalor archival data transfer. All the critical elements foridentification of data are covered. Station identifiers,parameter descriptors, time encoding conventions,unit and scale conventions, and comment fields areall part of the code.

SHEF was designed for interagency sharing of data,visual and machine readability, and compatibility withanticipated receiving databases. The widespread im-plementation of SHEF allows the same decoding soft-ware to process data from various agencies. New datasources can easily be added as they become avail-able. The visual nature of SHEF allows users quicklyto become familiar with it. SHEF fully qualifies thedata so that a receiving database has all the necessaryinformation to describe the data. Additional informa-

tion about SHEF can be found at the following URL(http://hsp.nws.noaa.gov/hrl/shef/ index.htm).

5.2. Basic formats — .A, .B, .E

There are three formats that make up SHEF.Through the use of parameter code characters to iden-tify the data, these three basic message formats havethe flexibility to transmit a wide range of hydromete-orological information. The formats are as follows:• .A — single station, multiple parameter• .B — multiple station, multiple parameter, header

driven• .E — single station, single parameter, evenly spaced

time series

5.2.1. SHEF .A formatThe .A format is designed for the transmission of

one or more hydrometeorological parameters observedat various times for a single station. The .A formatconsists of positional fields and the data string. Theformat can be used for stations that report several dif-ferent types of hydrometeorological data or report datawith uneven time spacing.

5.2.2. SHEF .B formatThe .B format can be used for the transmission

of one or more hydrometeorological parameters fromseveral stations for which many or all of the param-eters are the same and are observed at correspond-ing times. The .B format consists of three basic parts:header, body, and terminator. The header consists ofthe positional fields and the parameter control string.The body contains station identifiers and data with op-tional date/data overrides. The terminator ends the en-tire .B format message. The format is useful for a rou-tine morning roundup of precipitation and river datafor a group of stations.

The .B header provides all pertinent date-time andparameter code information needed to decode the datacontained in the body of the message. The order of theparameter list is flexible and can vary from messageto message. The data values in the body are associatedwith the order of the parameter codes supplied in theheader line. Any parameter that can be reported in the.A format can be reported in the .B format; however, if


more than three stations are to be transmitted routinely,the .B format should be more convenient.

5.2.3. SHEF .E formatThe .E format can be used for the transmission of

several values of a single hydrometeorological param-eter at an evenly spaced time interval for a single sta-tion. As shown further, the F format consists of posi-tional fields and the data string. This format is usefulin the transmission of any evenly spaced time seriesof observational, processed, or forecast data.

The .E format is very similar in structure to the .Aformat except for two minor differences. First, in orderto avoid ambiguity, the .E format accommodates onlyone hydrometeorological parameter while the .A for-mat can handle several. Second, the .E format requiresthe explicit specification of a time interval (incremen-tal or decremental) between data values. SHEF hasbeen adopted as a NWS standard transmission formatand has also been accepted as a Office of Federal Co-ordinator for Meteorology (OFCM) standard for

Fig. 2. Conceptual design of data acquisition, processing, data base establishment and accessability of data by users.

hydromet data transmission. A complete documentdescribing SHEF can be found on the WWW atthe following URL: http://hsp.nws.noaa.gov/hrl/shef/index.htm

6. Data management

6.1. Introduction

The management of climatic data in this elec-tronic/information age has become easier, faster, andmore efficient. Yet, management of the data is per-haps one of the most critical processes in any designconcept. There are several key areas where attentionneeds to be directed as one considers their particulardesign requirements. These areas include data collec-tion, data processing, quality control, archiving, dataanalysis and product generation, and product deliveryas shown in Fig. 2. Establishing a database for re-motely sensed data can be overwhelming and requires


large disk storage capacity. Potential users are as-sumed to have access to personal computers that have8mm tape storage that may be slow to retrieve. Theread-write CD systems offer a better alternative forstorage and access to the data. Volumes of remotelysensed and AWS data are so overwhelming that thecollection and processing of these data should be fullyautomated. Potential users of remotely sensed data arefully dependent on these data management facilitiesfor their studies, as relatively fewer users have theresources that allow for the analysis of remote sensedinformation as highlighted in WMO (1996).

6.2. Data processing requirements

The design of a data processing system is directedby the users needs such as the type of data, frequencyof observation, source of the data, and the format it isdelivered.

Obviously if the data are being delivered viahigh-speed satellite downlink the process is much dif-ferent than that being manually entered. Yet both theseextremes have a common data management require-ment, they have to be accepted into the system. Inother words, an information system has to have multi-ple data entry points, requiring multiple user/machineinterfaces, processing multiple types of information.

The PC environment (or the networked worksta-tion) offers flexible machine/human interfaces whilethe WWW adds dynamic and creative solutions notrestricted to local computer systems or databases.In any event, the information that enters a databasemaintained in a computer and the products retrievedby users are their responsibility. It is critical thata long-term relationship be established with boththe data suppliers and product users to ensure thathydroclimatic information is of the highest qual-ity to properly guide decisions made by naturalresource decision managers. This activity requiresconstant attention to detail. The importance of thetime devoted to basic data entry design cannot beoveremphasized.

6.3. Quality control requirements

The essence of quality control is the assessment andimprovement of imperfect information, by making use

of other imperfect information. The following high-light some of the reasons for data quality control:• To insure that the exact information needed is being

properly generated.• Inconsistent data need to be identified or will lead

to incorrect decision inputs.• For operational needs to detect problems that re-

quire immediate maintenance attention.• For some purposes, continuous data are required,

even if such data are estimated.• Maintain an audit trail on data for future needs to

update documentation.• Data are often important for legal or design consid-

erations.• An established QC process attaches individual data

values with flags that allow software to select datatolerance levels for product generation.Quality control should be thought of as an

end-to-end process, which is integral to the successfulacquisition, temporary storage, permanent archival,and subsequent retrieval, use and interpretation of thedata and information produced by a network.

Since there can be literally thousands of values tobe examined (depending on a network’s size), theremay be a need for automation and/or an intelligentexpert system approach. Although human interven-tion is needed, good automated processes will reducethe need for intervention, and provide supplementaryinformation to help the human make a decision. Ex-amples of quality control screening of data are foundin most all-domestic networks in the US. The NCDC(http://www.ncdc.noaa.gov) performs extensive qual-ity control on the US NOAA cooperative network.These include both manual (double key entry of datamanuscripts) and automated (temporal and spatialprocedures). Many automated networks, including theNWS automated surface observation system (ASOS)(http://www.nws.noaa.gov/) also have built in QCfilters. Whatever the process, record keeping is per-haps the most important. The establishment of QCflags for climate data is well documented for manynetworks. Individual datasets archived at a variety ofclimate centers describe the data flags used to iden-tify the level of QC (http://www4.ncdc.noaa.gov/ol/documentlibrary/datasets.html).

Today’s technologies of electronic processing ac-tually make the documentation problem less trou-blesome, and hence should not be avoided as you


develop your network QC process. The Americansociety for agricultural engineers (ASAE) has pub-lished draft guidelines reviewing the measurementand reporting practices for automatic agriculturalweather stations (Ley et al., 1994). It is an excellentsource of technical standards for data managementissues.

7. Metadata concepts

7.1. Station identification and history

In managing metadata it is critical to maintain allknown names and aliases and identification numbersfor stations. The accurate identification of data collec-tion stations, an obviously fundamental component ofmetadata, is more complex than it first appears. Ge-ographic ‘place’ names are the easiest and most userfriendly means of station identification. The follow-ing station description elements (Lazar et al., 1999)will be incorporated into the UCAN, on line, relationalmetadatabase:• Station identity (site name, aliases, and all identifi-

cation numbers including network identification)• Location (coordinates, elevation, geopolitical place-

ment, topography, etc.)• Equipment/instrumentation and its exposure• Data observing and dissemination practices (net-

work membership)• Data inventory, and• Temporal changes to any of this information.

The station history data will detail the changes inthe station over time. This includes, but is not limitedto, the location, naming and equipment. Such docu-mentation include the following:• Station location changes (latitude, longitude, and

elevation), station name changes,• Time of observation for each element and dates of

any changes,• Beginning and ending dates for each reported ele-

ment,• Addition or removal of element or sensor,• For hourly precipitation stations, type of rain gauge,• Type of recording equipment and dates of changes,

and• Observers names and dates of service.

7.2. Metadatabase management

The UCAN metadata database design employs arelational database, which incorporates the four rulesdiscussed earlier in order to maintain consistency,insure flexibility, and yet accommodate the inconsis-tencies that are so common in climate station historyinformation. The database design begins with a centraltable that defines a single station with a unique, in-ternal station identification number. This content-freeinternal station identifier is purely for database man-agement purposes (a primary key variable) and is notfor distribution to data and metadata users.

Multiple station identifiers, names and aliases,networks and all other station history informationare maintained in normalized tables linked by manyrelationships in the internal station (ID). Tables doc-umenting station attributes that change through timeincludes time variables (begin and end dates). Thestation network table contains a specific type ofstation identifier that is associated with a network(for example, the NCDC TD3200 network uses theNWS Coop station number). The final considera-tion in this metadata management system solutionis a rule for the depiction of climate data in reportproducts.

7.3. Database management

A wide variety of database choices face the cli-mate user community. The increased use of relationaldatabases to store significant amounts of data providesa tempting opportunity to create a climate databasethat comes with a significant amount of database man-agement software. These databases work well if youhave a relatively small database, run the database ina central location and do not plan to exchange largeamounts of data on a regular basis. There is usuallya licensing agreement attached to using a commercialdatabase, which can vary significantly based on thecomputer platform running the database.

8. NetCDF (Network common data form)

For the UCAN project, the climate databaseswould be very large (over one billion data values),


would be distributed among eight regional and fed-eral climate centers, and would need to support rapidaccess and exchange of data files between usersthroughout the US It was determined through pro-totype testing that NetCDF’s stored and retrievedtime series information more efficiently than anyother relational database available. This, coupledwith the freely available software tools and easein data exchange via the Internet, made NetCDFa logical choice. A full description of the entireUCAN Project can be found at the following URL,(http://www.srcc.lsu.edu/ucan.net/UCAN.html). TheURL also supports a demonstration of the UCANdata access methodology.

NetCDF is an interface for array-oriented dataaccess and a library that provides an implementa-tion of the interface. The netCDF library also de-fines a machine-independent format for represent-ing scientific data. Together, the interface, library,and format support the creation, access, and shar-ing of scientific data. The netCDF software wasdeveloped at the Unidata Program Center in Boul-der, Colorado. The freely available source is at(ftp://ftp.unidata.ucar.edu/pub/netcdf/) or from othermirror sites. NetCDF, is software for storing and re-trieving scientific data. More than a data format, thenetCDF package is a set of programming interfacesthat can be used with widely varying scientific datasets and by machines of widely varying architecture.

Table 1An example of the TD 3200 — summary of the day table showing variables and their associated dimensions

Variable Name Source Units Note

TMAX Daily maximum temperature 3200 ◦FTMIN Daily minimum temperature 3200 ◦FTOBS Temperature at observations time 3200 ◦FPRCP Daily precipitation 3200 0.1 in.EVAP Daily evaporation 3200 0.1 in.MNPN Daily minimum pan evaporation temperature 3200 ◦FMXPN Daily maximum pan evaporation temperature 3200 ◦FSNOW Daily snowfall 3200 0.1 in.SNWD Snow depth at observations time 3200 0.1 in.WTEQ Water equivalent of snow depth 3200 0.1 in.WDMV 24 h wind movement 3200 MilesSNnn Daily minimum soil temperature 3200 ◦FSXnn Daily maximum soil tempetature 3200 ◦FSOnn Soil temperature at observations time 3200 ◦FDYSW Daily occurrence of weather 3200

Multidimensional data may be accessed one point ata time, in cross sections, or all at once. Data are di-rectly accessible, permitting efficient access to smallsubsets of large data sets.

8.1. Climate element identification

The variables and their associated dimensions arenamed. An example of the TD 3200 — summary ofthe day (includes TD 3210) are described in Table1. Information about the data, such as what units areused and what the valid range of data values is, can bestored in attributes associated with each variable. Theprocessing history of a data set can be stored with thedata.

8.2. NetCDF transportability

The netCDF format renders netCDF files machineindependent. The netCDF package is particularly use-ful at sites with a mix of computers connected by anetwork. Data stored on one computer may be readdirectly from another without explicit conversion.The netCDF software has been used successfully ona broad range of computers, from PCs to supercom-puters. The netCDF library can be invoked fromc,c++, fortran, or perl programming languages.Data stored using programs in one language may beretrieved with programs in another language.


Many groups and organizations have adoptednetCDF for their scientific-data-access needs. Numer-ous freely available, licensed, and commercial pack-ages for data analysis and visualization can displayand manipulate netCDF data.

Unidata’s purpose in creating the netCDF libraryis to generalize access to scientific data so that themethods used for storing and accessing data are in-dependent of the computer architecture and the appli-cations being used. In addition, the library minimizesthe fraction of development effort devoted to dealingwith data formats.

Standardized data access facilitates the sharing ofdata. Since the netCDF package is quite general, awide variety of analysis and display applications canuse it. The netCDF library is suitable, for example,for use with satellite images, surface observations,upper-air soundings, and grids. By using the netCDFpackage, researchers in one academic discipline canaccess and use data generated in another discipline.

The Unidata netCDF package is available free ofcharge. It is also one component of a suite of soft-ware tools that Unidata distributes to universities.With these additional tools, universities can capturedata, store them in netCDF files, and display and an-alyze them as desired. The netCDF package containsc-language source code for the netCDF data-accesslibrary; source code for thec++ andfortran inter-faces; documentation of the netCDF library and utili-ties; and test programs to verify the implementation.

Unidata provides software revisions and upgradesto all its software tools, including the netCDFpackage. The netCDF software and documentationmay be obtained from the netCDF WWW site at(http://www.unidata.ucar.edu/packages/netcdf/)

9. Product generation and access

9.1. Formats and products

One of the exciting features of PCs is the abilityto produce quality products, formatted for easy read-ing and presentation, generated through simple wordprocessors, databases, or spread sheet applications.However, some thought needs to be given to what theproduct looks like and what it contains, before yourdatabase delivery design is finalized. The greatest dif-

ficulty many be encountered when producing a criticalreport and it is realized that key data or informationattributes are missing, especially when climatic dataare combined with agricultural, and human resourceinformation from separate databases. WMO (1986) isa good source for data formats and distribution.

9.2. Graphics

The presentation of agrometeorological data ingraphical form has also been made easier via today’scomputer applications. While some of us may notcare about a particular software package ‘graphicstyle’, it may be important to have access to a suit-able package to graph information that can be cut andpasted in a report or presentations. A wide variety ofsoftware now exists to plot both time series and spa-tial information. Information about time series plot-ting can be found at (http://www.wrcc.sage.dri.edu)and information on spatial plotting can be found at(http://grads.iges.org/grads/head.htm). Fig. 3 is an ex-ample of a time series plot of NRCS SNOTEL datacollected at a test site in Oregon, USA. An exampleof a spatial plot is shown in Fig. 4. Shades of colorare used to show gradients in air temperature andalso display contour lines for dew point temperaturedistribution.

9.3. Local processing versus remote processing

In general most computer systems now have thecentral processing unit (CPU) power to run a vari-ety of applications locally. However, access to largedatabases and CPUs using Telnet software and theInternet can make remote processing very attractive.This is especially true if your data center does nothave the database resources to support a spatially andtemporally relevant database. Graphics and productsshown previously can be run remotely and images andproducts downloaded to your computer via FTP. If youcontribute observations to a central database, and havethis computer access, you would benefit from the pro-cessing of information not collected by your agency,but available from the central facility. The Internet isclearly a new communications tool that must be con-sidered when designing a database and climatic dataprocessing operation.


Fig. 3. Time Series Plot of NRCS Snotel Data. Courtesy Western Regional Climate Center.

Fig. 4. Spatial plot of RAWS Temperature and Dew Point. Courtesy Western Region Climate Center using free GrADS software package.Temperature Field — Shaded: Station Temperatures (Red Values); Dew Point Field — Contoured.


9.4. Data and statistical computation

It is not our intent to cover the statistical techniquesappropriate for application to climatic datasets whichare thoroughly discussed in WMO (1983). Considera-tions in database and interface design requirements toaid in subsequent analysis are documented. Issues ofmissing data, extreme values, non-numerical values,or out-of-range values are common in data analysis.However, simple presentation of datasets to analyticalsoftware packages can be a challenge. New methods ofdata analyses have been developed using artificial neu-ral network analysis techniques (Michaelides, 1999).These methods have been used for estimating miss-ing data and completing time series and many otherapplications. Consider carefully your ability to extractprecise datasets from your database and making themavailable to your software package.

Common formats acceptable to most statisticalpackages and/or spread sheets include: comma ortab delimited ACSII text file structures, LOTUS 8,Microsoft Excel 8, SAS 8, or Quattro Pro 8 filestructures. The netCDF specialized structure and for-mat has the potential of streamlining statistical dataanalysis and include the following:• Raw daily lister• Compact daily lister• Monthly time series• Degree day summaries• Normal lister• Generalized climatology• Daily normals• Daily percentiles• Temperature summaries• Frost-last and first frost dates• Growing season length• Extreme daily values for the month• Monthly/annual model data• Frequency distribution and probability.

10. Summary

The importance for climate and agrometeorologi-cal data in the operational needs of agricultural pro-ducers, researchers, and agricultural weather and cropforecasters are gaining attentions in the domestic andthe international research community. Databases have

been developed in many countries although there arestill quite some countries where there is a need for es-tablishing databases that are accessible for their citi-zens and the international community. Modernizationof data acquisition, processing and archiving can be acostly proposition but steps need to be taken, to ensurethat such data are archived in a standardized mannerthat can be useful.

This paper outlines the current methods of datacollection and particularly introduces remote sensingtechniques that may be a solution in countries thatare considering upgrading their current system. Re-mote sensing data provides a spatial component ofcrop growth and development that cannot be collectedin ground-based measurements.

Computer technologies are advancing rapidly, andtherefore it is unrealistic to try and articulate allpotential solutions to agrometeorological informa-tion systems for there are surely more technologysolutions currently under development. It is hopedhowever, that the reader has been exposed to someof the key elements of what should be considered indesigning and implementing a computerized climateinformation delivery system.

An example for a conceptual data base managementsystem adapted by the US Department of Agricul-ture is presented as a framework for the acquisition,maintenance and distribution of climate and agrom-eteorological data for the 21st century. The accessof data to the national and international communitycan be resolved with the standardization of data basemanagement and electronic accessibility.

Databases in the US have been upgraded for ac-cess through the world wide web, and include dataarchived from other countries that have co-operativeprograms with US institutions. Perhaps the work cur-rently being done with UCAN in the US Departmentof Agriculture can be expanded into the internationalclimate community.

The inclusion of remotely sensed data in the cli-mate and agrometeorological database will take a ma-jor effort but the operational and research communityis currently using a number of pertinent parameters.The Technical Commission for Agricultural Meteo-rology of the WMO could move towards identifyingsuitable parameters and standardizing the format ofremote sensing data in the climate and agrometeoro-logical database.


Acknowledgements

Contributions for this chapter were received fromS.C. Michaelides, L. Moussa and I. Seitnazarov isgreatly appreciated. The authors wish to acknowl-edge the excellent technical review and recommenda-tions provided by Professor C.J. Stigter, Dr. RichardCoe and the other reviewers in preparation of themanuscript.

References

Doraiswamy, P.C., Stern, A.J., Zara, P.M., 1997. Monitoring cropprogress using NOAA AVHRR data adjusted for seasonalclimatic variation. In: Proc. 61th Am. Soc. Photo. Remote Sens.,Seattle, WA, 7–11 April 1997.

Doraiswamy, P.C., Zara, P.M., Stem, A.J., 1998. Integration ofremote sensing and simulation models for regional productionassessment. Agronomy Abst., p. 21. Annual Meeting of ASA,CSSA and SSSA, Baltimore, MD, 18–22 October.

Kogan, F.N., 1994. Application of vegetation index and brightnesstemperature for drought detection. Adv. Space Res. 15, 91–100.

Kogan, F.N., 1995. Droughts of the late 1980s in the United Statesas derived from NOAA polar orbiting satellite data. Bull. Am.Meteorol. Soc. 76, 655–668.

Kustas, W.P., Perry, E.M., Doraiswamy, P.C., Mora, M.S., 1994.Using satellite remote sensing to extrapolate evapotranspirationestimates in time and space over a semiarid rangeland basin.Remote Sens. Environ. 49, 275–286.

Lazar, A.V., Barthel, C.D., Kuiper, D.G., 1999. Metadata strategiesfor the efficient management and exchange of data throughout aunified climate access network. In: 11th Conference on AppliedClimatology, American Meteorological Society, Dallas, TX,10–16 January 1999.

Ley, T.W., Elliott, R.L., Bausch, W.A., Brown, P.W., Elwell,D.L., Tanner, B.D., 1994. Measurement and reporting practicesfor automatic agricultural weather stations (X5 OS StandardsProject). Paper No.942086, American Society for AgriculturalEngineers (ASAE) 1994 International Summer Meeting, St.Joseph, MO, USA.

Maas, S.J., Doraiswamy, P.C., 1996. Integration of satellitedata and model simulations in a GIS for monitoringregional evapotranspiration and biomass production. In:Proceedings of the National Center for Geographic Informationand Analysis. Third International Conference/Workshop onIntegrating GIS and Environmental Modeling, Sante Fe, NM,(http : //www.ncgia.ucsb.edu/conf/sf papers).

Michaelides, S.C., 1999. Artificial neural networks: a novelapproach in the analyses of meteorological data. Contributionfrom members in the International Workshop on OperationalApplications in Agrometeorology in the 21st century, Needs andPerspectives@, Accra, Ghana, February 1999. WMO, Geneva.

Moussa, L., 1999. Experiences of Niger in collection andmanagement of agrometeorological data and their uses byagricultural services and various users (French). In: CagMReport 77, Contribution from members in the InternationalWorkshop on Operational Applications in Agrometeorologyin the 21st century, Needs and Perspectives, Accra, Ghana,February 1999. WMO, Geneva.

Pasteris, P.A., Bissell, V.C., Bonnin, G., 1996. Standardhydrometeorological exchange format, Version 1.2. USDepartment of Commerce, Weather Service HydrologyHandbook No. 1, Office of Hydrology, National WeatherService, Silver Spring, MD, February 1996.

Pasteris, P.A., Marron, J.K., Johnson, G.L., Garen, D.C., 1997.Alliance for Progress B How the USDA-NRCS is meeting theclimate challenges of the 21st century. In: Proceedings B 10thConference on Applied Climatology, Reno, NV, 20–24 October1997. American Meteorological Society.

Rosenfeld, D., Wolff, D.B., Atlas, D., 1993. General Probabilitymatched relations between radar reflectivity and rain rate. J.Appl. Meteorol. 32, 50–72.

Rosenfeld, D., Amitai, E., Wolff, D.B., 1995. Improved accuracyof radar WPMM estimated rainfall upon application of objectiveclassification criteria with radar. J. Appl. Meteorol. 34, 212–223.

Seitnazarov, I., 1999. Technology and methods of collection,distribution and analyzing of agrometeorological data inDashhovuz velajat, Turkmenistan. In: CagM Report 77,Contribution from members in the International Workshopon Operational Applications in Agrometeorology in the 21stcentury, Needs and Perspectives, Accra, Ghana, February 1999.WMO, Geneva.

Thiele, O.W. (Ed.), 1987. On requirements for a satellitemission to measure tropical rainfall. National Aeronautics andSpace Administration, Research Publication, NASA RP-1 183,Washington, DC.

WMO, 1983. Guide to Climatological Practices, Second Edition.WMO Publication No. 100, Geneva.

WMO, 1986. Guide to Agricultural Meteorological Practices.WMO Publication No. 134, Geneva.

WMO, 1988. RAV Working Group on Agricultural Meteorology.WCP/AGM, CAgM Report No. 31, Geneva.

WMO, 1996. Agrometeorological Data Management. WMO/TD -No.748, CAgM Report No. 65, Geneva.

WMO, 1997. Meeting of the CC Working Group on Climate Data.Final Report, WMO/TD No. 841, WCDMP-No. 33, Geneva.

techniques for methods of collection, database management

Documents