presented at the thic meeting at the national center for atmospheric research, 1850 table mesa...

Presented at the THIC Meeting at the National Center for Atmospheric Research, 1850 Table

Mesa Drive, Boulder CO 80305-5602

July 19-20, 2005

Evolution of Archive Technologies at the

National Snow and Ice Data CenterRuth Duerr

National Snow and Ice Data Center1540 30th St, Boulder, CO, 80309-0449

Phone 303-735-0136 FAX: 303-492-2468E-mail: [email protected]

Evolution of Archive Technologies at the National Snow and Ice Data CenterPresented at the THIC Conference, Boulder CO, July 19-20, 2005

Outline

• A brief history of NSIDC• Current holdings• Current systems• Thoughts on the future


A brief history of NSIDC

• World Data Center for Glaciology 1957 - U.S. National Committee for the IGY awarded

the operation of WDC-A for Glaciology to the American Geographical Society under the direction of Dr. William O. Field

1970 - WDC for Glaciology transferred to the U.S. Geological Survey in Tacoma, Washington under the direction of Dr. Mark F. Meier

1976 - WDC for Glaciology transfers to the NOAA Environmental Data and Information Service; an agreement between the University of Colorado and NOAA placed the WDC at CU-Boulder, Colorado under the direction of Roger Barry



• National Snow and Ice Data Center 1982 - NOAA grants NSIDC its name 1983 - NSIDC receives a grant from NASA to

archive Nimbus 7 passive microwave data 1990 - NSIDC receives funding from NSF for

the Arctic System Science (ARCSS) Data Coordination Center (ADCC)

1993 - NSIDC receives first NASA Distributed Active Archive Center (DAAC) contract



• 1996 - Antarctic Data Coordination Center (ADCC) established with NSF support

• 1999 - Antarctic Glaciological Data Center (AGDC) established with NSF support

• 2002 - Frozen Ground Data Center (FGDC) established with International Arctic Research Center (IARC) support


Outline



Current Holdings

• Information Center ~44,000 monographs,

reports, serials, reprints, etc.


Current Holdings (continued)

Analog Archives• ~ 10,000 glacier photos• ~ 7,000 sea ice charts• ~ 1,440 maps• TBD cu ft of

manuscripts and other records


Data Holdings (continued)

Digital Archives• ~ 440 publicly advertised

data sets• 4.6 million granules in ECS

system• > 3.5 million files in

non-ECS systems

Archive Types• ~ 8 TB on-line• ~ 80 TB near-line• >5 TB off-line• Off-site backups for

primary data without recovery agreements


NSIDC Near-Line Archive Size

NSIDC Total Archive Size

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

10/6/0512/6/052/6/064/6/066/6/068/6/06

10/6/0612/6/062/6/074/6/076/6/078/6/07

10/6/0712/6/072/6/084/6/086/6/088/6/08

10/6/0812/6/082/6/094/6/096/6/09

Date

Terabytes(TB)


NSIDC Distribution Statistics

NSIDC Data and Information Request Totals Fiscal Years 1978-2004

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004Fiscal Year

Number of Requests

Sixty-six % of requests in FY 2004 were for ECS Products

ECS Product Distribution in GB per Month and Instrument Type

0

500

1000

1500

2000

2500

3000

3500

Feb '00May '00Aug '00Nov '00Feb '01May '01Aug '01Nov '01Feb '02May '02Aug '02Nov '02Feb '03May '03Aug '03Nov '03Feb '04May '04Aug '04Nov '04Feb '05May '05

Month

GB DistributedTerra MODIS

Aqua MODIS

Aqua AMSR-E

Total Dist.

ICESat Glas

ECS Product - Total Granules Distributed by Month and Instrument Type

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

Feb '00May '00Aug '00Nov '00Feb '01May '01Aug '01Nov '01Feb '02May '02Aug '02Nov '02Feb '03May '03Aug '03Nov '03Feb '04May '04Aug '04Nov '04Feb '05May '05

Month

Granules

Terra MODIS

Aqua AMSR-E

Aqua MODIS

TOTAL

ICESat GLAS


Outline


UCB GE Campus LanAbilene via

Front Range GigaPop

Cisco

GE switch

ECS

Router

Ebnet

Router

Firewall

LASP

GES DAAC (MODIS)AMSR-E SIPSI-SIPSPO.DAAC AMSR L1AEDOS

M&OLAN

(Includes M&O Intranet Server and

Backup Server)

Ingest & Distribution• ftp/file server• tape production• Disk Storage (FC and SATA @ RAID-5)• Rimage CD/DVD• AMSR L1A PDR/Met

ECS Production LAN

UCB non-ECS LAN

Web Services• Guide Documents• SOTC, All about…• Catalog/DIF generator• Client Interfaces - EDG,GISMO,PSQ, SNOWI - ECHO Client (WSRD)

Archive Services• MAID vtl w/ AMASS (transition from tape archive in progress)• Off-site archive

Raytheon(Denver)

Infrastructure Services•Email• Calendar Manager• Center-wide Intranet• Backup Services• SNIPS• IDS

Science Processing Dev /Production

• SSM/I, NISE, NRTSI - Std Processing Env.• GISMO/PSQ/WSRD -backend services• AMSR-E PDR/.Met

ICESat/GLAS Remote SCF Server•Visualization•Subsetting Services•(Includes storage forArchived GLAS products)

HEG Server• HDFEOS-to-GEOTIFF

ECS Subsetter• Spatial, temporaland parameter subsettingof granules (HDF-EOS Only)

Data Management• Data Dictionary•V0Gateway - Inv. Search/Results• Order Requests - subsetting services• ECHO transfers

Data Server• Manage the archive -STK Powderhorn w/ AMASS and ACSLS

• Insert data into archive• Search and Retrieve data from the archive

Ingest Polling Server•ASMR, AMSR-E• NISE• MODIS• ICESat

Data Pool• On-line storage of most recently ingested data (StorNext SAN)• Data accessible via WEB GUI or ftp

Infrastructure• Backup services• What’s Up• SNIPS• MSS and CSS• Email Gateway

Order Manager• Manages orders from V0Gateway, MTMWG, and Spatial Sub. Server• Transfers orders to PDS for media requests

NSIDC Architecture

VJT 7/05


NSIDC Data Catalog

• Contains metadata about each published data set GCMD, FGDC compliant

metadata• Used to drive web page

creation• Modification underway to

include OAIS/PREMIS compliant metadata for all data sets


EDG CLIENTMTMGW

Subscriptions

QA SCFs SCFs SCIENCE USERS GENERAL USERS

ECS NSIDC DAAC

NISE

NSIDC PDR Server

EDOS

AMSR L1A [1.3]

GBAD PDS [<1] AMSR-E PDS [1.4]

NISE Data

AMSR-E SIPS

L2+ Products [3.6]

Total Data Distribution • Aqua AMSR-E Data Products [7] • Aqua MODIS Data Products [16] • Terra MODIS Data Products [40] • ADEOS II AMSR Data Products [1] • GLAS Data Products [7]

[<1]

GDAAC LDAAC LaTIS

NISE Data[total <1]

MODIS SCFs

MODIS L2/L3 Snow/Ice Products

QA Updates

[2/T, 1/A]]

[<1/T, <1/A]

MODAPSMODIS L2/L3 Snow/Ice Products [120/T, 16/A]

[ ] : Archive Volumes at L+1yr in GB/day T: Terra A: Aqua

NASDA AMSR-E L0 Science & GBAD PDSs

8mm back-up 1st 90 days [1.4] Emergency back-up following L+90

GLAS L0 [5.8]

GLAS SIPS

GLAS L1+ Products [23]

Data/QA Updates

EDOS

LP DAAC

ASTER Anc [<1] GDAACAQUA Ancillary from EMOS [<1]

ECHO Metadata and Browse

NSIDC DAAC ECS Interfaces

NSIDC/UCB InterfacesNSIDC/UCB

NISE Data (ftp push)

sidads

NSIDC/ECSAMSR/AMSR-E L1A, PDR’s and met (ftp push)-

adcc / agdc(arcss)

ftp push

Public User Communities

ftp pull

Requests, results, orders(TCP Sockets)

E-mail andattachments

kryos(Email-server)

E-mail

html,.gif, jpeg,data, etc.

SSM/I Wentz Tas(RSS)

FedEx

AMSR/AMSR-E-JPL/PO.DAAC

Fastcopy

push

ftp pullSSM/I TBs

SNODAS(NORSC)

Glacier Photos

AVHRR imagery(Antarctic Ship)

ftp pull

PermafrostInvestigators

ftp push/pull

ARCSS Investigators

VorticityModelingData(USGS/Denver)rcp

NCEP GlobalReanalysis (CDC)

Coriolis

ftp pushftp push

NCEP GlobalReanalysis (UNH)

Via UCB Network Via NASA EMSn/ECS Media (Tape/CDR)

Ftp push/pull

Mediadistribution

glacierGISMO-E

bipolar

EDG/SNOWI

(Web server)

DocumentsImages

Forms

GISMO/PSQ

IMS (NCDC) ftp pull

Sea Ice Charts(Arg)

ftp push

Dehn Ice Charts

IceCharts/Reports (NIC)

AMSR-E NRT(MSFC)

ftp pull

Weekly Snow Map(Rutgers)

ftp pull

ftp pull

(GHRC/UAH MODIS Browse (ftp push)

DVD

CDftp

ftp pull

rSCFRequests

GLAS data(ftp push)

Subsetted GLAS dataDVD


COPAN MAID - Early Experiences

• Using AMASS to provide a file system front end to a COPAN 200t system configured as a L700 with 7 - 9940 drives COPAN to act as a drop-in replacement for the

STK 9710• Migration of data underway • The big surprise was performance• Minor surprise is the ever shrinking total

archive size


Outline



Thoughts on the Future

• NSIDC DAAC is pushing for an entirely on-line archive What technologies will allow this? (SATA

RAID, COPAN, or ???) What happens to media? How do we ensure preservation of these data

over time?

presented at the thic meeting at the national center for atmospheric research, 1850 table mesa...

Documents

primary data

national center

data sets4

ice data center1540

noaa environmental data

name1983 nsidc

glaciology transfers

meier1976 wdc