eurasia institute of earth sciencesistanbul technical university climate data formats deniz bozkurt...

Eurasia Institute of Earth SciencesIstanbul Technical University

Climate Data Formats

Deniz Bozkurt

web.itu.edu.tr/bozkurtdITU Eurasia Institute of Earth Sciences

[email protected]

The International Training Course on“Climate Analysis and Applications”

Alanya10-19 October 2011

The International Training Course on“Climate Analysis and Applications”

Alanya10-19 October 2011


Datasets

Climate Datasets


Raw data

“abcd12345”

Input system

Digital representation “11010001”

Climate Datasets


Climate Datasets

• Modeled data: gridded multi-dimendisonal data– Consisting of variables, dimensions and attributes

• Observations (station, reanalysis)


Climate Datasets

• Climate data can be stored as different formats in different platforms:

• Character format

• Packed binary format


Climate Data Formats - Character Format

• ASCII (American Standard Code for Information Interchange)

• It refers to a “text” file that is readable by the naked eye (it only contains the letters a-z, numbers, carriage returns, and punctuation marks.

• Each character is stored as digital

============

Binary100100011001011101100110110011011110101100010000011101111100111111001011011001100100

Hello, world


Climate data formats – ASCII


Climate data formats – ASCII

• Factors restrict the use of ASCII:

• Weak data storage and huge data size:

“9” 1 byte, “679.43” 6 byte, -0.123456E+05 13 byte

• No or not practical metadata


Climate data formats – Packed Binary

• A common data format for saving and storing of large size of the data

• “3.1” 24 bits (ASCII) can be packed as 5 bit

• Can be processed very quickly and effectively in data analysis


Climate data formats – Packed Binary


Some standard data formats

• Self-describing data format– All the information about data is stored inside of the data

– No need for an additional information for the identification of the contents of the file

– Machine independent

• Can be readable by various applications– NCL, IDL, GrADS, Matlab, Ferret, C, C++, Java, Fortran

– Komut satırı operatörler: NCO, CDO

• Most commonly used format in the oceanographic and atmospheric science for observational data and numerical modeling (IPCC datasets, NASA, NOAA, NCAR, UCAR…)

NetCDF (network Common Data Form) Format


Climate data formats - NetCDF

metadata

dimensions

variables

Variableproperties

attributes



Data storage in NetCDF format

• 3D (x,y,t) or 4D (x,y,z,t)



Data storage in NetCDF format

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

netcdf mynetcdf {

dimesions:

x=4;

y=4;

time=UNLIMITED;

variables:

float x(x);

float y(y);

int time(time);

float temperature(time,x,y);

data:

x = 10, 20, 30, 40;

y = 110, 120, 130, 140;

time = 31, 59, 90;

}



Data storage in NetCDF formatnetcdf mynetcdf {

dimesions:

x=4;

y=4;

time=UNLIMITED;

variables:

float x(x);

float y(y);

int time(time);


data:

x = 10, 20, 30, 40;

y = 110, 120, 130, 140;

time = 31, 59, 90;

Temperature = 111, 211, 311, 411;

}

141 241 341

131 231 331

121 221 321

441

431

421

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

time = 1

x = 1 to 4

y = 1




dimesions:

x=4;

y=4;

time=UNLIMITED;

variables:

float x(x);

float y(y);

int time(time);


data:

x = 10, 20, 30, 40;

y = 110, 120, 130, 140;

time = 31, 59, 90;

Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441;

}

time = 1

x = 1 to 4

y = 1 to 4

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time




dimesions:

x=4;

y=4;

time=UNLIMITED;

variables:

float x(x);

float y(y);

int time(time);


data:

x = 10, 20, 30, 40;

y = 110, 120, 130, 140;

time = 31, 59, 90;

Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441, 112, 212, 312, 412, 122, 222, 322, 422, 132, 232, 332, 432, 142, 242, 342, 442, 113, 213, 313, 413, 123, 223, 323, 423, 133, 233, 433, 143, 243, 343, 443;

}

time = 1 to 3

x = 1 to 4

y = 1 to 4

Y

X

Time


4D (x,y,z,t)

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

Altitude




some NetCDF commands

– ncdump file_name | less, to see the entire contents of the file

– ncdump –h file_name | less, to see the front portion of the data

– ncdump –v var_name file_name | less, to see the specific data within the file

– ncl_filedump to see the contents of different data formats


Climate data formats - GRIB

GRIB (GRIdded Binary) Format

• World Meteorological (WMO) standard, commonly used in meteorology

• Bit-oriented data format: very efficient for transmission/archival

- netCDF (float) 2-4 times larger than corresponding GRIB

• two dimensional horizontal grids only

- each grid has scale/offset for optimal precision

• requires external parameter table (s)



• Basically consists of:– Product definition section(PDS)

• Creater of the data, unit of the data, time stamp, vertical system of the data

– Grid description section (GDS), optional• Grid description, map projections

– Bit map section (BMS), optional• Defines which grid points will be presented

– Binary data section (BDS)• Contains the packed binary data

– End section



• Files are typically 1/2 to 1/3 of the size of normal binary files:

• Y: gridded data

• D: scale factor

• R: reference value

• E: binary scale factor

€

Y ∗10D = R + X ∗2E( )


• NetCDF data:• Raster

• Table

• Feature

• Directly readable

Climate data formats – ArcGIS Shapefile



Raster

Feature

Table



http://www.gisclimatechange.org/

eurasia institute of earth sciencesistanbul technical university climate data formats deniz bozkurt...

Documents