1 dapper: an opendap server for in-situ data joe sirott donald w. denbo, willa h zhu university of...
TRANSCRIPT
![Page 1: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/1.jpg)
11
DAPPER: An OPENDAP DAPPER: An OPENDAP Server for In-Situ DataServer for In-Situ Data
Joe SirottJoe Sirott
Donald W. Denbo, Willa H ZhuDonald W. Denbo, Willa H Zhu
University of WashingtonUniversity of Washington
PMEL/NOAAPMEL/NOAA
![Page 2: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/2.jpg)
22
What is Dapper?What is Dapper?
Web server that provides distributed Web server that provides distributed access to in-situ data via OPeNDAP access to in-situ data via OPeNDAP protocolprotocol
Clients include GrADS, MatlabClients include GrADS, Matlab
Available data:Available data:– PMEL EPIC databasePMEL EPIC database– NODC World Ocean Database 2001NODC World Ocean Database 2001
![Page 3: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/3.jpg)
33
Problems with in-situ dataProblems with in-situ data
Irregular (x,y,z,t)Irregular (x,y,z,t)
Not all variables available in all profilesNot all variables available in all profiles
Data from a given profile may be revised Data from a given profile may be revised as QC is appliedas QC is applied
Each profile usually stored as individual Each profile usually stored as individual filefile
No standard APIsNo standard APIs
![Page 4: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/4.jpg)
44
Problems with in-situ dataProblems with in-situ data
Irregular (x,y,z,t)Irregular (x,y,z,t)
Not all variables available in all profilesNot all variables available in all profiles
Data from a given profile may be revised Data from a given profile may be revised as QC is appliedas QC is applied
Each profile usually stored as individual Each profile usually stored as individual filefile
Use Climate Data Portal for aggregationUse Climate Data Portal for aggregation
![Page 5: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/5.jpg)
55
Problems with in-situ dataProblems with in-situ data
No standard APIsNo standard APIs
Use OPeNDAP Sequence datatypeUse OPeNDAP Sequence datatype
![Page 6: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/6.jpg)
66
Dapper architectureDapper architecture
CORBA (IIOP)
Climate Data Portal
MySQL
Java netCDFlibrary
netCDFprofile
netCDFprofile
Dapper
CDPservice
netCDFservice
OPeNDAP client
OPeNDAP Protocol(HTTP)
![Page 7: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/7.jpg)
77
Climate Data Portal (CDP)Climate Data Portal (CDP)
CORBA basedCORBA based
Metadata in database (MySQL)Metadata in database (MySQL)
Variables (measurements) in individual Variables (measurements) in individual netCDF filesnetCDF files
Developed by Don Denbo and Willa ZhuDeveloped by Don Denbo and Willa Zhu
![Page 8: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/8.jpg)
88
OPeNDAPOPeNDAP
Sequence example (header):Sequence example (header):Dataset {Dataset {
Sequence {Sequence {
Float32 longitude;Float32 longitude;
UInt32 time[tsize = 2];UInt32 time[tsize = 2];
Float32 latitude;Float32 latitude;
Int32 _id;Int32 _id;
Sequence {Sequence {
Float32 depth;Float32 depth;
Float32 temperature;Float32 temperature;
} profile;} profile;
} location = { { -42.01, {77, 3850926208}, 53.3402, } location = { { -42.01, {77, 3850926208}, 53.3402, 3645, { { 3, 16.698 }, { 5, 16.71 }, …3645, { { 3, 16.698 }, { 5, 16.71 }, …
![Page 9: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/9.jpg)
99
OPeNDAP (cont.)OPeNDAP (cont.)
Sequences accessible to clients via Java Sequences accessible to clients via Java or C++ librariesor C++ libraries
_id field in “outer” sequence allows unique _id field in “outer” sequence allows unique identification of an individual profileidentification of an individual profile
![Page 10: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/10.jpg)
1010
Dapper designDapper design
Consists of configurable Consists of configurable servicesservices– Climate Data Portal serviceClimate Data Portal service– netCDF servicenetCDF service
Services route OPeNDAP requestServices route OPeNDAP request
Implemented as Java servletImplemented as Java servlet
![Page 11: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/11.jpg)
1111
Dapper architecture revisitedDapper architecture revisited
CORBA (IIOP)
Climate Data Portal
MySQL
Java netCDFlibrary
netCDFprofile
netCDFprofile
Dapper
CDPservice
netCDFservice
OPeNDAP client
OPeNDAP Protocol(HTTP)
![Page 12: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/12.jpg)
1212
CDP service exampleCDP service example
1.1. HTTP request:HTTP request:
http://foo.org/.../http://foo.org/.../dset.cdp.dods?lon,latdset.cdp.dods?lon,lat, , depth,_id,temp&latdepth,_id,temp&lat>0&lat<20>0&lat<20
2.2. Dapper contacts CDP via CORBA IIOPDapper contacts CDP via CORBA IIOP
3.3. CDP accesses MySQL for metadata, appropriate profile files for CDP accesses MySQL for metadata, appropriate profile files for datadata
4.4. CDP returns data to Dapper via CDP APICDP returns data to Dapper via CDP API
5.5. Dapper returns to client as OPeNDAP sequenceDapper returns to client as OPeNDAP sequence
![Page 13: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/13.jpg)
1313
CDP service performanceCDP service performance
< 500 us/profile for coordinate info< 500 us/profile for coordinate info5 ms/profile to read one variable with z 5 ms/profile to read one variable with z axis containing 1000 ptsaxis containing 1000 ptsnetCDF library enhancements:netCDF library enhancements:– Parsing of netCDF header/attributes is Parsing of netCDF header/attributes is
optionaloptional– Use String(char[]) constructor instead of Use String(char[]) constructor instead of
String(byte[]) constructorString(byte[]) constructor2X speed up2X speed up
![Page 14: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/14.jpg)
1414
netCDF servicenetCDF service
Individual profile files available as Individual profile files available as OPeNDAP Grid datatypeOPeNDAP Grid datatype
netCDF library enhancements:netCDF library enhancements:– True streaming serverTrue streaming server
![Page 15: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/15.jpg)
1515
Future workFuture work
Stream data from CDPStream data from CDP
Distribute netCDF file processing using Distribute netCDF file processing using scatter/gather patternsscatter/gather patterns
![Page 16: 1 DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA](https://reader036.vdocuments.us/reader036/viewer/2022083005/56649f295503460f94c42a21/html5/thumbnails/16.jpg)
1616
AvailabilityAvailability
http://www.epic.noaa.gov/dapperhttp://www.epic.noaa.gov/dapper
[email protected]@noaa.gov