7 nov 20072007 geospatial interoperability summit iowa environmental mesonet: using open source gis...
TRANSCRIPT
![Page 1: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/1.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Iowa Environmental Mesonet: Using Open Source GIS Tools and Web
Services to Disseminate Environmental Data
Daryl HerzmannIowa StateUniversity
![Page 2: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/2.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Data, data, everywhere data
![Page 3: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/3.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 4: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/4.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
The Desperate Challenge
• Disparate Formats– Grib– METAR– Text CSV– Free form text– ADAS III– SHEF– Excel (gasp!)
• Disparate Network Protocols– HTTP– FTP– LDM IDD (tcp)– TCP stream– Modems
The data comprising the IEM is made available to us in the following formats and network protocols.
![Page 5: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/5.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Why an Open Source solution?
1. They are the best tools out there. UNIX mentality -> Do something small and do it well.
2. Transparency. We are a public institution doing public funded research.
3. Excellent Support• Public bug trackers• Community forums• IRC• Motivated developers
![Page 6: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/6.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Open Source SolutionPostGIS / PostgreSQLSpatial Database
Map ServerMapping Engine +Scripting
Red Hat Linux Operating System
PythonData processing andnetwork services
GDALSwiss Army Knifeof all things GIS
PHPDynamic Website
![Page 7: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/7.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Key Component: PostGIS
• Early IEM code (2001) was a series of flat files in Shapefile format and database of observations. This was a disaster.
• PostGIS provided the opportunity to manage data, metadata, and the GIS data in one location.– Network scalability– Transaction / stored procedure support– Speed / functionality with merge of GEOS library
![Page 8: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/8.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Look at all of my data!
• 1 billion observations• 1 terabyte of data
– Archives of NEXRAD, GOES Satellite, precipitation datasets, web cameras!, Weather Service products, …….
• Now what? Swim in all of my data?
![Page 9: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/9.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Here’s an idea, lets create a webpage where they can download the data!
• Daryl:– Creates webpage
• End user:– Finds my page– Fills out form– Saves data, attempts to
figure out format– Attempts to find geo-
referencing for their GIS.– Hopefully loads data into
app– Commits numerous errors
along the way
![Page 10: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/10.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
The Open Geospatial Consortium (OGC) Way
![Page 11: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/11.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 12: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/12.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
In 2002, we built OGC services, but they didn’t come….
• Why?– Where were the cool
clients?– ESRI’s OGC support
wasn’t there yet– It all felt slow as
compared with local disk access
– A bit ahead of its time
![Page 13: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/13.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
In 2007, they have come!
• Thanks to cool clients– Google Maps– OpenLayers– ESRI Interop Extension– Map Server and others
• ~ 2 million hits per day (~20 “maps” per second)
![Page 14: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/14.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Supporting the Load
• 12 systems comprise IEM web farm
• Various levels of redundancy, but no spatial database redundancy yet
• Roughly 30 GB in , 60 GB out each day
• Linux IPVS provides redundant entry point and load balancing
![Page 15: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/15.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Available OGC Services
![Page 16: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/16.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
http://mesonet.agron.iastate.edu/ogc/
• CONUS GOES (Infrared, Visible)
• CONUS NEXRAD Base Reflectivity, Storm Total Precip, 1 Hour Precip
• NWS issued advisories,watches, and warnings
• Iowa Rainfall Estimates
• Iowa Road Conditions
• IEM Data Services (temp, winds, etc)
![Page 17: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/17.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Raw NEXRAD Composites
Have a large amount of noise due to ground clutter. A simple GDAL scriptcomparing radar echos to its elevation quickly gets rid of most noise.
![Page 18: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/18.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Cleaned Composite
![Page 19: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/19.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Some IEM developed GIS webapps
![Page 20: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/20.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 21: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/21.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 22: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/22.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 23: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/23.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 24: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/24.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 25: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/25.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 26: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/26.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
4 May 2003, heavy rains on freshly tilled soils
25 Sep 2005, heavy rains on mature crops
![Page 27: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/27.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
GIS + webcams
IEM developed software to have the cameras automatically track the sun and the moon. Can track NEXRAD identified storms as well!Done using FOSS GIS tools.
![Page 28: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/28.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
What others have done.
![Page 29: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/29.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
![Page 30: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/30.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Lessons Learned
• Adopting OGC services can greatly reduce the complexity of serving out your data and using your own data!
• Performance hasn’t been an issue, most of the overhead is with HTTP.
• For Map Server, try to limit the number of mapfiles you maintain!
![Page 31: 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS Tools and Web Services to Disseminate Environmental](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649e2d5503460f94b1c606/html5/thumbnails/31.jpg)
7 Nov 2007 2007 Geospatial Interoperability Summit
Future Work
• Increase Metadata!
• Help support NetCDF in GDAL (?)
• Work with end users on getting IEM data into their platform.