sdi tecnological components and standardssiteresources.worldbank.org/intlacregtopurbdev/i… ·...
TRANSCRIPT
Contents
• The SDI architecture model
• Software components: the SDI stack
• Basic services
• Metadata
• SDI access control
• Current trends in SDI technology
SDI Architecture Model
• SDI uses a three-tier service-oriented model
Source: mapit.biz
Standardized
SDI Architecture Model
Same thing with a component-oriented view:
Clients
Middleware
Servers Features Coverages
Metadata update Catalogs
Geo-processing and catalog Services
Content Repositories
Other data
e.g., administrative, statistical, env. reporting
Service chaining: search, display,
access, e-commerce, ….
Direct data access
Access to transformed data, pictures, maps, reports,
multi-media content
Metadata search and retrieval for data and services
SIG
SDI Architecture Model
• The use of a communication layer based on standardized Web services ensures:
– Transparent access to data, regardless of format, technology… but we still need to harmonize the data content
– Interoperability between different brands of servers and clients
– Distributability of components in a network. The SDI philosophy is that each responsible organization maintains and publishes its datasets. Multiple clients, catalogs, etc. are possible
Software components: SDI stack
• Spatial DB = Relational DB + … – Geometry representation. Standard is SFS (Simple Feature
Specification)
– Spatial Index. Speeds up search and operators, sorting objects by location
– Spatial operators. Can be invoked in SQL sentences to measure, perform geometric operations (e.g. intersection, reprojection), create new objects, etc. Essential for data quality assurance.
• Needed for large datasets, many simultaneous accesses (e.g. concurrent editing)
• Relational databases also used for metadata
• Files still used for raster data (images, elevation grids)
SDI stack: Spatial Databases
• Example of spatial table
SDI stack: Spatial Databases
Encoded geometry field
• Simple Feature Specification (OGC) for SQL
SDI stack: Spatial Databases
• Well-Known Binary representation for simple features (OGC)
SDI stack: Spatial Databases
• Open-source (OS) implementations – SpatiaLite (extension to sqLite), supported by
qGIS
– MySQL spatial extensions (not fully standard)
– PostGIS (extension to PostgreSQL) • Excellent support for standards
• Widely supported in OS GIS, ArcGIS, Intergraph…
• Used by GeoNode
• Friendly administration tools (pgAdmin)
• Interesting extensions (pgRoute for routing)
The whole UK Ordnance Survey is stored in PostGIS (£1,000,000 savings compared to Oracle Spatial), as well as the national topographic data of France for the Institut Géographique National
SDI stack: Spatial Databases
• Proprietary implementations – Oracle Spatial
– MS SQL Server 2008 and later
– ArcSDE (now ArcGIS Server Basic) • Wraps spatial or non-spatial DBs for
use in ESRI software, getting good performance
• Some support in OS GIS, but…
• In general, not useful if you need to use non-ESRI software with the DB
SDI stack: Spatial Databases
SDI stack: GeoServers
• A variety of OS and proprietary technologies support OGC standards for geoservices
SDI stack: GeoServers
• MapServer (OS). Excellent performance, quality and symbology for Map services (image)
• GeoServer (OS). Excellent performance and support for Feature services (data), great integration with PostGIS, even for remote editing. Easy administration.
– Used in GeoNode
SDI stack: Catalog Servers
• Geonetwork (OS)
– Reference OGC implementation for catalog service
– Includes powerful Web metadata editing and administration
– Extensively used in catalog networks (UN, many SDIs)
– Used in GeoNode
• ESRI’s Geoportal Server – Free and Open source, but…
– Meant to be used with ArcGIS Online and ArcGIS for Server
SDI stack: Catalog Servers
• Intergraph’s SDI Portal – Geospatial Portal – Bundles OpenLayers with GeoExt UI to provide ready-to-
use components
SDI stack: Web clients
• Usually combine:
– A live-map component (map viewer) which can:
• Display data from standard Web geoservices
• Navigate, query data…
• Sometimes provide data editing
– Web user-interface UI technology for legend, search, symbology, tools, etc.
• Today these are based on AJAX, Flex, Silverlight or other highly interactive framework for Web apps
SDI stack: Web clients
• OpenLayers (OS) – Most used map viewer
– Supports Google Maps and other map providers
– Excellent performance, using tiled images
– Supports feature data, editing with WFS-T
– Used in GeoNode
SDI stack: Web clients
• GeoExt (OS) – Based on JavaScript toolkit ExtJS, applied to Web geoportals
– Highly interactive table of contents, menus, tools, dialogs, etc.
– Integrates OpenLayers. It easily extended and customized
– Used in GeoNode
SDI stack: Web clients
• ESRI Web Mapping API – Bundles ESRI’s map components with JavaScript, Flex or
Silverlight UI. Includes Web editing functionality.
SDI stack: Web clients
• GIS (Geographical Information Systems) – Heavy-duty data analysis and processing are still run in
desktop applications
– All of them support OGC standards, so they can use Web geoservices and catalogs as source for data (but compliance and compatibility is not always good)
– The best practice is not to download data locally, but work with remote data that is official and up-to-date
– The SDI model, together with GIS, can also be used to implement information systems within organizations, sharing common data and metadata
SDI stack: desktop clients
SDI stack: desktop clients
qGIS (FOSS) – We’ll use this one in the workshop
SDI stack: desktop clients
uDIG (FOSS)
SDI stack: desktop clients ArcGIS
SDI stack: desktop clients GeoMedia
Basic SDI Services
• Web Map Service (WMS) – Get maps as images
– Get attribute information about features in a specific map location
• Web Feature Service (WFS) – Get vector data with geometry and attributes based on filters. Not meant
for large data downloads!
– Remote feature editing (WFS-T)
• Web Coverage Service (WCS) – Similar to WFS, for raster data (multiband images, elevation grids - DEMs)
• Catalog Service for Web (CSW) – Search and read metadata
– Harvest metadata to other catalogs
– Remotely edit medatada
Basic SDI Services
• Web Map Service (WMS) requests:
– GetCapabilities: returns the description of the service (layers, styles, coordinate systems…) as XML text
http://neowms.sci.gsfc.nasa.gov/wms/wms?version=1.1.1&service=WMS&request=GetCapabilities
http://www.snitcr.org/cgi-bin/wms?map=snit.map&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetCapabilities
– GetMap: returns an image (jpeg, png) of a map composed of one or more layers, in the requested coordinate system. The layers and style (legend) are defined in the server but can be selected and changed (the legend) in the request
http://neowms.sci.gsfc.nasa.gov/wms/wms?VERSION=1.1.1&REQUEST=GetMap&LAYERS=MOD14A1_E_FIRE&WIDTH=640&HEIGHT=320&FORMAT=image/png&SRS=EPSG:4326&BBOX=-180.0,-90.0,180.0,90.0
http://196.40.0.82/cgi-bin/mapserv?map=c:\ms4w\wms\biodiversidad_basp10v4geog50k.map&LAYERS=AREAS_SILVESTRES_PROTEGIDAS%2CETIQUETAS&TRANSPARENT=true&VERSION=1.1.1&FORMAT=image%2Fpng&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS=EPSG%3A97057&BBOX=247206.87468627,1128978.9530588,427828.99937255,1309601.0777451&WIDTH=256&HEIGHT=256
Basic SDI Services
• Web Map Service (WMS) requests:
– GetFeatureInfo request: returns the attributes of one or more
objects of a specified layer at a certain location in the map. Meant as a ‘click-on’ query on a map viewer
http://demo.opengeo.org/geoserver/wms?LAYERS=topp:tasmania_water_bodies&QUERY_LAYERS=topp:tasmania_water_bodies&STYLES=&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetFeatureInfo&BBOX=141.5115,-43.933,150.8015,-39.288&FEATURE_COUNT=10&HEIGHT=256&WIDTH=512&FORMAT=image/gif&INFO_FORMAT=text/html&SRS=EPSG:4326&X=258&Y=188
The output format can be changed (plain text, XML…) to process it in our client application.
Basic SDI Services
• WMS GetFeatureInfo Exercise
– Try it live at http://openlayers.org/dev/examples/getfeatureinfo-control.html
– Use your Web Browser to spy on network requests and responses • Chrome: Tools /Developer Tools / Network tab
• Firefox: Tools / Web Developer / Web Console
Basic SDI Services
• Web Feature Service (WFS) requests:
– GetCapabilities: returns the description of the service (layers, filter
types, coordinate systems…) as XML text http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetCapabilities
– DescribeFeatureType: returns a description of the feature for a
specified layer http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=DescribeFeatureType&typeName=usa:states
Basic SDI Services
• Web Feature Service (WFS) requests: – GetFeature: returns the features that comply with a certain condition
(filter), based on location or attribute values (it is not meant for massive data dowload!)
http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetFeature&typeName=usa:states&maxFeatures=10&filter=
<ogc:Filter xmlns:ogc="http://www.opengis.net/ogc">
<ogc:BBOX>
<ogc:PropertyName>the_geom</ogc:PropertyName>
<gml:Envelope xmlns:gml="http://www.opengis.net/gml" srsName="EPSG:4326">
<gml:lowerCorner>-89.92773475911 32.386718678814</gml:lowerCorner>
<gml:upperCorner>-89.04882850911 33.265624928814</gml:upperCorner>
</gml:Envelope>
</ogc:BBOX>
</ogc:Filter>
Basic SDI Services
GetFeature response in GML format
<?xml version="1.0" encoding="UTF-8"?> <wfs:FeatureCollection numberOfFeatures="1"> <gml:featureMembers> <usa:states gml:id="states.20"> <usa:the_geom> <gml:MultiSurface srsDimension="2"> <gml:surfaceMember> <gml:Polygon> <gml:exterior> <gml:LinearRing> <gml:posList> -81.759758 33.195232000000004 ... </gml:posList> </gml:LinearRing> </gml:exterior> </gml:Polygon> </gml:surfaceMember> </gml:MultiSurface> </usa:the_geom> <usa:STATE_NAME>South Carolina</usa:STATE_NAME> <usa:PERSONS>3486703.0</usa:PERSONS> </usa:states> </gml:featureMembers> </wfs:FeatureCollection>
Test this live at http://openlayers.org/dev/examples/getfeature-wfs.html
Basic SDI Services
• Web Feature Service (WFS) requests for remote editing (WFS-T): – LockFeature: prevents other clients from editing selected features
while we work at them
– Transaction: creates, modifies or deletes features. Each transaction
consists Insert, Update, and Delete elements, performed in order
• Not all servers or clients implement these requests!
Basic SDI Services
• Web Coverage Service (WCS): – Similar to WMS but returns raster data with multiple band
information, elevation values, etc.
• Catalog Service for the Web (CSW): – Provides remote access to a DB of metadata records
(catalog) • DescribeRecord - returns info about the format of metadata records
• GetRecords - search for records, returning record IDs
• GetRecordsById - returns records, specified by their ID
• Harvest (optional) - create/update metadata by asking the server to 'pull' metadata from somewhere (other CSW server, a Web folder…)
• Transaction (optional) - create/edit metadata by 'pushing' the metadata to the server
Basic SDI Services
• Catalog Service for the Web (CSW): – GetCapabilities and DescribeRecord examples:
http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=GetCapabilities
http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=DescribeRecord
– GetRecords example:
http://www.fao.org/geonetwork/srv/en/csw?REQUEST=GetRecords&SERVICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/2005/gmd&CONSTRAINTLANGUAGE=CQL_TEXT&RESULTTYPE=results&TYPENAMES=csw:Record
A constraint or filter is usually added to get only the metadata that we are searching (for instance, containing certain words, or for a certain geographic area)
The CSW service, through the Harvest request, allows the federation and integration of metadata catalogs, automatically collecting metadata from some catalogs to another (for instance, to specific organizations to a national catalog)
Basic SDI Services
Metadata DB
Metadata DB
National Catalog Server
Web or desktop client
Local/entity Catalog Server
Harvest
Metadata editor
Metadata File Metadata Web Service Metadata
Geoportal, GIS client
Main Catalog
Entity metadata
Search
CSW request
http://catalog.smi-q.gob.ec
CSW Result
Metadata
Entity metadata
Metadata
Entity data
Archivo
File
http://server.dep.gob.ec/files/data.dat
Viewer Add WMS layer
Geodata
Servidor entidad
WMS request
http://server.dep.gob.ec/maps?map1
WMS result (image)
Add WFS layer
Query
WFS request
http://server.dep.gob.ec/wfs?layer1
WFS result (data)
Data visualization
Data download
Basic SDI Services How they all work together… (Animation)
Other Basic SDI ‘Services’
• Direct file download – Use it for geodata download instead of WFS or WCS, which should be
only for limited query
– Use it for documents and other files that cannot be accessed through standard Web services
– The download address can be written in the metadata, as we do for Web services
• News and alerts (RSS, GeoRSS, Atom…) – Conventional Web clients can get updated messages and news
– Users can subscribe to receive them in a variety of readers
What is "Metadata"?
• Metadata is a formal data documentation and it is critical to data discovery
• A metadata record is a set of information fields, usually presented as an XML document, which captures the basic characteristics of data as an information resource
• It represents the who, what, when, where, why and how of the resource
Metadata as seen in Geonetwork
What is "Metadata"?
What is "Metadata"?
Metadata as seen in ESRI’s ArcCatalog
Biodiversity & Environmental Resource Data System of Belize http://www.biodiversity.bz
Main uses of Metadata
• Organize and maintain an organization's internal investment in spatial data
• Provide information about an organization's data holdings to data catalogs, clearinghouses, and portals (it is the entry point of an SDI and for each SDI node)
• Provide information to process and interpret data received through a transfer from an external source
Value of Metadata
• Facilitates data maintenance
• Facilitate data discovery
• Enables reuse of data
• Informs potential users of inappropriate uses of a dataset
Challenges of using Metadata
• Exchanging records -> Use standards!
• Creating metadata takes time and patience -> Use templates and tools that automatically extract or set metadata (like GeoNode). Imperfect metadata is better than none
• Maintaining metadata is essential, especially for Web services and downloads (beware of URL changes!) -> set up processes or tools for automatic checkups
Metadata standards
• Some standards define the metadata content (fields, values, mandatory v. optional) – The most comprehensive is ISO 19115
• From this standards, custom profiles can be defined for specific countries, themes, etc.
• ‘Old’ metadata standards like FGDC or Dublin Core are now profiles of ISO 19115
• Other standards define appropriate exchange formats – The most accepted is ISO 19139, which defines an XML-
based format
Metadata content: Identification
• Important for data search
Metadata content: Identification
• Information in abstract: data model
– Entity types and their attributes and the domains from which attribute values may be assigned
– The names and definitions of features, attributes, and attribute values
Metadata content: Identification
Metadata content: Distribution information
• How to access the information?
• Multiple transfer options are possible
• Contact for physical media
• Fees
Metadata content: Data quality
• Positional and attribute accuracy • Completeness and consistency, • Sources of information • Methods used to produce the data • Appropriate uses (scales)
Metadata Content: Spatial Reference
Metadata content: Metadata information
• Language, date, update status
• Who is responsible for the metadata?
• Use constraint statements can be crafted to express scale, geographic, or temporal limitations to the data
• Liability statements should be written by legal staff to ensure that the legal requirements for use of the data are fully outlined
• In general, it is far better to publish your dataset with limitations within your metadata than to later attempt to generate them in response to an inquiry or lawsuit.
Metadata content: Access and use constraints
Access and use control
• Use constraints and disclaimers
– Web clients typically include liability limitations (disclaimers) and licenses that users must accept
– But geoservices can be accessed directly!
• Therefore, use constraints must be specified in the metadata and GetCapabilities response
• And services must be protected from non-authorized access
– Within an organization: generic IP filtering and firewalls
– External services: need an authentication mechanism
Access control for geoservices
• There are generic mechanisms for Service-Oriented Architectures
– Based on XML standards like SAML and XACML
– However, these standards are not easily used in SDIs because • Current geoservices do not
use SOAP for passing information
• Server and client software should be modified to comply
Access control for geoservices • Specific SDI
solutions have been proposed and used
• Existing clients and server don’t have to be modified, only ‘intercepting’ components need to be added
• Access can be controlled for specific layers, scales, features and attributes
See http://www.eurogeographics.org/sites/default/files/070504_SecurityInSDI_conterra_V2.pdf http://52north.org/communities/security/
Access control for geoservices
• In practice, however, most systems use simple HTTP authentication that provides a simple on/off swith for services depending on the user credentials
• Web browsers have a built-in implementation to ask for user and password
Current trends in SDI technology
• Gazetteer services (search location by name), based on WFS-G profile
Current trends in SDI technology
• Web Processing Service (WPS). Standard for discovering and calling remote processing modules – Used for typical GIS analysis like spatial analysis, running
distributed models
– Used for operations that are not easy to standardize in a specific Web service • Spatial Coordinates transformation
• Geocoding (address finding)
See, for instance: http://www.cartociudad.es/wps/WebProcessingService?service=WPS&request=GetCapabilities
Current trends in SDI technology Tiled maps
Cached and tiled Map Services
(WMTS, WMS-C…)
• Static maps can be pre-calculated and tiled at different scales, like a raster pyramid – Tiles can be directly server through tile services like WMTS,
WMS-C, Google Earth…
– Pre-calculated tiles can be used to quickly respond to conventional WMS requests
– An example is the GeoWebCache technology, used by GeoNode, which can be integrated with regular WMS servers like GeoServer
Current trends in SDI technology Tiled maps
Current trends in SDI technology Tiled maps
Current trends in SDI technology Tiled maps
• A problem with tiled map providers like Google, Bind, OpenStreetMap, is that they only provide tiles with a specific ad-hoc coordinate system (used to be EPSG:900913, now it is officially EPSG:3857)
• It is difficult to overlay this data layer on other coordinate systems
Remote on-line editing with WFS-T
Current trends in SDI technology
Remote collaborative editing
(OpenStreetMap)
Current trends in SDI technology
Remote geoprocesses (coordinate transformation, geocoding) with WPS
Current trends in SDI technology
Real-time data integration (RSS,
Common Alert Protocol)
Current trends in SDI technology
Standard Web services for reading and controling remote
sensors (SWE, SOS…)
Current trends in SDI technology
Integration with SOLAP data mining
Current trends in SDI technology
SDI Web services in the cloud
Current trends in SDI technology
New data sources (LiDAR, georeferenced Video &
street-level images)
Current trends in SDI technology