managing data interoperability with fme tony kent applications engineer imgs

Managing Data Interoperability with FME

Tony KentApplications Engineer

IMGS

IMGS

We deliver innovative spatial solutions

For the desktop, web and mobile

Built on our partner’s technology

Designed to meet the challenges of Government, Mapping Agencies, and Utility & Communications Customers

Safe Software

Powering the flow of spatial data with FME

3

Enabling people to use their spatial data where, when and how they want to

Most Used Spatial Interoperable Solution in Ireland

Why Spatial ETL?

Significant proliferation of different spatial data formats and types

Hundreds of formats, with more added each yearMultiple types of data stored in multiple systemsUnique data model requirements for each application

4

Why Spatial ETL?

Traditional approaches to data translation and data model manipulation are not viable

Complex, inefficient and time-consuming

5

Why Spatial ETL?Increasing pressure for access to spatial data

More users, beyond traditional GIS usersExpectations of real-time custom data views, 24x7

6

FME Capabilities

7

The only complete spatial ETL solution

Translate spatial data from one format to another

Transform spatial data into the precise data model you need

Integrate different data types into a single data model

Distribute spatial data to users where, when and how they need it

FME Desktop

Flexible and powerful spatial ETL toolset

Translate, transform and integrate data in hundreds of formats

Graphical authoring environment

8

Step 1 - Extract

Select and add the source dataset(s)

Step 2 - Transform

Add transformers to manipulate the

data as it moves from source to

destination Step 3 – Load

Load the transformed data into a

destination format and source

FME Workbench

9

Use simple point and click to easily define spatial data flows to translate, transform and integrate your data

Examples

Automating Ordnance Survey data updatesPushing NTF data to multiple GIS platforms

Stripping out unnecessary data

Adding custom styling and symbology – CAD

E.g. Eircom, ESB, Fingal County Council

Publishing data to internal public portals Bulk and transactional updates

Fire wall Friendly – use selected port

Completely automated

E.g. Dublin City Council

Open Data Challenge

You want to meet Open data requirements, but your data is organized rather differently

?

?

What FME does …

Build data bridges to your SDI

SDI Harmonization Core Concepts

Harmonization: implied requirement for building an SDI

Disparate sources must be mapped to a common destination data model

Core to the harmonization workflow is a process called schema mapping.

Delivered by services based on open standards

Harmonization Principles

Typical stages:

1. Evaluation

2. Assembly

3. Transformation

4. Validation

5. Publication

Based on the Spatial ETL concept (Extract, Transform and Load), as applied to INSPIRE SDI’s

Metadata – Data about data

Describes data structurestables

geometry types

data types

fields

Describes data contentcoordinate system

extent

modification date

quality, ownership, etc.

Metadata - Purpose

Key FME Metadata Capabilities

Reading

Writing

Updating

Harvesting

Validating

Integration with web services

Data Transformation - Schema

Reshape source data to match required destination schema

Schema mappingfeature type

attribute name

new attribute creation

code lists

conditional value

mappings

Feature Type Mapping in FME Workbench

Attribute Mapping in FME Workbench

Schema Mapping in FME

Value Mapping

FME Data Model Restructuring: Attribute Names & Values

FME SchemaMapper: INSPIRE geographic names

Name mapping

Name & value mapping

FME Workspace

Transformation: Geometry

Non-spatial to spatial

Geometry extraction (spatial to GML)

Representation transform: CAD drawing lines with labels to GIS polygonal features with attributes

Coordinate System Reprojection (ED50 to ETRF89)

Simple to complex geometrySource point and polygon data to multiple geometric representations (city as point / area, river as line / area)

Generalization and interpolationHighly granular national and regional datasets often require thinning to be usable on pan-European scales

Validation

Schema validation i.e. INSPIRE (xsds)

Data integrityUnique IDs

Geometric integrity (closed polygons)

Null values (nullable?)

Valid values: ranges and domain codes

Data gaps

Bounds

Network integrity

Custom validity rules specific to domain

Validation automation via FME Server upload

Ensure data quality throughout the data transformation process

Publish workspace to FME ServerStore the workspace in a central repository

Make your FME workspaces available to others –over the web

Register the workspace with one or more services (Data Streaming, Data Download, etc.)

Publication with FME Server

24

Format translation

Schema mapping

String and list manipulation

Data validation

Database load and extract

XML,GML,WFS: reading, validation, publication

Web services: WFS, WMS, integration with others

Metadata support

Enterprise services with FME Server

FME Tools for INSPIRE

FME can provide all the tools to help build support your data sharing needs:

Integrate your data sources

Manage your meta data catalogues

Transform your data to standard schemas

Publish the data in the required formats

Summary

Thank You

For more information:

Email: ckirk@imgs.ie or tkent@imgs.ie

Web: www.imgs.ie