workshop humboldt data harmonisation tools in action · introduction to the workshop and the...

0

© 2009 Humboldt Consortium http://www.esdi-humboldt.eu

Workshop

HUMBOLDT Data Harmonisation Tools in Action15 June 2009, 14:00 – 15:30

1


Workshop Agenda

Introduction to the Workshop and the HUMBOLDT Project

Data Harmonisation Issues and Requirements

Solutions from the HUMBOLDT Project

Demonstration of HUMBOLDT Framework Tools

Outlook

Discussion

2


Speakers

Marian de Vries (Delft University of Technology)

Thorsten Reitz (Fraunhofer Institute for Computer Graphics)

Astrid Fichtinger (Technische Universität München)

Eva Klien (Fraunhofer Institute for Computer Graphics)

Emanuele Roccatagliata (GISIG-Geographical Information Systems International Group)

Sisi Zlatanova (Delft University of Technology)

3


The HUMBOLDT Project

Marian de Vries

4


The HUMBOLDT Project (IP in FP6, 10/2006 to 09/2010)(Some of the) Objectives

Overcome barriers in multidisciplinary and

cross-border usage of geoinformation

Common tools and services to support

transformation needs

Integration of these tools and services into

existing SDI

Support for INSPIRE (tools for data

harmonisation) and GMES (theme-specific

services in scenarios)

Users’ perspective on the SDIFrom a data-centric view to a usage centric view

5


HUMBOLDT – Vision

Format

Spatial reference

Graphical form

Object generation

Language

Object semantics

Expected quality

Provision of information in

the language area / semantics of the

application domain

Information directly useable

Quality information / information about the

level of goal achievement

Formulation of the demand

Demand-orientedinformation

a

Data Domain X

“creating the possibility to combine data from heterogeneous

sources into integrated, consistent and unambiguous

information products, in a way that is of no concern to the

end-user” (A 3.5-D1) but should be traceable

7


HUMBOLDT – PartnersFraunhofer IGD (D) (Coordinator)ETRA (ES)Help Service Remote Sensing (CZ)LogicaCMG (UK)Institut Geographique National (F)Intergraph CS (CZ)ETH Zürich (CH)Delft University of Technology (NL)University of Rome „La Sapienza“ (I)Institute of Geodesy, Cartography and Remote Sensing (HU)Marine Information Service (NL)Kaunas University of Technology Regional Business Incubator, (LT)INI-GraphicsNet Stiftung (D)Technische Universität München (D)University of the West of England (UK)Institut Français de Recherche pour l'Exploitation de la Mer (F)National Environment Research Council (UK)Hellenic Centre for Marine Research (GR)Swedish Meteorological and Hydrological Institute (S)Telespazio (I)Geographical Information Systems International Group (I)Consiglio Nazionale delle Ricerche (I)Forest Management Institute (CZ)Instituto Geografico Portuges (P)Collecte Localisation Satellites (F)University of Gävle (S)Intergraph Deutschland (DE)

27 partners from14 countries

Quelle: HUMBOLDT2007

8


Data Harmonisation Issues

Marian de Vries

9

Combining data into integrated information

Integrated View

Data -> Maps

Topography

Transport

Elevation

Hydro

Soils, Vegetation

Weather

RiversRoadsImagesSoils, Vegetation

...

CatalogView

Framework(Clearinghouse)

10


Example: Tourism valorisation in a Protected Area

Application Process for setting up sustainable hiking paths with high interest

Requires harmonised data about protected sites

Source data sets are cross-border…

…and cross-theme

11


HUMBOLDT: + computational models (process models)

INSPIRE data interoperability components

(INSPIRE 2008)

12


Data Harmonisation Issues in HUMBOLDT Scenarios (I)

data formats

coordinate reference systems

conceptual schemas (data models)

classification schemes

scales / resolutions / level-of-detail

metadata profiles

natural languages

multiple representation of the ‘same’ spatial objects

spatial consistency issues at the border (edge-matching etc.)

13


Data Harmonisation Issues in HUMBOLDT Scenarios (II)

Different CRS and Scales in the Lake Constance RegionBaden-WuerttembergBavariaVorarlbergSwitzerland

DHDN/GK Zone 31:25:000

DHDN/GK Zone 41:25:000

MGI (Ferro) Austria GK West Zone1:50:000

CH 1903 LV031:25:000

Data sources:© Bayerische Vermessungsverwaltung© Landesamt für Geoinformation und Landentwicklung Baden-Wüttemberg© Land Vorarlberg© swisstopo

14


Data Harmonisation Issues in HUMBOLDT Scenarios (III)

Multiple representation (detail of image on previousslide)

River Leiblach

15


Data Harmonisation Issues in HUMBOLDT Scenarios (IV)

Spatial consistency:A protected area in Portugal (shapefile light green) and a protected area in Spain (wms dark green) that share a border

16


Data Harmonisation Issues in HUMBOLDT Scenarios (V)

Spatial consistency: A protected area in Portugal (shapefile dark green) and administrative boundaries in Spain (wms black and grey)

17


Data Harmonisation Issues in HUMBOLDT Scenarios (VI)

Spatial consistency: A road in Portugal (shapefile red line) and a road in Spain (wms green line)

18

Data Harmonisation Issues in HUMBOLDT Scenarios (VII)

«featureType»FloodArea

+ id: String+ geometry: GM_Surface+ gaugeId: String+ methodOfCalculation: MethodOfCalculationType+ uncertainty: Float+ waterLevel: Decimal

«featureType»Gauge

+ id: String+ geometry: GM_Point+ gaugeID: String+ nameOfRiver: String+ spotElevation: Integer+ managingAuthority: CI_Citation+ meanDischarge: Float

«featureType»WaterLev elMeasurement

+ id: String+ timestamp: DateTime+ gaugeId: String+ waterLevelValue: Float+ statisticAnnuality: StatisticAnnualityType

«featureType»DischargeMeasurement

+ id: String+ timestamp: DateTime+ gaugeId: String+ statisticAnnuality: StatisticAnnualityType+ currentDischarge: float

«featureType»Watercourse

«voidable»+ condition: ConditionType [0..1]+ fictitious: Boolean+ length: Length+ level: LocationType+ tidal: Boolean+ waterCourseHierarchy: WatercourseHierarchyType [0..1]+ width: WidthRangeType

«featureType»SurfaceWater

+ geometry: GM_Primitive+ id: Identifier+ levelOfDetail: IndicativeLoD [0..1]+ localType: LocalisedCharacterString [0..1]

«voidable»+ beginLifespanVersion: DateTime+ endLifespanVersion: DateTime [0..1]+ geographicalName: GeographicalName [1..*]+ hydroId: HydroIdentifier [0..1]+ origin: OriginType+ persistence: HydrologicalPersistenceType+ managingAuthority: CI_Citation

«featureType»StandingW ater

«voidable»+ elevation: Length+ meanDepth: Length+ surfaceArea: Area

0..*

10..*0..*

0..*

1

0..*

1

0..*

1

0..*1

Conceptual Schemas(clip fromERiskA)

19


State-of-the-art analysis of existing tools

No (open source) tool that supports all steps in a harmonisation workflow

Existing tools can often not be integrated easily in a service-oriented architecture or framework

Tools do not fully support testing whether a model or data set is consistent and correct

HUMBOLDT framework software addresses these gaps

Is open source

20


The HUMBOLDT Technology Framework for Data HarmonisationThorsten Reitz

21


Functionalities for covering the data harmonisation process as a whole

Analysis of sources

Creation of target

Input for transformation (mapping rules)

Source TargetSchema Transf.

Edge Matching …

Executed technical processes

User knowledge

22



Capturing domain knowledge / application-specific knowledge supportingthe definition of the information product (target schema, SRS, spatial extent, …) to which the processed data needs to be transformed

the identification of transformation needs

enhanced formalization of the transformation between two data schemas

Handling of transformation needs as part of the overall processing of an information request

Enhancement of the automation of the data harmonisation processes (depends on the possibility of capturing the required knowledge and the availability of this knowledge in the system)

23


Process Step: Creation of Target

Creation of target

Target

User knowledge

Defining the information product (target schema, SRS, spatial extent, …) to which the processed data needs to be transformed

24


The HUMBOLDT GeoModel Editor

Developed to support the process of data specificationMain goals:

Easy-to-use for application experts

Collecting all required information on the data

Providing formal representation of the information in order to feed it into a harmonisation process

Implemented on a framework (Eclipse)

Usage of a “spatial UML” (HUMBOLDT modelling language)Producing/providing a graphical and a textual representation of the data model

basic data model containing basic spatial data types

Support of vertical mapping: serialisation possible to transfer standards or other representation (e.g. XMI, GML, ISO19131, ...)

25


The HUMBOLDT GeoModel Editor

26


Schema Mapping

Astrid Fichtinger

27


Process step: Schema to Schema Mapping

Source Target


User knowledge

?

28


Schema mapping operations

1. Filtering: conditional statements applied to source data to filter features (extract sub-sets)

Source Schema (VA)Class „Gewaesser“

Att. „LEVEL“, values (selected)Definition ValueFluss, Bach ( > 20 m) (river, rivulet) 1

Fluss, Bach (5-20 m) 2

Fluss, Bach ( < 5 m) 3

Wasserleitung (unterirdisch) (pipe) 5

See, Teich (> 10 km) (lake, pond) 11

Sumpf, Moorboden (marsh, bog) 19

... ...

Class „Watercourse“

Target Schema (INSPIRE HY)

Class „Pipe“


Class „StandingWater“


Class „Wetland“


29



2. Reclassification of attribute values

Source Schema (BY)Class „Gewaesser“

Att. „Widmung“, values (selected)Definition Value1. Ordnung (Bundeswasserstraße)(1st order under federal law)

1501

1. Ordnung (Landesgewässer)(1st order under state law)

1502

Gewässer 2. Ordnung (2nd order) 1503

Gewässer 3. Ordnung (2nd order) 1504



Att. „waterCourseHierarchy “, values1st

2nd

3rd

4th

5th

other

30



3. Renaming of feature classes or attributes

4. Merge / split of features or attributes

Source Schema (VA)Class „Gewaesser“

Att. „NAME“ (string)



Att. „geographicalName/spelling/text“(CharacterString)

Source Schema (XY)Class „Building“

Att. „NameOfStreet“ (string)

Att. „HouseNr“ (string)

Att. „City“ (string)

Target Schema (XY)

Class „Building“

Att. „Address“ (string)

5. Change of attribute order

concatenation

31



6. Type conversions

Spatial types– Spaghetti lines to Network

– MultiLineString to LineString

– Line/Polygon to Point (centroid)

– Point/Line to Polygon (buffer)

– Polygon to Line (skeleton/centreline)

Other types– Integer to float

– String to numeric

Data source © Landesamt für Geoinformation und Landentwicklung Baden-Württemberg

Example from German watercourses:< 12 m wide: line> 12 m wide: polygon

Example from German watercourses:< 12 m wide: line> 12 m wide: polygon

32



7. Value conversions

Spatial conversions – Generalisation, e.g. simplification

– Conflation (but see: Merge)

Units of measurement

8. ‘Augmentation’Deriving values for target schema properties missing in source schema – based e.g. on values of other properties in source schema (ex. using interpolation)Adding derived spatial properties (e.g. centroid of a polygon)Filling in default property values in target schema

33


Schema mapping operations8. ‘Augmentation’

Source Schema (VA)Class „Gewaesser“Att. „LEVEL“, values (selected)Definition ValueFluss, Bach ( > 20 m) (river, rivulet) 1

Wasserleitung (oberirdisch) (pipe on /aboveground)

5

Wasserleitung (unterirdisch) (pipe below g.) 6

Gewässer unterirdisch (watercourse below g.) 7

See, Teich (Länge > 10 km) (lake, pond) 11

... ...

Target Schema (INSPIRE HY)Class „Watercourse“

Att. „origin“, valuesmanMade

natural

unknown

Target Schema (INSPIRE HY)Class „Watercourse“

Att. „level“, valuesonGround

aboveGround

belowGround

unknown

34


The Schema mapping language

RequirementsExpressive enough (must support transformation operations like renaming,

restructuring, reclassification, geometric & topological functions, etc.)

Implementation-neutral

Actual mapping code can be derived from it

Fitting the model-driven approach

Candidate: Ontology Mapping Language (OML)

35


Horizontal mappingHorizontal mappingSource Target

36


The HUMBOLDT Alignment Editor (HALE)

37

Improve tools for expert knowledge capture


Goal: Enable experts to create high-quality mappings of complexgeodata schemas

Task-based approach to support the domain expert in making the mapping process deterministic.

Continuous validation against a quality model

Continuous instance visualisation

Usage of conceptual spatial relations

Inclusion of irreconcilable mismatches into the final mapping

38

Continuous Validation against a quality model

Measuring the quality of a mapping requires the definition of a mapping quality model:Completeness Correctness

Sche

ma

Inst

ance

s

Completeness of mappings

over instances

Completeness of mappings

Completeness of functions

Completeness of functions

over instances

Consistency of Alignment over

Mappings

Consistency of Functions over

Instances

A significant subset of reference instances is continuously transformed according to thedefined mapping to monitor the quantified quality of the mapping.The end results of the quality analysis can become part of the mapping itself.


39

Instance visualisationDirect feedback of results of a mapping/transformation definition makes mismatches obviousto the expert that might otherwise not be detected.


40


The HUMBOLDT Alignment Editor (HALE)

HALE is a tool for geodomain application experts and GI experts, especially data custodians.

It provides means to create automatically executable horizontal mappingson the conceptual schema level and in addition in some aspects on the logical schema level.

It provides the means to ensure the validity of a mapping on the base of the source and target schema as well as on the base of the source and a reference data set.

It uses schemas created in the HUMBOLDT GeoModel Editor or other Modelling Tools.

It does not execute those mappings.

41

Demonstration


42

Result encoded as (g)OML XML file


43



Analysis of sources

Creation of target


Source TargetSchema Transf.

Edge Matching …

Executed technical processes

User knowledge

44


The HUMBOLDT Conceptual Schema Transformer (CST)

The CST is delivered as a Web Processing Service (WPS)

‘Executes’ the schema mappings made in HALE

Transformation of source data from source schema to target schema

Java platform

Makes use of GeoTools libraries

45


The HUMBOLDT Edge Matching Service (EMS)

46


The HUMBOLDT Edge Matching Service (EMS)

WPS implementation

Aligns LineStrings and Polygons of vector geometries so that they will be gapless and with no overlaps

Three modes of operation:

CleanDataset: when only one dateset is provided, then the gaps and overlaps will be removed.

Align-to-Reference: all candidate data sets will be transformed using the reference dataset so that points are moved up to a maximum distance also provided as input.

Distribute-Errors: In this case, there is no reference data set that can be used as "ground truth", therefore all geometries will be transformed. No point or edge will be moved further than a client-specified amount.

47


EMS Clean Dataset Process – WPS Client (uDIG)

48



49



50



51


Clean Dataset Process – WPS Client (uDIG)

52


EMS Clean Dataset Process – Results

Input:

53


EMS Clean Dataset – Results

Output:

54


EMS Clean Dataset – Results

Result:

55


Further HUMBOLDT Transformation Services

Coordinate Transformation ServiceImplemented and in use

Language Transformation ServiceEarly implementation

Multiple Representation Merging ServicePrototype implementation available

Quality Measurement in TransformationStatus: Specified

56


The HUMBOLDT Workflow Design and

Construction Service

57


WDCS

Analysis of input data sets to determine harmonisation requirements

Automated combination of processing services into executable workflows

Definition of “Basic Workflows” via Workflow Designer

58


Conclusion

Dr. Eva Klien

Fraunhofer IGD

TR

59


HUMBOLDT Project – The Wider Context

Technology FrameworkSupporting SDI enablement by providing the functionality for covering the

data harmonisation process as a whole

Offering the possibility to make use of single functionalities as part of

existing SDI What is Humboldt serving to the Community

60



Automated Execution of the

Processing Chain

Mediator Service

Result Presentation/Publishing/...

Non-HUMBOLDT

View and Download Services

Automated Discovery

Automated Analysis

Automated Processing Chain Setup

Information Grounding

ServiceHUMBOLDT Workflow Service

Known Dataset Manual AnalysisManual

Processing Chain Setup

Context Service (Defined Source-

Constraint)

HUMBOLDT Model Editor

Workflow Composer

61



Technology FrameworkSupporting SDI enablement by providing the functionality for covering the

data harmonisation process as a whole

Offering the possibility to make use of single functionalities as part of

existing SDI What is Humboldt serving to the Community

Collection of tools and services for harmonisation (and use) of spatial

data with a level of automation as high as possible

Provision as Open Source Software

62

Identification of HUMBOLDT products

There are three levels of software products defined and implemented:

1. The HUMBOLDT Framework: re-usable software components to

create geodata harmonisation applications DEVELOPERS, Open

Source

2. The HUMBOLDT Toolset: set of applications to support all activities

around data harmonisation process DATA INTEGRATORS, Open

Source

3. The HUMBOLDT Scenario Applications: Concrete End-User

applications using the framework components and (adopted) versions

of toolset applications END-USERS OF GEODATA


63


Contributions to the Community

Supporting SDI enablement

Providing functionalities for covering the data harmonisation process as a

whole

Serving the community dialogue with the Open Source Software development

and the HUMBOLDT User Community (http://www.community.esdi-

humboldt.eu / http://www.esdi-humboldt.eu/get_involved/user.html

Collaborations and NetworkingCollaboration with several projects from the GMES and eContentplus community (BOS4GMES, CASCADOSS, eSDI-Net+, ESDIN, …)Humboldt partners are strongly involved in the INSPIRE communityLink to the users community with HUMBOLDT Application Scenarios

http://www.community.esdi-humboldt.eu/



64

HUMBOLDT activities in relation to INSPIRE processes

HUMBOLDT

is a registered SDIC

participates regularly in consultation and review processes

participated in the testing phase of Annex I specifications

provides input (e.g. example data specifications) through scenarios

provides knowledge on data harmonisation processes


65


HUMBOLDT Open Source Software (in development status)http://www.community.esdi-humboldt.eu/

HUMBOLDT Model EditorFor the creation of UML application schemas

HUMBOLDT Alignment EditorDefinition of conteptual schema transformation

Workflow ServiceService Composition for handling transformation needs

Mediator ServiceExecution of transformation chains

Context ServiceDefinition of transformation products

Transformation ServiceExposed as Web Processing Services (e.g. CTS, Edge Matching)

We invite developers to test the design and

expandability of the framework.

Suggestions and contributions are highly

welcome

66


Thank you for your attention!

→ Discussion

67


Data harmonisation issues on two levels

At general level: Data model (conceptual schema)Spatial reference systemLevel-of-detail, scale / resolution(Data format)Terminology, semantics (meaning)Metadata profilePortrayal

At individual object level:

Edge matching

Solving conflation (doubles, etc.)

Other data quality issues

MdV

68

User Guidance & Task-based approach

In the goal of making an interactive mapping process more deterministic, theprocess is structured into tasks. Tasks have the following lifecycle:

A Task has the following elements:A Source: The manual or automatic process that generated the task.A Description: A textual description of the task.A Type: Creation/Modification/Removal of a mapping.A Value: Based on the frequency of the Type/ Attribute in the geographicinstance data.A Set of Elements (Types, Attributes, Mappings) that are connected to the task.

TR


69

Overall implementation architectureThe presented aspectsare implemented in theform of the HUMBOLDT ALignment Editor (HALE).As a basis, Eclipse RCP 3.4 is used. GeoTools2.5.5, Java Rules/Droolsand other libraries areused in support. Mappings arestored/exchanged usingthe OMWG OntologyMapping Language(OML)

TR


cmp HALE-Internal Components

HALE PluginUser Interface

Schema Explorer

Attribute View

Map View Table View

Function Wizard Quality & Tasks

RCP Alignment Platform

70


Protected Areas: The whole process…

TR

FrameworkHUMBOLDT Toolkit

Product Definition

Application Workflow Definition

Harmoni-sation

Processing

Framework

Application-Specific

Processing

Source Schema Creation

SchemaAlignment

Target Schema Creation

Application

Scenario

Application-Specific ProcessingHarmonisation processing

Harmoni-sation

Workflow Construction

Harmoni-sation Trans-

formation

Scenario Workflow Execution

Automated Data

Analysis

Result Rendering/Provision

Data Discovery

71


… and the adapted system for Protected Areas

TR

deployment Component Model

WFS Italy WFS Portugal

WPS Fraunhofer

PA Application Logica

Scale Harmonization WPS

Coordinate Transformation

WPS

Multiple Representations

WPS

OpenLayers Standard Client

Mediator Serv iceIGS + Workflow Serv ice

Context Serv ice

«use»

deployment Component Model

Desktop Toolkit

Toolkit

HALE GeoModel Editor

Model Repository

72

Different kinds of stakeholders

GIS (or geoscience) developer = developer of software, web services

Data custodian = responsible for providing (harmonised) data

Data integrator = works with (heterogeneous) geodata, integrates geodata for him/herself

or others in the organisation

End-user of geodata = works with already harmonised geodata (or geodata that doesn’t

need harmonisation or integration)

End-user of spatial information = use geo-information services (not dealing with geodata

itself)

73


Target DefinitionTarget Definition

TechnicalTechnical ProcessProcess

TargetTarget

Source

Processing

National Data Sourceson the same theme,e.g. parcels

Decision-basedspecifications

Technicalspecifications

Applications based ondifferent themes

INSPIRE HUMBOLDT Scenarios

HUMBOLDT Framework

Source

ProcessingSource

Processing

Data harmonisation processes

workshop humboldt data harmonisation tools in action · introduction to the workshop and the...

Documents