inspire-hands_on-data_transformation
DESCRIPTION
The INSPIRE Implementing Rules (IRs) on interoperability of spatial data sets and services and for network services include requirements for setting up a Spatial Data Infrastructure in Europe for supporting environmental policy making as well as policies with impact on the environment. To help Data provider with technical aspects of the IRs as well as with its correct implementation, INSPIRE Technical Guidelines (TG) were developed for each 34 data themes (INSPIRE data specifications) and for the different types of INSPIRE network services (discovery, view, download and transformation). Spatial objects are mapped, digitalized and stored in a GIS data sets or (spatial) database. Normally, the structure of the data will depend on the specific needs for which the data are collected and used. In order to provide them in compliance with INSPIRE, these source data sets have to be transformed to match the data model prescribed by INSPIRE and have to be provided through INSPIRE download services. This training will show and illustrate through "hands on" exercises how data sets can be transformed and provided through INSPIRE-compliant services by covering the following topics: 1) Data transformation: This session gives an introduction and explanations about encoding rules, mapping original attributes into the INSPIRE data models and vocabularies and extending data models and vocabularies. 2) Download services: This session will explore the procedure of providing transformed dataset into through an INSPIRE network service, e.g. through an WMS (for view services) or WFS or ATOM feeds (download services). 3) "Hands on" session: This session will give an overview of different architectural approaches (e.g. on-the-fly transformation and stand-alone offline transformation) and concrete software solutions for transforming spatial data and creating INSPIRE-compliant services.TRANSCRIPT
INSPIRE Hands on Data transformation amp download services
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert et al
2 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Objectives
bull to raise awareness about the technical application transformation of
data sets in order to ensure compliance with the INSPIRE Directive
bull you will get a feeling and basics what a transformation process for
data sets into data services means how to apply datasets INSPIRE
compliant
bull Learning outcomes
bull how to understand the INSPIRE DS to use xml schemas to deal
with controlled vocabularies
bull how to transform datasets to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
bull hellip
3 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Whatrsquos going on today
bull Intro to INSPIRE data specs (retrospection)
bull Relationship between source data and INSPIRE themes amp features
bull Exercise mapping data into INSPIRE requirements
bull Transformation Configure mappings amp validation
bull Continue transformation amp exercises
bull Publish Data GeoServer
bull Discussion
INSPIRE Hands on Introduction - Just as a quick reminder
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
6 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
what it is about hellip
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
2 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Objectives
bull to raise awareness about the technical application transformation of
data sets in order to ensure compliance with the INSPIRE Directive
bull you will get a feeling and basics what a transformation process for
data sets into data services means how to apply datasets INSPIRE
compliant
bull Learning outcomes
bull how to understand the INSPIRE DS to use xml schemas to deal
with controlled vocabularies
bull how to transform datasets to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
bull hellip
3 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Whatrsquos going on today
bull Intro to INSPIRE data specs (retrospection)
bull Relationship between source data and INSPIRE themes amp features
bull Exercise mapping data into INSPIRE requirements
bull Transformation Configure mappings amp validation
bull Continue transformation amp exercises
bull Publish Data GeoServer
bull Discussion
INSPIRE Hands on Introduction - Just as a quick reminder
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
6 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
what it is about hellip
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
3 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Whatrsquos going on today
bull Intro to INSPIRE data specs (retrospection)
bull Relationship between source data and INSPIRE themes amp features
bull Exercise mapping data into INSPIRE requirements
bull Transformation Configure mappings amp validation
bull Continue transformation amp exercises
bull Publish Data GeoServer
bull Discussion
INSPIRE Hands on Introduction - Just as a quick reminder
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
6 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
what it is about hellip
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE Hands on Introduction - Just as a quick reminder
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
6 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
what it is about hellip
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
6 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
what it is about hellip
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
7 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE
bull Comprehensive data inventory (Monitoring amp Reporting IR)
bull Facilitate data discovery through standardised discovery
services amp metadata (IR on Network Services amp Metadata)
bull Data sharing (IR on Data and Service Sharing Article 17)
bull Facilitate data access by allowing standardised view download
and transformation (IR on Network Services)
bull Facilitate data use and interoperability by adopting common
cross-domain models to exchange data (IR on Data
Interoperability)
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
8 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE components
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
9 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Directive
General rules to establish an Infrastructure
for Spatial Information in Europe for
Community environmental policies
Policies or activities which impact on the
environment
INSPIRE is built on the SDIs established and
operated by the Member States
Spatial data held byon behalf of public
authorities
Does not require collection of new data
INSPIRE is a Framework Directive
Detailed technical provisions in Implementing
Rules
JRC iswas the technical coordinator
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
10 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Thematic Scope Annex I
Coordinate reference systems
Geographical grid systems
Geographical names
Administrative units
Addresses
Cadastral parcels
Transport networks
Hydrography
Protected sites
Annex II
Elevation
Land cover
Ortho-imagery
Geology
Annex III
Statistical units Area managementrestriction regulation zones amp reporting units
Buildings Natural risk zones
Soil Atmospheric conditions amp Meteorological geographical features
Land use Oceanographic geographical features
Human health and safety Sea regions
Utility and governmental services
Bio-geographical regions
Environmental monitoring facilities
Habitats and biotopes
Production and industrial facilities
Species distribution
Agricultural and aquaculture facilities
Energy resources
Population distribution ndash demography
Mineral resources
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
11 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Legally binding documents
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
12 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs)
+
No 12532013 21 Oct 2013
No 1022011 (code values Annex I)
No 10892010 (Annex I)
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
13 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE Implementing Rules (IRs) - ISDSS
defines the requirements of
bullspatial object types
bulltheir attributes
bullassociation roles
bullcode lists
bullcode list values
bulllayers for the spatial data theme
for all EU Member States (MS) public
sector (PS) organisations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
14 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Implementing Rules vs Technical Guidelines
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
15 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
Framework Documents
TG Annex I Annex II amp III
Metadata amp services
hellip
hellip Interoperability of spatial data sets amp services
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
16 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
17 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
class Env ironmentalMonitoringFacilities
GCM Base Types 2 Additional classes from GCM Observations
DataType
ISO FDIS 191562011 Observations and Measurements
laquofeatureTyperaquo
AbstractMonitoringFeature
laquovoidableraquo
+ reportedTo ReportToLegalAct [0]
constraints
Observation and ObservingCapability
laquofeatureTyperaquo
Env ironmentalMonitoringProgramme
laquofeatureTyperaquo
Env ironmentalMonitoringNetwork
laquovoidableraquo
+ organisationLevel LegislationLevelValue
laquodataTyperaquo
ReportToLegalAct
+ legalAct LegislationCitation
laquovoidableraquo
+ reportDate DateTime
+ reportedEnvelope URI [01]
+ observationRequired Boolean
+ observingCapabilityRequired Boolean
+ description CharacterString [01]
laquofeatureTyperaquo
Observ ingCapability
laquovoidableraquo
+ observingTime TM_Object
+ processType ProcessTypeValue
+ resultNature ResultNatureValue
+ onlineResource URL [01]
laquofeatureTyperaquo
AbstractMonitoringObject
+ inspireId Identifier
+ mediaMonitored MediaValue [1]
+ geometry GM_Object [01]
laquovoidableraquo
+ name CharacterString [0]
+ additionalDescription CharacterString [01]
+ legalBackground LegislationCitation [0]
+ responsibleParty RelatedParty [0]
+ onlineResource URL [0]
+ purpose PurposeOfCollectionValue [0]
laquofeatureTyperaquo
Env ironmentalMonitoringFacility
laquovoidableraquo
+ representativePoint GM_Point [01]
+ measurementRegime MeasurementRegimeValue
+ mobile Boolean
+ resultAcquisitionSource ResultAcquisitionSourceValue [0]
+ specialisedEMFType SpecialisedEMFTypeValue [01]
constraints
GeometryRequired
Observation and ObservingCapability
If Observation(s) are attached to an AbstractMonitoringFeature this must have
an ObservingCapability attached to it The ObservingCapability must reference
the same Domain Phenomenon and ProcessUsed as the Observation
inv hasObservation-gtnotEmpty() implies observingCapability-gtnotEmpty() and
hasObservationOM_ObservationfeatureOfInterest =
observingCapabilityfeatureOfInterest and
hasObservationOM_ObservationobservedProperty =
observingCapabilityobservedProperty and
hasObservationOM_Observationprocedure = observingCapabilityprocedure
NetworkFacility
laquovoidableraquo
+ linkingTime TM_Object
AnyDomainLink
laquovoidableraquo
+ comment CharacterString
GeometryRequired
Geometry and
representativePoint cant be
empty at the same time
inv geometry -gtnotEmpty() or
representativePoint -gtnotEmpty()
laquofeatureTyperaquo
Env ironmentalMonitoringActiv ity
+ inspireId Identifier
laquovoidableraquo
+ activityTime TM_Object
+ activityConditions CharacterString
+ boundingBox GM_Boundary [01]
+ responsibleParty RelatedParty
+ onlineResource URL [0]
Hierarchy
laquovoidableraquo
+ linkingTime TM_Object
EF-Level
Base Types 2LegislationCitation
+ identificationNumber CharacterString [01]
+ officialDocumentNumber CharacterString [01]
+ dateEnteredIntoForce TM_Position [01]
+ dateRepealed TM_Position [01]
+ level LegislationLevelValue
+ journalCitation OfficialJournalInformation [01]
Base Types 2DocumentCitation
+ name CharacterString
laquovoidableraquo
+ shortName CharacterString [01]
+ date CI_Date
+ link URL [1]
+ specificReference CharacterString [0]
laquoFeatureTyperaquo
observ ationOM_Observ ation
+ phenomenonTime TM_Object
+ resultTime TM_Instant
+ validTime TM_Period [01]
+ resultQuality DQ_Element [0]
+ parameter NamedValue [0]
constraints
observedProperty shall be a phenomenon associated with the
feature of interest
procedure shall be suitable for observedProperty
result type shall be suitable for observedProperty
a parametername shall not appear more than once
laquoFeatureTyperaquo
observationOM_Process
laquoFeatureTyperaquo
General Feature Instance
GFI_Feature
observ ation
Observ ationContext
+ role GenericName
laquometaclassraquo
General Feature Model
GF_PropertyType
root
+ memberName LocalName
+ definition CharacterString
laquotyperaquo
Records and Class MetadataAny
root
laquofeatureTyperaquo
ProcessesProcess
laquovoidableraquo
+ inspireId Identifier
+ name CharacterString [01]
+ type CharacterString
+ documentation DocumentCitation [0]
+ processParameter ProcessParameter [0]
+ responsibleParty RelatedParty [1]
laquoTyperaquo
Observable Properties
AbstractObservableProperty
+ label CharacterString [0]
laquodataTyperaquo
ProcessesProcessParameter
+ name ProcessParameterNameValue
+ description CharacterString [01]
laquofeatureTyperaquo
OperationalActiv ityPeriod
+ activityTime TM_Object
+operationalActivityPeriod
laquovoidableraquo1
realises
Phenomenon
+observedProperty1
+propertyValueProvider
0 Domain
+featureOfInterest
1
Domain
+featureOfInterest
laquovoidableraquo01
+generatedObservation
0ProcessUsed +procedure
1
ProcessUsed
+procedure1
+hasObservation
laquovoidableraquo0
+result
Range 0 +relatedObservation 0
Phenomenon +observedProperty
1
+uses
laquovoidableraquo
0
+involvedIn
laquovoidableraquo
0
+relatedTo
laquovoidableraquo
0
+belongsTo
laquovoidableraquo
0
+contains
laquovoidableraquo
0
+supersedes
laquovoidableraquo 0
genealogy
+supersededBy
laquovoidableraquo 0
+observingCapability
laquovoidableraquo 0
+broader
laquovoidableraquo
01
hierarchy
+narrower
laquovoidableraquo
0
+triggers
laquovoidableraquo
0
+setUpFor
laquovoidableraquo
0
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ inspireId Identifier
+ location GM_Point
+ hilucsLandUse HILUCSValue [1]
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ specificLandUse LandUseClassificationValue [1]
+ observationDate Date
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ inspireId Identifier
+ extent GM_MultiSurface
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
18 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Encoding
Encoding guidelines (D27)
updated during Annex II+III development
Minor changes in the encoding rule
bull references to code list values
bull association classes
GML application schema - XML schema
kind of templates that is used to express a set of
conformance rules for an GML file
bull defines also relations to other (base) schemas by
imported namespace
These are our Target Schemas
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
19 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Code lists
4 types of INSPIRE code lists
according to extensibility
a) not extensible ndash only values included in IRs are allowed
b) narrower extensible ndash values included in IRs and narrower
values are allowed
c) freely extensible ndash values included in IRs and any other values
are allowed
d) empty ndash any values are allowed
For code lists of types (b) (c) and (d) additional values have to be
published in a register
TG-DS may include additional proposed values that will be published in
the INSPIRE code list register
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
20 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Registers
EC provides central registries
for INSPIRE resources
published on
httpinspireeceuropaeuregistry
bull registers code lists themes application schemas fcd
bull browsing and accessing register content
bull Formats HTML XML Atom JSON and RDFSKOS
bull Multilingual content (based on IR content)
Open to external contributions
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
21 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Conceptual data models
Registers
bull objects types properties amp relationships
bull cross-domain harmonization
bull based on a common modelling framework
bull managed in a common UML repository
Harmonised vocabularies
bull to overcome interoperability issues caused by free-text andor multi-lingual content
bull allow additional terms from local vocabularies
Encoding
bull conceptual models independent of concrete encodings
bull standard encoding GML but also possible to derive other encodings (eg based on RDF)
bull provide unique and persistent identifiers for reference to resources
bull allow their consistent management and versioning
Key pillars of data interoperability
described in INSPIRE Conceptual Framework documents
D26Methodology
for Specification
Development D2103 Common
data models
D29 OampM
Guidelines D25 Generic
Conceptual Model
D27 Guidelines
for Encoding
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
22 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
UML models xml schemas registers that what JRC delivers amp maintains
httpinspireeceuropaeuindexcfmpageid2listdatamodels
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE Hands on How to read Technical Guidelines ndash basic notations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
Chris Schubert
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
24 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Technical Guidelines (TG-Data Specification)
1 Scope 2 Overview and Description 3 Specification scopes 4 Identification information 5 Data content and structure Application schemas Feature catalogue Notations Voidable characteristics Enumerations Code lists Identifier management Geometry representation Temporality representation 6 Reference systems units of measure and grids Theme-specific requirements amp recommendations 7 Data quality 8 Dataset-level metadata 9 Delivery incl Encoding 10 Data Capture 11 Portrayal 12 Bibliography Annex A (normative) Abstract Test Suite Annex B (informative) Use cases Annex C (normative) Code list values Annex D (informative) Examples
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
25 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Basic notations in TG-DS
D25
-G
eneric C
onceptu
al M
odel
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
26 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
Stereotypes
Fixed set of values
Closed or extendable set of values
D25
-G
eneric C
onceptu
al M
odel
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
27 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE identifier
bull Used to reference the object
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
A namespace to identify the data
source The namespace is owned by
the data provider
A local identifier assigned by the data
provider The local identifier is
unique within the namespace ie
no other spatial object carries the
same unique identifier
namespace will consist of two parts The first part will identify the data
provider within the member state and the second part will be used to
distinguish between different data sources maintained and provided by the
data provider ltlt―NLTOP10NLgtgt may be the namespace for spatial objects
in the TOP10 NL product in the Netherlands
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
28 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Object life-cycle
4 cross-thematic attributes
bull beginLifespanVersion DateTime
Date and time at which this version of the spatial object was inserted
or changed in the spatial data set
bull endLifespanVersion DateTime [01]
Date and time at which this version of the spatial object was
superseded or retired in the spatial data set
bull validFrom DateTime [01]
The time when the phenomenon started to exist in the real world
bull validTo DateTime [01]
The time from which the phenomenon no longer exists in the real world
INSPIRE base types and basic notions
D25
-G
eneric C
onceptu
al M
odel
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
29 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
INSPIRE base types and basic notions
ltltVoidablegtgt elements
bull Related to INSPIRE obligation
bull Data providers are authorized not to provide voidable elements
bull EITHER if no corresponding data is captured
bull OR if no corresponding data can be derived from other existing data at
reasonable costs
D25
-G
eneric C
onceptu
al M
odel
When a voidable element is not provided the reason should be given
Unpopulated Not part of the dataset maintained by the data provider
Unknown Not known to and not computable by the data provider
Withheld Confidential and not divulged by the data provider
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
30 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Delivery
the use of GML as the default encoding is proposed
Rules for default encodings for application schemas are defined in
GML application schema also called XML schema
Eg httpinspireeceuropaeuschemasgelu30GriddedExistingLandUsexsd
No (legal) obligation to use GML in INSPIRE For INSPIRE download services - Media types used for spatial data sets are listed httpinspireeceuropaeumedia-types bull x-shapefile bull x-filegdb bull imagetiff bull google-earthkml+xml bull hellip
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
32 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Land Use ndash INSPIRE data theme
bull httpinspireeceuropaeuindexcfm
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
33 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Existing amp Planned Land Use
Land Use - description of land in terms or its socio-economic and
ecological purpose - territory characterized according to its current and
future functional dimension or socio-economic purpose and is split in
two different types
The Existing Land Use
Geographical data-sets that provide Land Use information at the time
of observation
The Planned Land Use
Which corresponds to spatial plans defined by spatial planning
authorities depicting the possible utilization of the land in the future
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
34 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Application schemas included in the IRs
The types to be used for the exchange and classification of spatial
objects from data sets related to the spatial data theme Land Use are
defined in the following application schemas
The Existing Land Use
Existing Land Use application schema (organized as a partition of a
given area)
Gridded Land Use application schema (organized as a set of pixels)
Sampled Land Use application schema (organized as a set of
discrete observation points )
Planned Land Use application schema
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
35 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Sampled Land Use ndash UML model
Application schema corresponds to a dataset that depicts the reality of the land surface at
discrete location on the earth Often these datasets are collected for statistical purposes to
provide estimates of land use over wider areas
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
36 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name inspireId
Definition External object identifier of the land use sample
Description
An external object identifier is a unique object identifier
published by the responsible body which may be used
by external applications to reference the spatial object
The identifier is an identifier of the spatial object not an
identifier of the real-world phenomenon
Voidable false
Multiplicity 1
Value type Identifier (data type)
Name location
Definition Location where the land use sample is taken
Voidable false
Multiplicity 1
Value type GM_Point
Name beginLifespanVersion
Definition
Date and time at which this version of the spatial
object was inserted or changed in the spatial data
set
Voidable true
Multiplicity 1
Value type DateTime
Name endLifespanVersion
Definition
Date and time at which this version of the spatial
object was superseded or retired in the spatial data
set
Voidable true
Multiplicity 01
Value type DateTime
Name hilucsLandUse
Definition Land use HILUCS classes that are present in this existing land use sample
Description
NOTE The Sampled Existing Land Use model enables the provision of
information on land uses inside one land use object The
ExistingLandUseObject may be associated with 1 to many HILUCSLandUse that
represents the Land Uses for the polygon from the economical point of view It
makes possible the assignment of more than one HILUCSLandUse existences
when they cannot be managed by HILUCSPresences
Voidable false
Multiplicity 1
Value type HILUCSValue (code list)
Name hilucsPresence
Title land use presence
Definition Actual presence of a land use category according to
HILUCS within the object
Voidable true
Multiplicity 1
Value type HILUCSPresence (union data type)
Name observationDate
Title Observation Date
Definition The observation date associated to a description
Description
Defines the observation date of the description It could be the
date of an aerialsatellital acquisition or a field survey The
observation date allows the user to have accurate date of when
the description was made in the real word In a database not
all object informations are necessarily captured at the same
time
Voidable true
Multiplicity 1
Value type Date
Name specificLandUse
Definition Land Use Category according to the nomenclature
specific to this data set
Description
Reference to an entry in the classfication that is part
of the SpecificLandUseClassification provided by the
data producer
Voidable true
Multiplicity 1
Value type LandUseClassificationValue (code list)
Name specificPresence
Definition Actual presence of a land use category within the
object
Voidable true
Multiplicity 1
Value type SpecificPresence (union data type)
Name validFrom
Definition The time when the phenomenon started to exist in
the real world
Voidable true
Multiplicity 01
Value type Date
Name validTo
Definition The time from which the phenomenon no longer
exists in the real world
Voidable true
Multiplicity 01
Value type Date
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
Association role
Name dataset
Definition Data set to which this sample belongs
Voidable false
Multiplicity 1
Value type SampledExistingLandUseDataSet (spatial object
type)
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
37 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Feature catalog VS UML model
Name extent
Definition The convex hull of all the instances of the spatial object type ExistingLandUseSample
Voidable false
Multiplicity 1
Value type GM_MultiSurface
Name name
Definition Human readable name of the data set
Voidable false
Multiplicity 1
Value type CharacterString
Name member
Definition Reference to the members of the sampled existing land use data set
Voidable false
Multiplicity 0
Value type ExistingLandUseSample (spatial object type)
class Sampled Land Use
laquofeatureTyperaquo
ExistingLandUseSample
+ hilucsLandUse HILUCSValue [1]
+ inspireId Identifier
+ location GM_Point
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ hilucsPresence HILUCSPresence
+ observationDate Date
+ specificLandUse LandUseClassificationValue [1]
+ specificPresence SpecificPresence
+ validFrom Date [01]
+ validTo Date [01]
laquofeatureTyperaquo
SampledExistingLandUseDataSet
+ extent GM_MultiSurface
+ inspireId Identifier
+ name CharacterString
laquolifeCycleInfo voidableraquo
+ beginLifespanVersion DateTime
+ endLifespanVersion DateTime [01]
laquovoidableraquo
+ validFrom Date [01]
+ validTo Date [01]
+dataset
1
+member
0
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
38 Training INSPIRE the INSPIRE Hands on - Data transformation amp download services
EC JRC
Hierarchical INSPIRE Land Use Classification System (HILUCS)
The Land Use data specification supports two systems of classification the obligatory ndashHILUCS multi-level classification system - all
application schemas (existing and planned) the (optional) specific classification system in use in a member
state (LUCAS NACE SEEAhellip)
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
LUCAS Land usecover statistics
Understanding the Source Schema and Data Set
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Index
1 What is LUCAS
2 Difference between Land Cover and Land Use
3 Data provided by LUCAS
4 What does LUCAS look like
41 LUCAS attributes
42 LUCAS codelists
5 Our Sample Data Set
6 Links
40
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
What is LUCAS
Is the EUacutes harmonized land use and land cover survey
+ module providing Soil information (TOPSOIL)
Implemented by Eurostat with the member states collaboration
Survey from 2006 Updated every 3 years
Surveys are made in-situ
after a design phase for selecting the sample locations
1 point every 2 square kilometers
41
2006
11 countries
2009
23 countries
2012
27 countries
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
42
Difference between Land Cover and Land Use
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Data provided by LUCAS
43
LUCAS
Statistical Tables
Primary Data
Photos
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
44 httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
What does LUCAS look like
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
45
LUCAS attributes
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
More than 50
Identifier
Location
Time
Land Use
Land Cover
Soil
Water
Others
Point_ID X_LAEA Y_LAEA STRATA NUTS0 NUTS1 NUTS2
area2 WH PESO_F2 SURV_DATE OBSERVED OBS_TYPE GPS_PROJ
GPS_PREC GPS_Y_LAT GPS_EW GPS_X_LON
G Y_LAT EW X_LONG
OBS_DIST OBS_DIRECT LC1 LC2 OBS_RADIUS LU1 LU2
LC1_SPECIES LC1_PERCEN
T LC2_SPECIES
LC2_PERCENT
AREA_SIZE TREES_HEIG
HT FEATURES_W
IDTH
LAND_MNGT WM_WATER
_MNGT WM_SRC_IR
RIGATION WM_TYP_IR
RIGATION WM_DELIVE
RY_SYST SOIL_SURVE
Y SOIL_PLOUG
H
SOIL_CROP SOIL_STONES SOI_LABEL PHOTO_POI
NT PHOTO_N PHOTO_E PHOTO_S
PHOTO_W TR1 hellip
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
LUCAS code lists
Classifications are hierarchical
different levels of information
structured broad-level classes
allow subdivision into more
detailed sub-classes
At each level the defined classes are
mutually exclusive
46
Classification
U100 ndash Agriculture Forestry and
Fishing
U110 - Agriculture
U111 ndash Agriculure
U112 ndash Fallow and abandoned
land
U113 Kitchen gardens
U120 - Forestry U130 Fishing hellip
U200 ndash Manufacturing
and energy hellip Classification
U100 ndash Agriculture Forestry and
Fishing
U200 ndash Manufacturing
and energy hellip
U110 - Agriculture
U120 ndash
Forestry
U130 ndash
Fishing hellip
U111 ndash
Agriculure
U112 ndash
Fallow and abandoned land
U113 ndash
Kitchen gardens
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Our sample Data Set SampleLUCASshp
47
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
Download France and Spain 2009
data
Merge in a single CSV
Create shapefile from
coordinates fields
Create a crossborder
selection
Export to shapefile
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
LUCAS viewer httpeceuropaeueurostatstatistical-atlasgisviewermyConfig=LUCAS-2012xml
About LUCAS httpeppeurostateceuropaeuportalpageportallucasintroduction
httpeppeurostateceuropaeuportalpageportallucasdata
The EUrsquos land use and land cover survey
httpeppeurostateceuropaeucacheITY_OFFPUBKS-03-13-587ENKS-03-13-587-ENPDF
Example of indicator in Eurostat Database
httpappssoeurostateceuropaeunuishowdodataset=lan_lcv_ovwamplang=en
LUCAS TOPSOIL data httpeusoilsjrceceuropaeuprojectsLucasDatahtml
LUCAS microdata used Spain httpeppeurostateceuropaeuportalpageportallucasdocumentsES_2009csv France httpeppeurostateceuropaeuportalpageportallucasdocumentsFR_2009csv
LUCAS Metadata httpeppeurostateceuropaeucacheITY_SDDSenlan_esmshtmrelatedmd1392278553624
48
Links
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Transformation ETL never heard hellip
ETL is a process to integrate different information from multiple
applications (data base management)
ETL is short for
Extract is the process of reading amp understand information
Transform is the process of converting the extracted information by using
rules or by combining the data with other information
Load is the process of writing the information according a target
schema
Thatrsquos why it is important (we showing again)
understand the specific scope TG-DS getting close to the domain
expertise
understanding the data models helps
have an idea about the content of frame work documents
basic knowledge about gml (technicians will help you)
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Transformation
simple transformation - renaming assign new properties
complex transformation - reclassification geometry calulation
origin conformant
transformation
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Transformation is an ETL repeatable procces
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Transformation needs domain expertise
load target SCHEMA define INSPIRE mapping
create transformation rules
analytical task
run transformation
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
architectural approaches an Overview
One-off transformation + external web based services | AtomWFShellip
One-the-fly transformation | AtomWFShellip
One-offthe-fly transformation + integrated web based services |
AtomWFShellip
but what happened with changes (update)
need to maintain donrsquot forget
Consider for choosing an
approach (operational process)
will the data set in future
bull+- static
eg geology
bull under frequently change
eg land use
bull under permanent change
eg air quality reporting
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
One-off transformation + external web based services
positive aspects are
transformed once for all users
bull better performance when delivering the
data
bull no transformation during delivery
bull Free choice of software components
negative aspects are
Requires storage and management of
transformed data
in addition to original data
bull high processing effort
bull the entire database is transformed
bull to be maintained also transformed data
This approach is usefull if data quite stabil
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
On the fly transformation+ integrated web services
positive aspects are only the original data has to be
maintained only the requested data has to be
transformed bull limited on the geometric amp semantic
request
negative aspects are Performance issues bull high processing required before delivery
especially for large volumes of data and complex transformations
bull the same data is potentially transformed multiple times
bull Unless caching or pre-processing mechanisms has to be used
This approach is useful if data is continuously or frequently updated
eg SnowflakeGoLoader-PublisherWFS GeoServer ()
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
One-off transformation + integrated web based services
positive aspects are
Data transformed offline can be
managed in same system as original
data eg in the same database management
system not as GML files
bull lsquoOn-the-flyrsquo get more performance
because of predefined data structure
negative aspects are
Data provider still has to store
transformed data
no really FOSS-GIS solutions (current)
This approach is useful if data is continuously or frequently updated
eg ArcGIS for INSPIRE
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
HALE
Setting up and Understanding the workbench
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Index
1) Whatrsquo s HALE
2) Starting HALE
3) Configuring the workbench
4) Schema Explorer tab in detail
5) HALE GUI
6) Functions for transformation
7) Links
80
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Whatrsquo s HALE
HALE HUMBOLDT Alignment Editor
- Part of HUMBOLDT Harmonisation Toolkit
- Maintained by the Data Harmonisation Panel
Tool useful to
- Create mappings (alignments)
- Validate
- Transform data
- Styling (SLD generation)
Current version 280 (from 2013-11-13)
- Working on the next version 290
- We can see the features added or fixed in the issue tracker
website
81
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Starting HALE
1 Download
2 Unzip
3 Double click on HALEexe
82
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Configuring the workbench
83
1 Import a source Schema
FILEIMPORTSOURCE SCHEMA SampleLUCASshp
2 Import a target Schema
FILEIMPORTTARGET SCHEMA SampledExistingLandUsexsd
3 Import Source Data
FILEIMPORTSOURCE DATA SampleLUCASshp
4 Remove the FeatureType donacutet needed
EDITEDIT TARGET MAPPEABLE TYPES
REMOVE THE SampleLandUseDataset
Remember to save the
project
Source Schema amp
Data
Target Schema
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Schema Explorer tab in detail Elements
84
Feature Type Attributes Data type Cardinality Number of features
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Asterisk Mandatory field
Arrow XML attribute
Question mark Choice
Schema Explorer in detail Simbology
85
Icons Data Types
Icons Attribute information
Text
Number
Geometry
Complex
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Schema Explorer in detail Simbology
86
No color Not mapped
Green Mapped explicitly with a relation
Yellow Mapped implicitly due to the mapping of a sub-property
Purple Value assignment independent of the source schema
Background colors State of the mapping
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
HALE GUI
Usable
Easy to use and understand
Flexible
GUI completely customizable
Complete
Provides different tabs and
predefined views of the same
project
87
This is very helpful and useful to look easily if our transformation
is correctly done
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
88
Schema Explorer
Alignment
Error log Report list
Default View
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
89
Data View
Source Data
Transformed Data
Alignment
Properties
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
90
Map View
Source Data map Transformed Data map
Source Data attributes Transformed Data
attributes
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Functions for transformation
General
Retype
Merge
Join
Create
Date extraction
Regex Analysis
Rename
Assign
Generate Unique Id
Classification
Formatted string
Geometric
Ordinates to Point
Network Expansion
Calculate Length
Calculate Area
Centroid
Compute Extent
Groovy
Groove Retype
Groovy Create
Groovy Merge
Groovy script
Inspire
Inspire Identifier
Geographical Name
Numeric
Mathematical Expression
Generate sequential I
91
You can arrive to the same output by using different tools
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Some functions
Functions Description
Type relation
Retype For each instance of the source type an instance of the target type is created
Property relation
Rename Copy a source property to a target property
Assign Assigns a value to a target property
GenerateUniqueID Assigns a generated unique id to a target property
Classification Map classifications
Formatted String Creates a formatted string based on a pattern and the input variables
92
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Links
HALE website httpwwwdhpaneleuhumboldt-frameworkhalehtml
HALE Wiki httpwwwesdi-communityeuprojectshalewiki
HALE Tutorial httpwwwdhpaneleuhumboldt-frameworkhale-tutorialhtml
HALE download httpwwwesdi-communityeuprojectshalefiles
HALE User Guide httphaleigdfraunhoferde280helpindexjsp
93
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE network services
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE principles Data should be collected once and maintained at the level
where this can be done most effectively Combine seamlessly spatial data from different sources and
share it between many users and applications (the concept of interoperability)
Spatial data should be collected at one level of government and shared between all levels
Spatial data needed for good governance should be available on conditions that are not restricting its extensive use
It should be easy to discover which spatial data is available to evaluate its fitness for purpose and to know which conditions apply for its use
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Relevant components
INSPIRE Network services
In INSPIRE data should be made available where best managed 1 Discover
Expose metadata through INSPIRE compliant discovery service
2 View
Interactive view of data through an INSPIRE compliant data service
Unified portrayal through OGC Styled Layer Descriptor (SLD)
3 Download
Web Feature Service
Atom feeds
4 Transform
ETL
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Discovery services
bdquodiscovery services making it possible to search for spatial data sets and
services on the basis of the content of the corresponding metadata and
to display the content of the metadataldquo
httpinspire-geoportaleceuropaeudiscovery
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE Metadata Distinguish between
bull Spatial object metadata
bull Dataset-level metadata
Tools available at JRC site
bull INSPIRE Metadata editor
bull INSPIRE metadata validator
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
View services
bdquoview services making it possible as a minimum to display navigate
zoom inout pan or overlay viewable spatial data sets and to display
legend information and any relevant content of metadataldquo
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
View and discovery services at INSPIRE Geoportal
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Download services
bdquodownload services enabling copies of spatial data sets or parts of such
sets to be downloaded and where practicable accessed directlyldquo
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE Download services
Available options (now)
bull Atom feeds
bull WFS (Web Feature Service)
Work ongoing for
bull INSPIRE compliant download service based on OGC Sensor
Observation Service (SOS)
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Download services Predefined
Direct access
Service implementation
Predefined dataset download service
Direct access download service
SOS X X
WFS Х Х
Atom Х
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Relevant components
INSPIRE Geoportal bull Central access point to the INSPIRE
infrastructure and resources (250 000+)
ldquoThe facerdquo of INSPIRE
bull Connection to all MS network services
cross-border data discovery and visualisation
support to
policy making
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Introduction to OSGEO Live What we use
bull Apache
bull GDAL (ogr2ogr)
bull PostgreSQLPostGIS
bull Geoserver
bull Geonetwork - open source
bull Text editor
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
European Commission Joint Research Centre
Institute for Environment and Sustainability
Digital Earth and Reference Data Unit
wwwjrceceuropaeu
Serving society
Stimulating innovation
Supporting legislation
INSPIRE Hands on Summary Data transformation
Chris Schubert et al
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
INSPIRE ldquostatusrdquo Leaving the conceptual level
Final TG-DS II+III (30)
published end of 2013
Update TG-DS I published
2014
Final XML schema published
2014
Starting with the Implementation
There are still open issues eg
cross-cutting coherence
validation - ATS abstract test
suite
(compliance vs conformance)
etchellip
INSPIRE Maintenance and Implementation Framework
bull Support implementation
bull Corrective maintenance
bull Evolutive maintenance
hellip for exchange of experience and good practice Designed by NINJAinfographics
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations
Discussion and Conclusion
bull Learning outcomes bull how to understand the INSPIRE DS
bull to use xml schemas
bull to deal with controlled vocabularies
bull how to transform datasets
bull to create gml files
bull how to create INSPIRE-compliant web services with GeoServer
Did we met expectations