the geodata warehouse of the swiss federal sdiconceptual view of the fsdi «ingest once, use...
TRANSCRIPT
Federal Department of Defence, Civil Protection and Sport
Federal Office of Topography swisstopo
Dr. Patrick Gamma
Federal Office of Topography swisstopo
Esri European User Conference
Salzburg, 15 October 2015
The Geodata Warehouse of the
Swiss Federal SDI
2Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Agenda
• Introduction
• Legal Basis, Terms & Definitions
• Life Cycle of Geodata
• Architecture of the Geodata Warehouse
• Technology
• Next Steps & Conclusions
3Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
What is the value of your Geodata?
Imagine that your organisation has lost an important dataset…
• Could you recreate it at all?
• Possibly with some information loss?
• Price for recollecting / reconstructing?
• What is the impact of not having the data?
• Good strategies (besides classic IT approach):
Use the data
Open data
• Our strategy: the above, plus a Geodata Warehouse, a «living archive»
4Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Could you archive a geodataset from
1995 without any loss?
Imagine that your organisation should file a
geodataset, that was used or created 20
ago…
• Could you still read it?
• Is the expertise available to convert it into
the required formats?
• Would you be able to provide the necessary
information, e.g. Documentation, Metadata,
Data Model, Representation Model?
5Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Just some thoughts about data loss...
• Data loss is one of the greatest risks for a NMA on the enterprise level, caused by e.g.
Technical problems, or even on purpose
Deleting the wrong data
• Some losses could only be detected after years, at the time when someone tries to access the data
• One can not keep all data limited storage costs
• Even if you keep all data, you probably won’t have the software to read it in the future
• Decide about which datasets should be kept and for how long, before being sent to the archives
• You should provide the documentation about a dataset while the know-how (people!) is still available
• Future effort for preparing & sending the data to the archives is minimal (ROI)
6Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Legal Basis, Terms and Definitions
• The Federal Act on Geoinformation (GeoIA, 2008)
and the Ordinance on Geoinformation (GeoIO,
2008) essentially define 2 states / conditions in
which a geodata set can be:
• Long-term availability
The authorities responsible for the collection,
updating and management of official geodata
guarantee the long-term availability of this geodata.
• Archived
The Federal Archives conserves the geodata
«in use»GIS Community
«not in use»No GIS
7Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The «Life Cycle» of Geodata (1)
8Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The «Life Cycle» of Geodata (2)
9Bundesamt für Landestopografie swisstopo
KOGIS
Kolloquium swisstopo
6. März 2015
The Aim of Long-term Availability…
… is to conserve official geodata for a limited period…
… in such a way that their quantity and quality are maintained…
…and they are available for continuous active use.
• Online availability should extend not just to the data fonds that are current at a given time but also to defined older snapshots (in the sense of time series).
From: Project Ellipse: Concept for the archiving of official geodata under federal legislation; Concept report. Federal Office of Topography swisstopo, Swiss Federal Archives, Version 1.3, 14.09.2013, p. 38
10Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Long-term Availability vs. Archiving
of Geodata
«Frozen pizza from home delivery»«Hot pizza from pizza service»
Criterion Long-term availability Archiving
Preservation period Limited in time Unlimited in time
GIS functionality Yes No
Formats Current formats, ready for use Archivable formats
PrincipleKeep the data accessible, both
current and older releases
Keep permanently and
safely
Frequency of use Frequently Seldom
11Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Long-Term Availability
Conceptual View of the FSDI
«Ingest once, use anywhere»
12Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Requirements for the GDWH
• Ensure the long-term availability of official
geodata
• Import, preserve and export the geodata for
use within the FSDI
• Reduce or limit redundancy
• Minimise software manufacturer dependencies
• For swisstopo and for other federal agencies
(replace swisstopo legacy application, make it
useable for others)
• Fit the system in the overall FSDI framework
and the corresponding processes
13Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
OAIS Open Archival Information System
Reference Model ISO 14721:2012
Submission
Information
Package
Archival
Information
Package
Dissemination
Information
Package
14Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDI
System ArchitectureB
usi
ne
ssLo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Standardised Directory Structure
Standardised Data Formats
Consistency Checks
Accompanying documents
Aggregation Layer
DB-Access LayerTransformation
FunctionsConsistency
Checks
Services
GDWH Data Dictionary
Service
Geodata Delivery
Geodata Validation &
Ingest
Geodata Transformation
GenerateTransformAggregate
«Product independent»Original-/Master DataArchivable
Current GIS technology«Product dependent»Replacable/exchangeableGenerated from
Filesystem Layer
Interfaces to FSDILocal Catalogue
15Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Filesystem & Aggregation Layer Demo
16Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Bu
sin
es
s
Lo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Original Data2015
Aggregation Layer
Services
Software2015
Geodata Warehouse FSDIMigration Strategy for Software and Data
FilesystemLayer
Original Data2020
Convert
2015 2020
Migrate
Software2015
GIS Data2015
Aggregation Layer
Software2020
GIS Data2020
Services
Software2020
No change
Gene-
rate
Migrate
«Product independent»Original-/Master DataArchivableUntouched
Current GIS technology«Product dependent»Replacable/exchangeableGenerated from Filesystem
Layer…or converted from
Aggregation layer
Interfaces to FSDILocal CatalogueNew/adapted functionality
17Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDIMigrating over Time
2015 2020 2025 2030
2020 2025 2030
2020 2025 2030
2020 2025 2030
2020 2025 2030
18Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Consistency Checks / Validation for
Raster and Vector Data (1)
19Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Consistency Checks / Validation for
Raster and Vector Data (2)
20Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Archivable Data Formats
Fundamental Factors supporting LTA
in the Geodata Warehouse FSDI
Consistency Checks
System Architecture
Checksums File DatesStructuresMapping…Daily, Weekly,
Monthly
StandardisedOpenDocumented
BackupMirroringAccess Control Inhouse Servers
Dual Data TierHuman Readable StructuresService oriented
Classic IT Measures
Long-term Availability
in the Geodata Warehouse
21Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Geodata Warehouse FSDI
Tools and TechnologiesB
usi
ne
ssLo
gic
Tie
rD
ata
Tie
r
Filesystem Layer
Standardised Directory Structure
Standardised Data Formats,
e.g. model based
Consistency Checks
Accompanying documents
Aggregation Layer
DB-Access LayerTransformation
functionsConsistency
Checks
Services
GDWH Data Dictionary
Service
Geodata Delivery
Geodata Validation &
Ingest
Geodata Transformation
RepresentTransform
TIFFTXT, CSVPDF/AILI, ITFXML
ArcGIS DesktopFME DesktopGDAL LibraryREFRAME LibraryINTERLIS ToolsLAS ToolsESRI FGDB
ArcGIS ServerFME Server
C# .NetPythonJavaScript
22Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Shop
ArcGIS Web Services
Geodata Warehouse FSDI
Service oriented and modular Elements (1)
ShopExport
Manager
Web UI
FilesystemLayer
Aggregation Layer
Geodatasets ◄►Products
ArcMap
GDWH Core
.EXE FME
GDAL
Cockpit
Win Cmd
Web UI
GDWH Web Services
Web UI
Fire
wall
Order Manager
Web UI
Cockpit
Win Cmd
Web UI
Web UIFME
Desktop
23Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
VectorRaster
BusinessLogic Tier
Data Tier
Services
Aggregation
Layer
Filesystem
Layer
ArcGIS Server
Geodata Warehouse FSDI
Service oriented and modular Elements (2)
Export Manager
GDWH Core
FME Server
INTERLIS Checker
GDWH Services
.EXE GDAL
FGDBFGDB
TIF XML ILI ITF
XTFCSV TXT XML
Web Server
LAS ToolsLASlib
LAS
LAS
• Additional software components to support specific data types can
be added if necessary, e.g. LAS or 3D
24Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
Next Steps and Conclusions
• The new Geodata Warehouse enables the
Swiss Federal Spatial Data Infrastructure to
provide an environment to preserve geodata
(sustainable / long-term availability), both for
swisstopo itself and other federal agencies
• Swisstopo is currently adapting the remai-
ning systems of the FSDI to achieve this
goal, including the processes
• A geodataset becomes more interesting with
increasing age no need to archive
25Federal Office of Topography swisstopo Esri European User Conference, Salzburg
15 October 2015
The End