processing and dissemination of satellite remote sensing...
TRANSCRIPT
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 1
Processing and dissemination of satellite remote sensing data in an heterogeneous environment.
Armin Burger, Paul [email protected], [email protected]
Community Image Data portal project
http://cid.jrc.it
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 2
Presentation structure
Project’s objectives
A few words about the Institution
Step 1: from a tape repository to a digital repository
Step 2: extending this digital repository by building services around it
Conclusion
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 3
European Commission (EC)
The Commission is independent of national governments.
Its job is to represent and uphold the interests of the European Union (EU) as a whole.
It drafts proposals for new European laws, which it presents to the European Parliament and the Council.
It is also the EU’s executive arm – in other words, it is responsible for implementing the decisions of Parliament and the Council.
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 4
Joint Research Centre (JRC)
The European Commission’s
Research-Based Policy Support Centre
The JRC provides customer-driven scientific and technical support to Community policy making.
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 5
Use of satellite imagery at the JRC
Twenty projects using Satellite Remote Sensing (SRS) data
Annual expenditure on SRS data > 7 M€
100 000’s of Low Resolution images
More than 12 000 High Resolution images
Over 500 000 km2 of VHR images
Overall, more than 55 TB of SRS data
Scattered storage and no central catalogue
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 6
First setup of an online archiveInitiated on the archive of one project end 2003.
MARS-PAC project is the largest civilpurchaser of satellite imagery in the EU.
Largest archive of HR and VHR data in the JRC
Goal: To be able to search for images from a web browser, to visualize them and get a linkto the files.
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 7
Characteristics of this first setupSRS data loading application
Works with raw HR and VHR dataExtracts metadata and stores them in a spatial DB (PostgreSQL/PostGIS)
SRS data pre-processingConversion of the data to TIFF format and georeferencing.Histogram stretching and conversion to ECW
SRS data disseminationSearch and browse imagery (ECW)Web mapping (Mapserver)Download of raw or georeferenced data
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 8
Outcome of this first setupSuccessful as initial objectives were reached
Several limitations were identified:disk space not easily extendablerestricted to raw HR and VHRno fine grained authentication possible (users and IPRs)no WMS or WCS servicesevere restrictions in usability by other projects at JRC
Management wish to build on it and provide a site-wide service
Expected cost savings in hardware and staff
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 9
Survey of potential usersInitiative welcomed!!!
Community Image Data portal project started!
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 10
CID portal project objectivesProvide a centralized storage facility with backup and a data upload application.
Provide a data acquisition service.
Provide a service with IT infrastructure in High Availability mode.
Build a central catalogue of all SRS data available at JRC with a CSW service and a web interface to the catalogue.
Provide secure access according to IPRs and user credentials viadirect file access at high speed (NFS v4, CIFS)OGC protocols (WMS, WCS)other (web mapping, KML...)
Provide some SRS data processing services:[semi-]automated orthorectification
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 11
CID portal – Staff and BudgetBudget to initiate the project:
Hardware: 200 K€Software development: 130 K€
Orthorectification softwareModules for authentication and catalogue interface
Annual resources:
IT consultancy and small developments: 50 K€1.5 engineers and 0.7 IT technician.
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 12
CID portal overview JRCData Owner
DataProvider
JRC staff
User
DATAstorage
METADATAdatabase
SRS dataprocessing
SRS dataacquisition
SRS dataloading
AUTH.service
NFS4CIFS
AUTH.service
CSWWMS WCS
KML
CID spec.service
CIDdata serving
serviceActor
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 13
CID services overview
AUTH: Authentication and Authorization service
SRS data acquisition: web application for SRS data ordering
SRS data loading: website, web service and desktop interface
SRS data processing: metadata extraction, registration in a central catalogue and orthorectification for HR and VHR data.
CID data serving: direct file access and OGC services
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 14
Open source software and CID
The preferred option at CID because:
Knowledge available within CID staff
Almost all functionalities required are available in FOSS
Some functionalities available only in FOSS!
Good support from the user community
No cost to acquire the piece of software
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 15
Base infrastructure
OS: Xen with Debian 4 (80%) and Windows 2003 server
Base services:Bind, NTP, MIT Kerberos, LVS
Monitoring and statistics:Zenoss, Awstats, JMeter
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 16
Authentication and Authorization Service
Extremely important service:Enforcement of Intellectual Property Rights restrictions Sensitivity of some datasets (e.g. crisis areas)
Technologies used:OpenLDAP with KerberosNFS v4 and CIFS with ACLsCentral Authentication Service (CAS)PostgreSQL with PostGISCustom authorization application interfacing with OGCservices (PHP/Drupal framework)
AUTH.service
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 17
SRS data acquisition service
Web based applicationinterfacing with 6 SRSdata providersfor 9 missions.
Technologies used:DotNetNuke framework
SRS dataloading
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 18
SRS data processing SRS dataprocessing
orthonode
1
orthonode
2
orthonode
x
JBoss cluster
raw image rectification process final ortho image
Digital Elevation Model
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 19
CID Data serving service
Web mapping [p.mapper, CAS]
Catalogue Web interface [PHP/Mapscript]
CSW [GeoNetwork]
WMS, WCS [PHP/Mapscript, CAS]
Direct file access [NFS, CIFS]
Download [PureFTPd]
Official CID website for referenceinformation [typo3]
CIDdata serving
service
Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 20
In the coming months...By end 2007:
All backend and base services up and running in HAProcessing service placed in its production environmentCompletion of the web portal
Q1 / 2008:OGC services implementedSRS data loading service implementedSRS data acquisition service moved to the new environmentFirst import of data and intense testing
Q2 / 2008:Import of other datasetsPublic access to the CID portal (http://cid.jrc.it)