vancouver uses fme to open data to the world
DESCRIPTION
"Attend this session to learn how the City of Vancouver uses FME to: • Provide data for both their Open Data and Road Ahead initiatives. • Improve the City's Road Ahead map with more current information and open formats. (Learn how FME is being used to retrieve the source data from SQL Server and Oracle Spatial, and to generate SDF3/SDF2 files and a GeoRSS/RSS feed integrated with Google Street View, which is accessible from the Road Ahead home page. Plus, see how the Road Ahead FME workspace is also published to FME Server where the relevant data format files can be generated on the fly via data streaming services.) • Prepare datasets for the City's OpenData website on a weekly basis. "TRANSCRIPT
Vancouver Uses FME to Open Data to the World
Meng Li, GIS Systems Analyst, City of Vancouver
Jonathan Mark, GIS Manager, City of Vancouver
March 29, 2010
We’ll Talk About
Overview The Open Data Initiative and ICIS support The Road Ahead Website Upgrade Safe software in use at the City
Open Data Infrastructure Platform Workflow, ETL scripts, and deployment
The Road Ahead Website Upgrade Project ETL, Atom, GeoRSS, data streaming experience
The Open Data Initiative
The Open3 Council motion of May 2009 Open data Open standards Open source software More sharing with ICIS
Directive to make City’s data freely accessible subject to Security concerns Privacy concerns Terms of use
Open Data Website
September 2009 launch About 50 layers, many hard to find Mostly static datasets, limited formats FME used for most of initial creation
January 2010 launch About 100 layers, all easy to find Most in DWG, KML, SHP with VanMap groupings Weekly updating (thanks to FME software) Can display in Google Maps or Bing Map easily ICIS-specific data groups for GIS & AddressBC
Demo
The Road Ahead Website
Map in AJAX Viewer of MGE using SDF3 files Atom and GeoRSS feed data generated Source data: SQL Server and Oracle Spatial FME ETL process to update SDF3, SDF2 &
feed data hourly 7am-7pm Mon to Fri
The Road Ahead feed data being widely used Feed data with Street View via data
streaming
Safe Software in Use
FME Desktop The City has gradually increased its pool of
licenses to 10 Being used for IMS, Open Data, Road Ahead,
and a variety of day-to-day data ETL needs
FME Server Acquired in November 2009 Important foundation component of our GIS
software infrastructure Moving Open Data and Road Ahead to Server
Open Data Infrastructure
Requirements of source and target data: Public VanMap consistency
layers/groups specifications (filtering, grouping, naming) source formats: MDB, DWG, SDF, SQL Server, Oracle
Target Data Catalogue in OpenData Website formats: DWG, KML/KMZ, SHAPE, CSV, XLS naming convention: low casing and no spaces file structure: data folders for FTP and Web servers flexible for data packaging: via published parameters ICIS specific data specifications
Automated Execution of the Data Extraction Scripts via Windows scheduler tasks
Open Data—Processes/workflow
ETL Workspace and scripts loaded with Python pre-
processing, filtering, routing & post-processing logic
VM source
outputs
Windows Scheduler Task
Open Data: one Workbench script
shape
DWG
KML
Open Data—Automation
Scheduler Tasks for ETL Scripts (run weekly)
Open Data - Deployment
All files in KML/KMZ format uploaded weekly to the Open Data web server
All files in other formats uploaded weekly to the public FTP server
FME ETL Scripts (now Desktop) Updates for Flexible Deployment
Open Data- Challenges
Data Catalogue Design: groupings and classification
Required Data Groupings and File Structure
Naming Conventions: machine and user friendly
KML Specification Compliance/Validation Issue
Data fitness of use: attribute requests & selection
The Road Ahead Website
The Road Ahead—New Features
Hourly update replacing nightly update: FME ETL job runs hourly from 7am to 7pm
weekdays update SDF3/SDF2 (for VanMap) and feeds
data
MapGuide Enterprise AJAX Viewer uses SDF3 Atom and Geo RSS feed data generated The Road Ahead feed data replaced an email
list and is being widely used (including EOC) Feed data with Street View via data
streaming
The Road Ahead—Data Extract
Source Data SQL Server--point & street-based projects Oracle Spatial--street network reference data
Output files: SDF3 file of point & line features with 3 Road Ahead categories Atom xml file, and a GeoRSS xml file SDF2 files for existing VanMap Road Ahead layers Feed xml file with Google Street View (not in production)
FME script data extract job automation
Road Ahead— Workbench file
The Road Ahead-FME Server Deployment
The Road Ahead FME script workspace published to our FME Server (2009)
The same ETL script job is run hourly against the FME Server: "D:\FMEServer\Clients\FMEServerConsole\fmeserverconsole.exe"
RUN_WORKSPACE Roadahead_dataextract/uat_roadahead_sdf3_georss_sdf2_lp.fmw
Feed data (Atom and GeoRSS) integrated with Google Street View is accessible live via data streaming services from FME Server
The Road Ahead-FME Server Interface
http://gisweb9/fmedatastreaming/Roadahead_dataextract/prd_roadahead_georss_lp_sv.fmw
Feed data:
The Road Ahead- Challenges
FME Date Formatting: customized transformer
Feed format (RSS or Atom) and specification issues: unique ID for feed data, duplicate feed entries on disjoin hundred block streets
FME Server user account: ID required cross domain access
Authentication limitation: use FME 2010!
Summary
FME is very good solution for both Open Data and Road Ahead (and lots of other things!)
FME Desktop and Server have greatly improved the openness and interoperability of City’s GIS infrastructure platform
FME Server 2010 will expand our abilities in supporting GIS, Open3, Road Ahead and other initiatives going forward
Thank You!
Questions?
For more information: [email protected] [email protected] IT Department City of Vancouver