open data portals: 9 solutions and how they compare
TRANSCRIPT
Open Data Portals9 Solutions and How They Compare
Dale
Lut
z
Stew
art H
arpe
r
Why Open Data? And Why Now?
Citizens are demanding access to open data like never before.
We want it.
Great advances have been made in technology. Citizens are tech
savvy and aware.
We have the ability.
Better decisions are being made by the mass population.
Benefits are undeniable.
Why does this matter?
Build innovations on top of institutes and empower citizens across the globe.
What is Open Data?
“Open data and content can be freely used, modified, and shared by anyone for any purpose”
opendefinition.org
“Open Data is free, public data that can be used to launch commercial and nonprofit ventures, do research, make data-driven decisions, and solve complex problems.”
Open Data 500
Open Data Audiences● Citizens who want to examine the data
and answer questions they have.● Researchers and journalists who want to
gather and analyze the data to tell stories.● Developers who want to use the data to
build applications
Source: City of Winnipeg
Who
Pro
duce
s Op
en D
ata?
All levels of governments
Intergovernmental Organizations
Non Governmental Organizations Private Companies
Academic Institutes
Governments are doing better
PDFs make for poor Open Data
spatialityblog.com/2011/08/31/mapping-hurricane-nyc/
NYPD Example
Released collision data
in PDF
Citizen created a PDF Scraper Tool so the data could
be read
NYPD responded and their
data is now on Socrata
Limit public interaction
Ingredients for Open Data Portal Success
UpdatingDatasets should not be static and
should be regularly updated.
How to keep your data up-to-date:
Provide your data as a published feed (e.g. RSS) or API.
Connect your open data platform directly to the master database.
Enlist the help of FME to sync your master data store with your open data repository.
Coordinate SystemProjections
(Spatial Data Only)
Projection Support● Advanced users should be able to choose their
projection. ● Local (e.g. State Plane or British National Grid)
and global projections should be provided. ● Globally we would recommend:
● WGS 84 Lat/Lng (EPSG: 4326)
● Spherical Mercator (EPSG: 3857)
Ensure Data Quality
Pay Attention To Data Quality● Ensure you’ve covered off our Data Quality
Checklist○ FME can automate the checking
● Check out csvlint.io -- a free CSV quality checker
Delivery MechanismsFormats / APIs
Data should be available in...
Tabular:● CSV● JSON (API)
Spatial: ● GeoJSON (API)● Shapefile
Tabular:● Excel● XML
Spatial: ● AutoCAD DXF/DWG● Esri File Geodatabase● GML● KML● Mapinfo TAB
Suggested: Supplementary:
9 Solutionsfrom A-Z
9 Solutionsfrom A-Z
ArcGIS Open DataAmazon Web Services (AWS)CKANDataPressDKANFTPGitHubJunarOpenDataSoft Socrata
ArcGIS Open DataConfigure your own branded Open Data site
on top of ArcGIS Server or ArcGIS Online.
OPENDATA.ARCGIS.COM
Popular choice for Esri users.Open data builds directly on top of your published ArcGIS services.
Supported data types:● Hosted (AGOL) feature
services● ArcGIS for Server
feature services● ArcGIS for Server map
services
● Image services● CSVs● Other - Web maps, URLs,
Word docs and PDF● Other services via Koop
Data and metadata can be viewed in the browser.
Can create simple histograms, line, donut or scatter charts to look for patterns without downloading the data.
All data is downloaded in WGS 84, and you can also download the data as KML, Shapefile or via the API (JSON,
Geoservice, WMS).
Demo
Sample AGOL Hosted Site
Amazon Web ServicesRun/roll your own powerful open data platform.
AWS.AMAZON.COMwith help from WordPress & FMECLOUD.COM
By leveraging the lower-level services of AWS (e.g. S3, EC2, RDS) and making use of FME Cloud as a data-mover, you
can produce an extremely fault-tolerant, scalable and powerful service that is
easy to maintain and cost effective.
Demo
Data Migration Project: GeoStor
CKANOpen source data portal - providing tools to streamline publishing, sharing, finding and
using data.
CKAN.ORG
Leading open source data portal with over 300 open source data management extensions. Particularly
suited for large organizations.
Easy data uploading. Fast search experience.
A rich JSON API allows for integration with FME.
CKAN can plot geographic data in an interactive map.
Real World:The City of Surrey partnered FME Server to allow any CKAN dataset to be downloaded in any format and any projection.
DataPressWordPress and CKAN seamlessly integrated
and hosted on the cloud.
DATAPRESS.IO
Fast and simple data publishing
WordPress integration allows you to write blog posts, design pages, and manage menus and content in a
simple web interface
- Hosted on the cloud -
DKANComplementary offering to CKAN. Open
source Drupal plugin with a cloud-hosted version available.
NUCIVIC.COM/DKAN
Integrates CKAN features into DrupalSimple to deploy and maintain
Complete set of content management featuresEspecially suited to anyone using Drupal
Self-hosted or cloud version available
Real world: used by data.gov.uk
Simple file catalog service.
Free hosting with sites like DataHub.io, FTP, Google Drive and GitHub is a good place to start with Open Data.
Cloud file storage solutions can be used to store and serve large volumes of data.
Web interface can be built on top of the storage system.
Demo
FTP
Demo
GitHub
JunarCloud open data platform with focus on ease of use, powerful analysis and visualizations.
JUNAR.COM
SaaS PlatformProvides a standard data portal as well as APIs that enable
developers to integrate data into their applications.
● Government of Chile● City of Sacramento● City of Palo Alto
Real World
OpenDataSoftFocus on ease of use, automated API
generation and interactive visualizations.
OPENDATASOFT.COM
Proficient working with large datasets, as
they leverage Elasticsearch, which ensures near real-time search and analysis.
SaaS platform
Geospatial format support
Data publishing and management via live dashboards
SocrataA platform that turns data into a utility that can be discovered, consumed, visualized,
analyzed, and shared.
SOCRATA.COM
Easy to publish data to Socrata using a WebUI, desktop sync tool, or API
Upload CSV, Excel, and TSV files natively with support for Shapefiles, KML, KMZ, and GeoJSON
There is both a published and working copy of the dataset
Many tools around metadata management and workflow
Efficient search
Rich set of tools allow you to visually inspect data
Metadata is not front and center: the focus is on the information within the data itself
Impressive analytics toolsPublic Data as a Utility
Demo
Socrata (Chicago)
Solutions TakeawayThere are many top quality open data
solutions available. Do your own research and see which one is right for you.
Hackathons (i.e. BigApps NYC 2015)
Normalization of data at national level so we can compare cities
globally
Open Private Sector
Focus shift: data quality, not quantity
More cities & governments following NY
The Future of Open Data
No tech excuses
Questions? We have answers.
Thank you
Want more resources about moving data?
blog.safe.comknowledge.safe.comwebinars.safe.com