worldmap: a spatial infrastructure to support teaching and research (brown bag talk)

A Spatial Platform to Support Research and Collaboration

http://worldmap.harvard.edu

Ben Lewis blewis@cga.harvard.edu

MITCambridge, September 2014

Outline

• Why lat/long is valuable and what research support operations should do about it

• WorldMap as straw man– Overview of current system– New projects

• Global services registry (OGP integration?)• Dataverse integration• HarvardX integration• Neatline integration

• Discussion

Lat/long (location) as organizing facet

(http://www.architectmagazine.com/technology/meet-the-geodesigner.aspx

Spatial search

• Improve access to both spatial and non spatial materials with a “spatial visualization facet”– Books, documents– Paper maps– Local spatial data– Remote spatial data – Local map display and processing services– Remote map display and processing services– Wikipedia– Social media – tweets, etc– The web

Build platforms that lower barriers to:

• Finding spatial materials which reside inside or outside the academy

• Visualizing this data• Using this data• Sharing this data and views of it• Mashing up private and public data• Spatializing materials that are not yet spatial

Support spatial and spatio-temporal visualization

• Make it easy to visualize datasets on a map(A great way to see temporal patterns is on a map http://worldmap.harvard.edu/tweetmap )

• Make it easy to mashup private and public data• Make it easy to share spatial data and views of data• Make it easy to crowd curate data

Support the research life cycle

* Research Lifecycle:• Scoping• Data gathering / exploration• Synthesis / analysis• Writing / communication• Publishing / delivery

Archiving?

WorldMap In a Nutshell

• Designed to lower barriers for researchers who wish to use spatial technology

• Web-based, cloud hosted• Open source software• Service oriented architecture

Allows researchers to…

• Organize: their own (large) mapping datasets and share them

• Visualize: maps with data-driven symbology• Publish: data to the world or to just a few

collaborators• Mashup / Combine: one’s own data with data

provided by others• Analysis: basic but can be easily extended• Collaborate: by letting several people edit the same

map

12,000 Registered Users14,000 Data Layers3,500 Map Collections800,000 Visits

Statistics

Traffic By City

Built on Open Source Software

Loosely Coupled Approach for Adding Major Capabilities

MapWarper Neatline

DataverseEdX

New

WMS WMS

http

Service Registry

CSW

Contributing Organizations(WorldMap and GeoNode)

• Boston Area Research Initiative - BostonMap• UNICEF – Education Access in Cameroon• Cornell University – Global Health Map• UN University – Wildlife Enforcement Monitoring• Virtue Foundation – Women in the World• Amazon - Hardware• Others…

• World Bank – GFDRR, Dominode, Risiko• U.S. State Dept. – ROGUE, HIU, Syria Damage Assessment• NOAA - GeoCloud• UN World Food Program – WFP Geonode• Australian Govt. - AIFDR, TsuDAT• MapStory Foundation – MapStory App• Others…

GeoNode

WorldMap

Niche between desktop and web

Analysis

Ease

of U

se

Collaboration WorldMap

web apps

desktop apps

Openness is key

• Open registration• Open access to data• Open service protocols (WMS, WFS, ESRI Rest)• Open data formats (Shape, GeoTIFF, GeoRSS,

KML, Json, CSV)• Open source code (GPL on github)• Runs on open source operating systems

(Linux) and could run on Windows

Data Connectors to…

• Google maps, Open Street Maps, Bing, MapQuest, ESRI• Geo-tweets• Google Street View• Google Earth• Flickr, GeoRSS• Geonames, Google Places, Yahoo Places• Social Explorer, Yelp• WMS, WFS, ESRI REST• More to come…

Examples…

• Map applications people and organizations have created using WorldMap…

Reischauer InstituteJapan Earthquake Archive

Boston Area Research InitiativeBoston Data

University of BarcelonaHistoric Planning Data

Professor Colin GordonMapping Decline, St. Louis

Virtue FoundationWomen-run NGOs

What’s Coming in WorldMap

• Services Registry• HarvardX Integration• Dataverse Integration• Neatline Integration• Geo-tweet Archive

WorldMap and the Services Registry

• WorldMap is a web-based, open source, collaborative mapping platform developed at Harvard CGA since 2012

• The National Endowment for the Humanities Implementation Grant. – Key objective: create a comprehensive and

sustainable map service registry which researchers and the public can use to discover, create and share any work that can be represented spatially.

Lower the Barrier to Geodata Access, Across Disciplines

Internet

Commercial Systems

local data

Government Systems

Other Institutions

Esri REST, WMS, WFS, RSS etc.

dow

nloa

d

download

download

uplo

ad service view

servi

ce vi

ew

WorldMap Data and Service

Registry

Key Goals• Support discovery of the millions of web maps

that are exposed but not easy to find (the “dark” geoweb)

• To allow anyone to mashup content from any source from within any mapping application

• Enable non-IT professionals to create their own map services without IT support

• Crowd-source map data curation in a metadata-weak environment

Creating a Global Service Registry

-- A basic piece of geo-infrastructure that doesn’t exist

• Build registry of web map services (millions of map layers)• Make API available so any system can use it• Provide a fast, faceted search interface• Allow anyone to add new services to the registry• Maintain uptime statistics on each service• Use WorldMap usage statistics to improve search

(eventually bring in stats from systems outside WorldMap which use API)

How Many Services Are Out There?

• We estimate millions, each containing many map layers totaling petabytes of data which is currently VERY hard for the average researcher to find and use.

• Try this to estimate number of Esri REST servers (15million)– allinurl: http "arcgis rest services" mapserver -test -kml -kmz -

sitemap -query

• Try this to estimate number of WMS servers (47 thousand)– allinurl: http "?request getcapabilities" -test

Service Registry Challenges

• Metadata - tagging and usage statistics• Projections - cascading• Persistence - caching• Discovery - central index, usage statistics• Performance - caching

Open API to Registry

• Public, RESTful API• Access all (public) map layers within WorldMap• Access all service layers outside WorldMap• Access all Maps (collections of layers) within

WorldMap• Search on information:

– Metadata– Usage statistics– Attribute info (for local layers)

Distributed Map

Services

Services

API

ServiceRegistry

Distributed Users

WorldMap

OpenLayers,Leaflet

Esriclients

Any map client

Find and bindto layers

Crowd curation, user submitted services

WorldMap

Service caching,

reprojection

WorldMapLocal

Services

UptimeChecker

WorldMapCore

ServiceCrawler

*Common Crawl

*Start with hadoop search of Common Crawl dataset http://commoncrawl.org/

Faceted Services Search UI Mockup (draft)

More information…

WorldMap http://worldmap.harvard.edu

Center for Geographic Analysis http://gis.harvard.edu

Ben Lewisblewis@cga.harvard.edu

HarvardX Integration

Dataverse Integration (Social Science Archive)

Neatline Integration map-based story telling platform

TweetMap Archive

Demo of Service Registry Test Server

• http://107.22.231.227/– Show services– Add service– Find service layers as layers– Metadata for services– Layer page for services– Statistics for services– Saving map with service layers

Service Registry Architecture

End

Information Exists Within Spectrum of Publicness and Curatedness

• PublicnessPrivate - Group – University – Consortium - Public

• CuratednessData created – Reviewed by expert – Reviewed by crowd

Our job

• Create platforms (of people and software and objects) to lower barriers to data access:– Data hosting– Data formatting– Metadata creation– Data and view sharing– Search tools – local and remote resources– Connecting systems where relevant– Analytical services– Publishing