maps2graphs: a socially scalable method for generating high-quality gis datasets based on google...

.

Problem Statement.Big Problem is.....

.

... that it is hard to come by realistic road maps for large-scale ITSsimulations.

.Small Problem is.....

.

... that even with Google Maps, quotas make it difficult to create large datasets

alone -- need social scaling (Google Maps quota: 2.5k reqs/day)

• EV battery replacement infrastructure simulations 03

• supply chain research -- again, battery replacement infrastructures

• autodrive simulation? -- in real life by around 2020?

• ... anyway who needs realistic road maps

03 myself+1 "...EV Use in Environments with V2H and Battery Replacement..." ITS Tokyo (2013)

M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 2/22...

2/22

.

Traditional GIS


3/22

.

GIS: The Shapefile• default in GIS datasets today

• a non-GPS = special format -- see ArcGIS manual 11• precision problems -- .SHP is all about relative shapes, not coordinates

11 Esri "ArcGIS Japan" http://www.esrij.com/products/arcgis (2013)


4/22

.

ArcGIS versus Google Maps• ArcGIS is

◦ paid◦ needsdatasets to work◦ address2location and placename2location are restricted by

quotas (credit)

◦ no APIs -- just plugins for PowerPoint and smartphone apps

• GoogleMaps is

◦ free with quotas, pay9$/m for nearly infinite quotas◦ all data is in one place

◦ many APIs -- Location, Direction, Renderer, ...

◦ address2location and placename2location are seamless -- no need tocare which you use


5/22

.

Google Maps : Route Datatype


6/22

.

Objectives


7/22

.

Objectives

• anwebapp which would create a graph -- all-to-all roadmap -- from alist of locations on top of Google Maps

• application has to be cloud based -- like Google Drive, Dropbox, etc... todistribute work socially

◦ terminology: serviceless app is a webapp that runs without a webserver

◦ otherwise need support from Google itself -- which is unlikely

• application has to scale socially -- full range from 1 to 10k collaborators

• rapid generation of datasets◦ example: need graph of 1k locations by tomorrow!


8/22

.

Definitions

• socially scalable: same design for any number of collaborators

• serverless webapp:1. a webapp that runs from a cloud drive2. uses cloud drive to store data3. can be shared by many collaborators

• dataset: a graph (nodes and links) of routes among many locations

1. easily merged from many pairwise routes


9/22

.

maps2graphs: design


10/22

.

Basic Idea

Full RESTful APIUser

URL-based API

Web

app

Dat

a


11/22

.

More Detail

Cloud Drive API

Collaborator

Owner

Serverless WebappUser-Generated Content

Content API

Mas

ter c

opy

Slav

e co

py

(syn

c)


12/22

.

maps2graphs: Datatypes

v11 v11 v11 v11 v11

v21 v21 v21 v21

v31 v31 v31

v41 v41

hA2A matrix of 5

Cloud Drive

becomes...0100?001001?01task.status.txt

task.list.json[“address 1”,“address 2”,“address 3”,….

]

base64( JSON as string)….

[collaborator].[time].[random tail]

…M.Zhanikeev -- [email protected] -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 13/22

...

13/22

.

maps2graphs: Batch

Collaborator Cloud Drive API Content API

Newbatch

Randomize

new file (chunk)

task.list.json

task.status.txt

vxy

write chunk(one line)

finalize file (set name)

read/fill/updatetask.status.txt

• specific forDropbox --Dropbox JS API in 01

• randomization is important tominimize overlaps with othercollaborators

• atomic: status string isupdated task.status.txt is updatedonly when the batch file issuccessfully closed

• continuous: batch bybatch

01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)


14/22

.

maps2graphs: social tricks


15/22

.

Social Features

• maps2graphs is a crowdsourcing effort

• much literature on crowdsourcing 04 05 06 07

• important feature is distribution: most people contribute very little, fewcontribute a lot

• translation for maps2graphs: most people will use the app for a fewminutes, few will run is for days -- just leave it running in the browser

04 G.Chatzimilioudis+3 "Crowdsourcing with Smartphones" IEEE J. Internet Computing (2012)

05 B.Yoo+2 "Travel Time Estimation Using Mobile Data" J. Transportation Studies (2005)

06 C.Wu+3 "Crowdsourcing Multimedia QoE Evaluation..." IEEE Trans. on Multimedia (2013)

07 D.Sui+2 "Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)..." Springer Book (2013)


16/22

.

Actual Stats (later example)

0

70

140

210

Batc

h siz

e

Clie

nt 1

0

70

140

210

Batc

h siz

e

Clie

nt 2

0

70

140

210

Batc

h siz

e

Clie

nt 3

0 20 40 60 80 100 120 140Hour (relative to project start)

0

70

140

210

Batc

h siz

e

Clie

nt 4

.Aggressive Approach..

.

... high intensity until out of quota,

then hibernate until next 24H cycle

• high intensity: 1 req every 6 seconds

• quota: 2500 req/day per IP

• out of quota roughly in 3-5 hours

• stay slightly below quota for personal use


17/22

.

maps2graphs: implementation


18/22

.

Implementation

• maps2graphs: source code in 01

• working code at 02 --- you can run it right now!

◦ or just go tohttp://tinyurl.com/maps2graphs◦ 300 Family Marts in Northern Kyushu

01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)

02 myself "Maps2Graphs: Example Crowdsourcing Project" http://tinyurl.com/maps2graphs (2013)


19/22

http://tinyurl.com/maps2graphs

.

Screenshot

• simple interface

• shows current volume (of current batchfile)

• shows details about current and pastroutes, quota status, etc.

• if you need to abort, press commit andgo


20/22

.

End Result


21/22

.

That’s all, thank you ...


22/22

.

[01] myself (2013)Maps2Graphs Project Pagehttps://github.com/maratishe/maps2graphs

[02] myself (2013)Maps2Graphs: Example Crowdsourcing Projecthttp://tinyurl.com/maps2graphs

[03] myself+1 (2013)...EV Use in Environments with V2H and Battery Replacement...ITS Tokyo

[04] G.Chatzimilioudis+3 (2012)Crowdsourcing with SmartphonesIEEE J. Internet Computing

[05] B.Yoo+2 (2005)Travel Time Estimation Using Mobile DataJ. Transportation Studies


22/22

.

[06] C.Wu+3 (2013)Crowdsourcing Multimedia QoE Evaluation...IEEE Trans. on Multimedia

[07] D.Sui+2 (2013)Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)...Springer Book

[08] Google (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core

[09] Google (2013)Google Maps APIhttps://developers.google.com/maps/

[10] (2013)JSON formatwww.json.org


22/22

.

[11] Esri (2013)ArcGIS Japanhttp://www.esrij.com/products/arcgis

[12] W3C (2013)HTML5W3C Candidate Recommendation at http://www.w3.org/TR/html5/


22/22

maps2graphs: a socially scalable method for generating high-quality gis datasets based on google...

Technology

google maps api

social gis datasets

google maps arcgis

google drive

social features maps2graphs

google mapscloud

collaborator maps2graphs

large datasets