maps2graphs: a socially scalable method for generating high-quality gis datasets based on google...

.

Problem Statement.Big Problem is.....

.

... that it is hard to come by realistic road maps for large-scale ITSsimulations.

.Small Problem is.....

.

... that even with Google Maps, quotas make it difficult to create large datasets

alone -- need social scaling (Google Maps quota: 2.5k reqs/day)

• EV battery replacement infrastructure simulations 03

• supply chain research -- again, battery replacement infrastructures

• autodrive simulation? -- in real life by around 2020?

• ... anyway who needs realistic road maps

03 myself+1 "...EV Use in Environments with V2H and Battery Replacement..." ITS Tokyo (2013)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 2/22...

2/22

.

Traditional GIS

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 3/22...

3/22

.

GIS: The Shapefile• default in GIS datasets today

• a non-GPS = special format -- see ArcGIS manual 11• precision problems -- .SHP is all about relative shapes, not coordinates

11 Esri "ArcGIS Japan" http://www.esrij.com/products/arcgis (2013)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 4/22...

4/22

.

ArcGIS versus Google Maps• ArcGIS is

◦ paid◦ needsdatasets to work◦ address2location and placename2location are restricted by

quotas (credit)

◦ no APIs -- just plugins for PowerPoint and smartphone apps

• GoogleMaps is

◦ free with quotas, pay9$/m for nearly infinite quotas◦ all data is in one place

◦ many APIs -- Location, Direction, Renderer, ...

◦ address2location and placename2location are seamless -- no need tocare which you use

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 5/22...

5/22

.

Google Maps : Route Datatype

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 6/22...

6/22

.

Objectives

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 7/22...

7/22

.

Objectives

• anwebapp which would create a graph -- all-to-all roadmap -- from alist of locations on top of Google Maps

• application has to be cloud based -- like Google Drive, Dropbox, etc... todistribute work socially

◦ terminology: serviceless app is a webapp that runs without a webserver

◦ otherwise need support from Google itself -- which is unlikely

• application has to scale socially -- full range from 1 to 10k collaborators

• rapid generation of datasets◦ example: need graph of 1k locations by tomorrow!

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 8/22...

8/22

.

Definitions

• socially scalable: same design for any number of collaborators

• serverless webapp:1. a webapp that runs from a cloud drive2. uses cloud drive to store data3. can be shared by many collaborators

• dataset: a graph (nodes and links) of routes among many locations

1. easily merged from many pairwise routes

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 9/22...

9/22

.

maps2graphs: design

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 10/22...

10/22

.

Basic Idea

Full RESTful APIUser

URL-based API

Web

app

Dat

a

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 11/22...

11/22

.

More Detail

Cloud Drive API

Collaborator

Owner

Serverless WebappUser-Generated Content

Content API

Mas

ter c

opy

Slav

e co

py

(syn

c)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 12/22...

12/22

.

maps2graphs: Datatypes

v11 v11 v11 v11 v11

v21 v21 v21 v21

v31 v31 v31

v41 v41

hA2A matrix of 5

Cloud Drive

becomes...0100?001001?01task.status.txt

task.list.json[“address 1”,“address 2”,“address 3”,….

]

base64( JSON as string)….

[collaborator].[time].[random tail]

…M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 13/22

...

13/22

.

maps2graphs: Batch

Collaborator Cloud Drive API Content API

Newbatch

Randomize

new file (chunk)

task.list.json

task.status.txt

vxy

write chunk(one line)

finalize file (set name)

read/fill/updatetask.status.txt

• specific forDropbox --Dropbox JS API in 01

• randomization is important tominimize overlaps with othercollaborators

• atomic: status string isupdated task.status.txt is updatedonly when the batch file issuccessfully closed

• continuous: batch bybatch

01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 14/22...

14/22

.

maps2graphs: social tricks

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 15/22...

15/22

.

Social Features

• maps2graphs is a crowdsourcing effort

• much literature on crowdsourcing 04 05 06 07

• important feature is distribution: most people contribute very little, fewcontribute a lot

• translation for maps2graphs: most people will use the app for a fewminutes, few will run is for days -- just leave it running in the browser

04 G.Chatzimilioudis+3 "Crowdsourcing with Smartphones" IEEE J. Internet Computing (2012)

05 B.Yoo+2 "Travel Time Estimation Using Mobile Data" J. Transportation Studies (2005)

06 C.Wu+3 "Crowdsourcing Multimedia QoE Evaluation..." IEEE Trans. on Multimedia (2013)

07 D.Sui+2 "Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)..." Springer Book (2013)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 16/22...

16/22

.

Actual Stats (later example)

0

70

140

210

Batc

h siz

e

Clie

nt 1

0

70

140

210

Batc

h siz

e

Clie

nt 2

0

70

140

210

Batc

h siz

e

Clie

nt 3

0 20 40 60 80 100 120 140Hour (relative to project start)

0

70

140

210

Batc

h siz

e

Clie

nt 4

.Aggressive Approach..

.

... high intensity until out of quota,

then hibernate until next 24H cycle

• high intensity: 1 req every 6 seconds

• quota: 2500 req/day per IP

• out of quota roughly in 3-5 hours

• stay slightly below quota for personal use

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 17/22...

17/22

.

maps2graphs: implementation

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 18/22...

18/22

.

Implementation

• maps2graphs: source code in 01

• working code at 02 --- you can run it right now!

◦ or just go tohttp://tinyurl.com/maps2graphs◦ 300 Family Marts in Northern Kyushu

01 myself "Maps2Graphs Project Page" https://github.com/maratishe/maps2graphs (2013)

02 myself "Maps2Graphs: Example Crowdsourcing Project" http://tinyurl.com/maps2graphs (2013)

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 19/22...

19/22

.

Screenshot

• simple interface

• shows current volume (of current batchfile)

• shows details about current and pastroutes, quota status, etc.

• if you need to abort, press commit andgo

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 20/22...

20/22

.

End Result

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 21/22...

21/22

.

That’s all, thank you ...

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...

22/22

.

[01] myself (2013)Maps2Graphs Project Pagehttps://github.com/maratishe/maps2graphs

[02] myself (2013)Maps2Graphs: Example Crowdsourcing Projecthttp://tinyurl.com/maps2graphs

[03] myself+1 (2013)...EV Use in Environments with V2H and Battery Replacement...ITS Tokyo

[04] G.Chatzimilioudis+3 (2012)Crowdsourcing with SmartphonesIEEE J. Internet Computing

[05] B.Yoo+2 (2005)Travel Time Estimation Using Mobile DataJ. Transportation Studies

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...

22/22

.

[06] C.Wu+3 (2013)Crowdsourcing Multimedia QoE Evaluation...IEEE Trans. on Multimedia

[07] D.Sui+2 (2013)Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI)...Springer Book

[08] Google (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core

[09] Google (2013)Google Maps APIhttps://developers.google.com/maps/

[10] (2013)JSON formatwww.json.org

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...

22/22

.

[11] Esri (2013)ArcGIS Japanhttp://www.esrij.com/products/arcgis

[12] W3C (2013)HTML5W3C Candidate Recommendation at http://www.w3.org/TR/html5/

M.Zhanikeev -- maratishe@gmail.com -- Maps2Graphs: Social GIS Datasets Based on Google Maps API -- http://tinyurl.com/marat131204a --- 22/22...

22/22