talk: "using open data and crowdsourcing to develop cyclestreets"
TRANSCRIPT
Using Open Data and
Crowdsourcing to develop
CycleStreets
Martin Lucas-Smith CycleStreets.net
@CycleStreets
What does CycleStreets do?
Cycle journey planner Online service, 2m journeys so far
Photomap / Cyclescape Campaigning tool
CycleStreets: who? Simon Nuttall Routemaster
Martin Lucas-Smith Webmaster
… and various people helping out in various ways!
CycleStreets: history Cambridge-only cycle journey planner
Originally written for Cambridge Cycling Campaign
Launched June 2006
Google Map –based 5,000 lines drawn over
satellite imagery
Google doesn’t give you data: just cartography
47,000 journeys planned
15,000 photos added
CycleStreets: history Lots of requests for same thing in other places
around the UK
Result is CycleStreets
We are using OpenStreetMap for our data
We don’t have money for an OS license
Community aspect really important anyway
OpenCycleMap cartography
Went to public beta in March 2009
2m journeys so far
Mainly word-of-mouth so far
CycleStreets: UK-wide
CycleStreets Journey planner
[Quick demo]
http://www.cyclestreets.net/
OpenStreetMap
Google (North Cambridge)
OpenStreetMap (North Cambridge)
“OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world.” - Wikipedia
Collaborative:
Jul 2007: 9,000 people; April 2012: Almost 600,000
Project:
Not just a map - mass of ideas, processes, data, outputs
Free:
Free financially and Free as in open
Editable:
Constantly changing
Of the world:
Global, not just UK where it started
OpenStreetMap “OpenStreetMap creates and provides free geographic data such as street maps to anyone who wants them.
“The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways.”
OpenStreetMap UK – Ordnance Survey:
Very high quality, but ...
Cost can be prohibitive
(particularly voluntary sector)
Derivative data restrictions
Ordnance Survey has claimed derived data rights when you place something over one of their maps
Incompatible with direction of the Internet, where data is being ‘mashed’ together to make useful information and visualisations
Central control – change slower
Crowdsourcing principle
“Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.”
http://crowdsourcing.typepad.com/
Everyone knows a little bit about something in their area. Put that together and you get:
OpenStreetMap
OpenStreetMap website default style
OpenStreetMap
Cloudmade ‘Fresh’ style (#997)
OpenStreetMap
Cloudmade ‘Googley’ style (#5138)
OpenStreetMap
OpenCycleMap
OpenStreetMap
OpenCycleMap
OpenStreetMap
CycleStreets data view
OpenStreetMap
CycleStreets data interrogation
OpenStreetMap
http://tolu.giub.uni-bonn.de/karto/osm-3d/Screenshots/Dresden/Dresden2.jpg
OpenStreetMap
Glosm 3D (Russia)
OpenStreetMap
http://wiki.openstreetmap.org/wiki/File:Rostock-warnemuende.leuchtturm.osm-3d.jpg
OpenStreetMap
http://wiki.openstreetmap.org/wiki/File:Seamap.png
OpenStreetMap
OpenStreetMap
http://opengeodata.org/pretty-osm-derived-art-maps
OpenStreetMap
Urban accessibility of Castelfiorentino
OpenStreetMap
Bike Hub app, uses CycleStreets routing
OpenStreetMap
First tactile map based on OSM data published on May 12, 2009
CycleStreets Journey planner
OpenStreetMap
Marikina Mapping Party cake (4th Mapping Party in the Philippines)
Data collection Structured ground surveys – main source Ground surveys, performed by a mapper
On foot, bicycle or in a car or boat.
Usually collected using a GPS unit
Government data sources Landsat 7, US TIGER data, OS OpenData
Commercial data sources AND from Netherlands
Traced from satellite imagery e.g. Yahoo!, Microsoft Bing have donated
Objective data OSM is a store of objective data
Everything must be verifiable
Subjective data is not welcome
Subjective assessment is the realm of the consumer of the data E.g. Cycle journey planner decides on the likely
niceness of a street based on objective attributes like speed limit, width, surface quality
My cycle to work would be different to my mum’s: we have different preferences for a ‘good’ route
OpenStreetMap
ITO World animation 'OSM 2008 - A Year of Edits'
Data collection
Mapping takes place individually or in groups
Ground surveys
Individuals or groups survey using GPS and taking notes
Made easier by GPS technology 2000: Bill Clinton switches on wider GPS
availability
Mid-2001: GPS units available for $100
2004: GPX standard (GPS data transfer) widespread
Photo attribution unknown – please do contact us if you know
Mapping parties A group of openstreetmappers and novices
Go to area & map it exhaustively, usually over a weekend
Dividing up an area between participants and mapping it
Mapping by car, cycle or walking
Social aspect important: people can meet up and talk (usually at a pub) between mapping sessions
Photo: David Earl
Photo attribution unknown – please do contact us if you know
Mapping parties
e.g. Walking Papers: Print current state, annotate, load back in http://walking-papers.org/
Photo attribution unknown – please do contact us if you know
Social context
Social context important Community decides on data collection and structure norms
appropriate to their situation
The mapkibera project is training locals people of Kibera, Nairobi to create a map with OpenStreetMap
Technologies used depend on circumstances
Map Kibera
Social context
Importing other people’s data? Massive debate within the OpenStreetMap community
(Assumes donated data is compatibly licensed)
One view: importing data gives the impression that an area doesn’t need to be mapped in person and reduces volunteer input
TIGER data import in US very problematical http://www.slideshare.net/harrywood/wherecampeu-session-state-of-the-states-in-openstreetmap
Another view: importing data gives a massive head-start and means we can get into much more detailed mapping
Data creators vs Data consumers have different perspectives
CycleStreets needs a reasonably complete map!
Social context
Is objectivity always possible? WikiProject Gaza
Practical issues
How do you represent a location where only some people can enter/exit?
Photo attribution unknown – please do contact us if you know
Social context
How do you represent a location where only some people can enter/exit?
Photo attribution unknown – please do contact us if you know
Social context Crisis Mapping:
WikiProject Haiti
Before January 12, 2010
Then NOAA, GeoEye, DigitalGlobe flew planes over the area, and donated their imagery for tracing purposes People around the world at their computers contributed to effort
Roads, buildings and refugee camps of Port-au-Prince mapped in just two days
“The most complete digital map of Haiti's roads”
Haiti The resulting data & maps have been used by
several organisations providing relief aid, such as the World Bank, the European Commission Joint Research Centre, the Office for the Coordination of Humanitarian Affairs, UNOSAT, others
Informal data structure
No formal specification of how to represent things
No database schema – just key-value pairs
Reflects the social context of the users
Users make it up as they go along
Communities of interest norms
Conventions established, then stability
User/collector cycle embeds the convention
Informal data structure
Nodes & Ways, Tags
http://wiki.openstreetmap.org/wiki/Map_Features describes the (many) conventions formed so far
Examples Motorway represented as: “highway=motorway”
Local street: “highway=residential”
Guided bus! “highway=bus_guideway”
Fence: “barrier=fence”
Cycleway: “highway=cycleway”. But what type?
“cycleway=lane”
“cycleway=track”
“cycleway=opposite_lane”
POIs: “amenity=postbox”, “shop=charity”
Not to forget... “amenity=pub”
Adding data
Potlatch 2 – www.openstreetmap.org (www.geowiki.com)
Adding data
Potlatch 2 – www.openstreetmap.org (www.geowiki.com)
Potlatch 2 editor
[Quick demo]
http://www.cyclestreets.net/edit/
Adding data
JOSM – Java OpenStreetMap Editor – advanced users
Adding data
ArcGIS plugin for OpenStreetMap (free)
The ArcGIS Editor provides: • Simple tools to upload and download OSM data • An OSM-compatible geodatabase schema to locally store OSM data • An OSM symbology template for faster editing • Conflict-resolution tools for reconciling data back to the OSM database
OSM / Google Maps Google doesn’t provide any data – just a picture
Also doesn’t always have information needed by cyclists/walkers – park paths, cut-throughs, pubs! (Though is improving)
OSM Google maps
OSM vs Ordnance Survey Depends what scale
Question is intended use
“Good enough” notion OSM will never be good
enough for utility companies needing exact location of pipes
But for many other uses, OSM appropriate and good enough
Sutton Coldfield B72:
OSM vs Ordnance Survey
Costs money – not free
Big difference is the license – not free (libre)
Plot points on a map and the OS claim some rights to that
Derivative data issues
Major problem in the age of the internet, where data is being shared, mixed, repurposed
By contrast, OSM uses a Creative Commons license
Challenge to traditional mapping agencies
Ordnance Survey seeing more competition
OSM and internet sharing more generally forcing a change in business models?
Lowering data use costs
Lowering data collection costs
Forcing derivative data restrictions to be removed
Challenge in the small-scale map data area
Opens new opportunities
Businesses like Microsoft, Google and others presumably spend a small fortune on mapping data
Bing Maps (Microsoft) and MapQuest (AOL) now actively putting money and resources into OSM project
Perhaps speculatively
OSM will provide them with a cheaper way of providing data with far fewer restrictions in future?
Quality assurance issues
Can we trust open data?
Depends whether it’s ‘good enough’ for your use
Can we trust formalised data?
Tales of lorry satnavs for instance
Balance between accuracy and speed/volume
Arbury Park in OSM as it was built – others slower
Quality around the country variable
How can we ascertain this?
Vandalism
But there’s the ability to watch an area for changes
More people = more vigilance or more vandalism?
Difficulties we face with OSM
Coverage not uniform
Lack of quality control: makes harder to engage Local Authorities
Vandalism a concern for some though not in practice
Subjective data?
Maybe in future: lack of static IDs – unique numbers for features change
Ability to engage local mappers when an area is deficient
Many of these problems will go away as OSM matures
Challenge to traditional cartography
Cartography is a major area of interest within the OpenStreetMap community
Cartography is becoming more automated as Web 2.0 steams ahead
http://maps.cloudmade.com/
3D vector rendering – send data to device, not bitmaps
Cloudmade map renderer demo
[Quick demo]
http://maps.cloudmade.com/ Click ‘Edit map style’
Click on a design to start from Click ‘Clone Style’ in the bottom-right
Use the ‘Object Visibility’ box on the right to remove/add features
At the heart of the OpenStreetMap project is a database holding all the map data that people work with.
Left: editors people use to enter data into the database
Right: all sorts of interesting uses for the data, e.g. ...
OpenStreetMap ecosystem
Non-commercial
Commercial / profit-making use absolutely fine As long as people adhere to the license, i.e. give attribution and
allow downstream users to share/re-use the data
Maps of very many kinds
Web routing
SatNav devices
Data analysis (e.g. accessibility analysis)
Placefinding
GPS background
Humanitarian
...
OpenStreetMap uses
OpenStreetMap: Summary
Applies the Wikipedia approach of crowd-sourcing
Extremely flexible
Free (cost) and Free (libre)
Challenging traditional map agencies / business models and government funding models
Communities of interest and norms
Much scope for research
Varied uses: maps, electronic devices, humanitarian, ..
CycleStreets using it
As more data goes in, more uses, so more people add data, so more people use it, so ...
CycleStreets Cycle journey planner
Journey planner: features Plan route from A-B, anywhere in UK
Simple user interface (we hope!) Click-click-plan, and simple Namefinder
Gives set of route choices (fastest, quietest, balanced)
Takes accounts of hills
Turn-by-turn directions
Photos-en-route
Journey planner: features Distance, time, CO2 avoided, Calories
Google Street View at any point
Localised versions for easy linking E.g. cambridge.cyclestreets.net
Link methods E.g. www.cyclestreets.net/journey/to/cb1+2py/
‘Fly in Google Earth’
Export to GPS
Feedback system
Photomap
Photomap: features Icons on map (per type of feature)
Click to view image and info
Add photo
Crowdsourcing: lots of people, but each donating a small effort
Categorisation
E.g. “Show me all the cycle parking problems in Cambridge”
Mobile Key features on small
screen
iPhone app
Android apps
Mobile HTML5 app
All open source – help welcomed!
Jakob Nielsen: “Best Application Designs” - April 2012 (Lightweight Applications category
Mobile
Other apps now incorporating our routing
API - data interface
Bike Hub – great world-first iPhone bike real-SatNav
In the leading Boris Bike app, ‘London Cycle’
Why? Fundamentally, we want to see “More people
cycling, more safely, more often”
New cycle users face many challenges in UK:
Poor infrastructure, traffic hostility
Confidence cycling (address with training)
Cultural/identity issues: not yet mainstream
Lack of utility bikes in shops
Routes – different to car routes!
We try to tackle the last problem
... and the first (through the Photomap)
How a routing engine works
• Find route with lowest score, i.e. least ‘friction’
• ‘Shortest path algorithm’ - Standard problem in computer science, we use A* method
How it works (briefly)
1. Data comes from people collecting data on-street for OpenStreetMap
Remember: Is factual data only – e.g. presence of road, surface, type
NOT “I think this is a nice cycle route”
2. We take OSM data ‘off the shelf’
Though we’re part of the community in practice
Import several times a week: fresh data
Conversion process is complex – interpreting the data
How it works (briefly)
3. Score each type of path:
4. Take account of hills (add/remove penalty)
5. Account for turn delays (work ongoing)
6. Take account of detailed cyclist behaviour (ditto)
How it works (briefly)
7. Compress the network, to make the system much faster (system called ‘Cello’):
Park: 4 nodes & 7 ways After: 3 nodes & 3 ways
8 9
9
A
B C
D
A
B C
4
10
6
3
6: BC
7: AD,BD 9: AC
How it works (briefly)
So each path / road / shortcut / etc. now has a score
Higher score = worse for cycling (more ‘friction’)
User comes to the site
8. Find the lowest total score from A to B
9. Route is found
10. Repeat for quietest, fastest modes – each have different scores
11. Routes shown to user
Draw over the cartography We are using OpenCycleMap by Andy Allan
‘Tiles’ which form a static background once a route has been planned – i.e. we just put this behind a line we have calculated
Route feedback goes to OSM contacts
Getting involved: open sourcing
All 3 mobile apps now open-sourced
Main journey planner being open-sourced
Latest update at http://cycle.st/b2221
Codebase currently harder to install than it should be
Currently modularising more heavily
Converting to Git
Cyclescape open-source
Very keen for greater involvement
www.github.com/cyclestreets
Transport Direct CJP www.transportdirect.info/Web2/JourneyPlanning/FindCycleInput.aspx
£2.4 million (from tax)
92,000 journeys planned (dated Jan 2011, total now = ??)
£26.09 per journey
£1m – budget for 2011
32 areas (professionally surveyed)
CycleStreets www.cyclestreets.net
£28k
458,000 journeys planned (dated Jan 2011, reached 2m as of now)
6p per journey
£130k needed
UK-wide (but depends on OSM completeness)
Back in January 2011...
UKGov ... BUT things have now moved on a bit
We’re working with the DfT through their data contractor to get data into OSM – funded project
DfT have been very receptive to the open data potential
We think cycle journey planning is most effective when done by local people using Open Data
Merging tool created
We continue to work to ensure that CycleStreets is solution of choice
Big Society –compliant We tick all the boxes:
Collaborative: involves local people
Low cost: datasets have no license fee, agile delivery
Trusted: for the people, by the people
Open Data
Citizen involvement: combines skills and input of large numbers of people (collecting data)
Quality delivery: problems can be fixed easily
Transparency: more people oversee the data and spot problems or potential improvements
http://www.green-alliance.org.uk
Cabinet Office
Was main feature on data.gov.uk for months!
Local Authorities www.cyclestreets.net/localauthorities
http://cyclejourneyplanner.westsussex.gov.uk/
www.cyclingscotland.org
UK Collision Map www.cyclestreets.net/collisions
Cyclescape Campaigning toolkit
For campaign groups around UK
Match, using geography, who is interested in what
blog.cyclescape.org
Ruby on Rails (new for us!)
Really would welcome coders github.com/cyclestreets/toolkit/
