osmose, quality assurance tool
TRANSCRIPT
Osmose
SoTM 2013
Osmosequality assurance tool
Birmingham 2013-09-06Frdric Rodrigo (c)left 2013 - CC-BY-SA v3.0
http://osmose.openstreetmap.fr
Context
QA toolSince 2008 by Etienne Chov
Maintained by Jocelyn Jaubert and Frdric Rodrigo
Sources available on gitorioushttps://gitorious.org/osmose/
Ecosystem
No duplicate analysis from KeepRight and Osm Inspector
But integrate good idea from minor other tools
Idea from Osmose integrated into JOSM validator by don_vip (building intersection, powerline...)
Support
France mainland and oversea
HOT activated countries
All full French speaking countries and Quebec
Switzerland, Belgium, Luxembourg and Iceland
Hiring for community to support local server for new countries
The Osmose Stack
Frontend
Backends
API
Backends Input
XML or PBF export
Not all supported country are available at Geofabrik
OSM-FR generate also exporthttp://download.openstreetmap.fr/extracts/
Diff available
Need for Osmose
Analyser - SAX
Process one primitive at time
Each analyser is a pluginCheck node, way and relation
Can report errors
Use external dataWikipedia tree species
Running filter on language and country
Analyser - SAX
Format: number, date, postcode, wikipedia...
Naming: abbreviation, too many space, uppercase...
Bag tag: ref in name, typo, deprecated...
Ensure multiple tags coherence from wiki rules:multipolygon, parking lanes...
Inconsistency between tags: toll and fee, opposite cycleway without oneway...
class Structural_UnclosedArea(Plugin): def init(self, logger): self.errors[1100] = { "item": 1100, "level": 3, "tag": ["geom", "fix:imagery"], "desc": T_(u"Unclosed area") }
def way(self, data, tags, nds): if "area" not in tags or tags["area"] == "no": return if nds[0] != nds[-1]: return [(1100, 0, {})]
Analyser - Osmosis
Osmosis, a lossless schema
Multiple objects checking at once
Geospatial analysis
DBLot of indices
create/drop DB: at each Osmose run, lake of disk sppace
Keep DB: Quebec (diff mode)
Analyser Osmosis Non spatial
Done by topology
Roundabout analyserWrong highway level on roundabout
Missing oneway on link highway
Roundabout shortcut
Roundabout crossing
Analyser Osmosis - Geospatial
RoundaboutMissing junction=roundabout
Reverse roundabout
Deadend by highway levelSmall gap in highway level continuity
Power lineIsolated tower
Unconnected powerline
Missing tag power=tower
Predictive position of unmapped power=tower
Analyser Osmosis - Convergent
SoundexPhonetic index
Index all name, way kind + way name
Compare numerous name again similar with low frequency
Automatic typo on tags detectionNot work
Analyser - Merge
OpenData IntegrationIntegration vs Importation
Help contributor to add external data
Row external data set (csv) + Osmosis DB
MappingColumn to tag
Complex mapping
Analyser code is just configurationEasy to add new data set
Analyser - Merge
Conflation by ref
Conflation by distance
DetectOSM object without ref
OSM object not found in OpenData set
OpenData object not found in OSM
OpenData by OSM as sub product
Analyser Merge
French school
Wikipedia
Level crossing
self.csv_encoding = "ISO-8859-15"self.osmTags = { "amenity": "post_office" }self.osmRef = "ref:FR:LaPoste"sself.osmTypes = ["nodes", "ways"]self.sourceSRID = "4326"self.defaultTag = { "amenity": "post_office", "operator": "La Poste", "source": "data.gouv.fr:LaPoste 01/2013" }self.defaultTagMapping = { "ref:FR:LaPoste": "identifiant", "name": "nom", "wheelchair": lambda res: "yes" if self.bool[res["autonome_fauteuil_roulant"]] else "limited" if self.bool[res["pas_ressaut"]] else "no" }self.conflationDistance = 1000
Analyser OSMBin
OSMBin: Osmose proprietary DB formatFlat storage, access in O(1)
Multipolygon / boundary closed fast checking
Analyser External tools
OpenStreetBugClose OSB entries when close it on Osmose
Need support OSM Notes
Configuration
Configuration by extract (country and smaller part)
List of analysersLanguage: mainly french specific
Local way of mapping
Source available: Bing hires, French cadastre...
OptionsProjection
Driving side
Language
Boundary clipping
Extract have buffer on boundaryDuplicate error
False error (e.g. unconnected)
Clip on exact boundary polygon after analyse
About Time
Full run every 2 days
Take 80% of availible CPU
Diff mode
Speedup by running analyser only on change
Sax
Osmosischanged tables : edited nodes, ways, relations and transitive dependencies
Keep DB between runs
Not all can be run in diff modeError based on other OSM data (clustering, connection...)
Full run every month to keep safe
The Osmose Stack
Frontend
Backends
API
Errors
Backends send errors to centralized Frontend
Errors report are by Analyser and Country (aka source)
Error file replace all errors of his kind on frontendExcept on diff mode
Error
Source (country, analyser)
IdentityClass
Subclass
MetainfoLevel: 1, 2, 3 as severity
Item number: display category and subcategory
TagsOther classificatio:, by nature of OSM object, highway, waterway, name...
What is require for fix: chair, imagery, survey
Error fixes
Analyser can propose some error fixes
Add, delete or modify many tag at once
Create a new node with tags
The Osmose Stack
Frontend
Backends
API
Frontend - Map
SelectorLevels of severity
Items
Other options available on URL (see API)
MarkersStatus: open, close, false-positive
JOSM remote
JOSM fix
Wiki help
LayersHeatmap
Debug map
Stats
Errors listing
History graph
By userRSS
Update Matrix
France - Level 1
Add new analyser
Natural fix over the time
Heavy fix
Haiti All errors
Correct error with locals
France Cadastre building
overlap
New gapdetection
Heavy fdix
French OpenData School
Bugs
OpenData Update
The Osmose Stack
Frontend
Backends
API
API
External fixing tools
Stats
JOSMconfidential plugin
Quality Assurance Tools script
API doc on WikiURL parameters apply any where in the frontend
Hardware
Backendsosm3.openstreetmap.fr HOT and Quebec
osm5.openstreetmap.fr France
osm6.openstreetmap.fr France
Iceland server: Iceland
Frontendosm103.openstreetmap.fr osmose.openstreetmap.fr
Conclusion
osmose.openstreetmap.fr o.osmfr.org
https://gitorious.org/osmose/
We need youTranslation: frontend UI and backend errors
Run your own Osmose local backend server
Errors correction!
Long list of analysers request on french Trac
Localise language or country specific analyserMain st Main street
le-de-France - Total time 3h17Colonne E
Download2
Osmosis Import50
Indexing35
SAX Analyser43
Osmosis Analyser67