About me • OpenStreetMap: Mapper since 2010 • OpenStreetMap Foundation Japan: Vice President since 2014 • Main activity: Community support • OpenStreetMap Foundation: member since 2013 • Daily Work on JIPDEC • OpenData & Geospatial consultant
This guy on SNS
Today’s Agenda 1. General information about Import. 2. Past Imports & Current relationship on Japan region. 3. Import methodology (Based on my experience).
General Information about Import
Abstract • Importing (also known as Bulk Importing) is the process of uploading external data to OSM. • [snip] • Unlike editing the map or uploading GPX traces, importing combines an existing dataset with the OSM dataset, and thus typically includes a complex merging process. ——Quote from OSM wiki - import
Import? It’s easy thing. Install “opendata" JOSM plugin and put shapefiles onto it, then UPLOAD!!
3 perspectives about import 1. Technical 2. Operational 3. Legal File format convert Dedicated account License compatibility Tagging scheme Organize a task team Special permission You have to meet those requirements
Oh WHY? I have a data and want to import to improve the map!
Some tragedy in past…
Unnecessary tags From “Worst of OSM”
Row quality From “Worst of OSM”
Misalignment From “Worst of OSM”
Duplicate entry From “Worst of OSM”
So OpenStreetMap has import guidelines
Procedure of import • Step 1 - Prerequisites • Step 2 - Community Buy-in • Step 3 - License approval • Step 4 - Documentation • Step 5 - Import Review • Step 6 - Uploading From “OSM wiki import/Guidelines” Note: “Community” mentioned above include both local community & international community. So it is mandatory to be reviewed from international members on “Imports" Mailing List
History of imports on Japan Region
Japan National OpenData Strategy start Import Guideline is discussed and improved 2008 2010 2012 2014 2017 MLIT: Kokudo Suuchi Joho (国⼟土数値情報, KSJ) data import, 2008-2011 Yahoo Japan!/ALPS: Road data import, 2011-2014 MLIT: KSJ Municipality boundary re-import, 2014-2015 Municipality : City planning OpenData, 2013-2016 • Urayasu City building footprint • Sabae City building footprint • Sizuoka City building footprint • Tondabayashi City road and building footprint
Current relationship from OSM community • OSM community has major relationship with • Open Data advocates • Researchers, Civic tech activists, Developers • Civic, NPO activities • Town walk activity • Government Officers • Geospatial Information Distribution (Japanese geo data platform) • Local government officers • Central government officers (especially MLIT, GSI)
Major Data from GSI • Traceable license of their GSI MAP (地理理院地図) tiles. • Base map, disaster response • GSI map has special advantage for accurate positioning (due to it is created under Survey law quality assurance). • And most important, it is distributed XYZ tile format!!
GSI data which we could not use • OSM could NOT import Raw data from GSI. • Survey Data (測量量成果) like Fundamental Geospatial Data (基盤地図情報) is protected under Survey law. • This data is super useful, but very hard to tackle to…
Current relationship with Local governments • Some local governments are viewing OpenStreetMap as… • use case of their Open Data. • Free (sometimes “as beer”, “less cost”) map data platform • Touchpoint for Civic activity (esp. through Mapping Party) • Each municipality has different view on OpenStreetMap. • It should be acceptable. • At first stage, OpenStreetMap have to make our data useful to civic, includes those governments.
Challenge for next step Officers at local government = “Local knowledge specialist”
Import methodology (Based on my experience)
3 perspectives about import 1. Technical 2. Operational 3. Legal File format convert Dedicated account License compatibility Tagging scheme Organize a task team Special permission
File format convert • Projection convert: GDAL (ogr2ogr) • File Format & encoding convert: ogr2osm • Especially shapefile -> osm xml • Shapefile attribution to OSM tag convert • I recommend Paul Norman’s version • Tell your FOSS4G friends for more details :) Technical
Tag Scheme • Investigate original data column to determine which column to be used. • Need discussion on talk-* ML and imports ML. • Most of original column is USELESS. Technical
Tag Scheme sample See OSM wiki Technical “Import/Catalogue” for more details.
ogr2osm script sample Technical
Dedicated account • Make a import dedicated account to separate your work from your own account. • [sample] • Urayasu_bld_import • helicoysfh_tondabayashi_import • KSJ2_adm_bnd_imprt • It is very easy to revert if some trouble happens. Operational
Organize your Team • Collaborative import is becoming popular. • Tasking Manager is a great tool to manage collaboration of team members. • e.g. imports by HOT team, Mapbox team, local communities and so on… • This could be reduce tasks for 1 person. • And it helps training other members. Operational
In the other words…
Assemble your Sentai!! CC BY 2.0 Kyoryuger Roll Call GIF, flickr Operational
Or you have to be a Lonely Hero… CC BY, Kevin Dooley “batman” on flikr
TIPS at work • Merging with existing data is the most hardest work. • Divide tasks for small amounts. • Especially if it is about building or address import. • OSM US team has published some tools regarding address conflation during NYC and SF import. • Validation is mandatory. Operational
License compatibility • ODbL has very limited license compatibility Legal
Legal perspective • CCBY is not compatible with ODbL. • Taiwan Open Government Licence is not compatible as well. (Japanese one is also) • So you have to ask government if we try to import the data into OpenStreetMap. Legal
Special Permission • OpenStreetMap Foundation has published a template for asking permission. • Based on CC BY 4.0 publication (also 3.0, 2.0). • 2 items of incompatibility • Attribution on “Contributor” Page (Attribution is not signed to OSM data itself) • Technical restriction on Downstream (e.g. DRM. This is obviously different) Legal
NON! • Think different :) • Application to officers = Making a touchpoint. • A chance to introduce, discuss and include them. • A chance to walk together!
逆境 逆境ナイン 3巻表紙より
Import is hard. But it has very effective to improve data coverage. And also effective to improve collaboration.