A year of OSM - video showing editing activity during 2011
In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):
You can see the video in HD on Vimeo. Happy New Year!
How Did He Do That?
This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.
The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.
With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.
Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.
I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it. Finally I added the sound track again by using mencoder and uploaded it to Vimeo.
Mappy New Year, everybody! May your year be filled with good GPS reception, legible survey notes, and prompt changeset uploads.
Ed Freyfogle of Nestoria has been involved in the OpenStreetMap
We’d like to welcome two new members to the board of the Humanitarian OpenStreetMap Team:Schuyler Erle – Long time pioneer of open Source geo-hacking, co-founder of the OSGeo Foundation and co-author of “Mapping Hacks”. Of course he is also a long time OpenStreetMapper and was an early proponent of OpenStreetMap for disaster response. John Crowley – Instrumental in helping with the release of imagery for use in OpenStreetMap, requiring patience and strategy in interfacing between big government and more informal organizations. With his connections John will be able to help shape the HOT for funding and future collaborations.
Schuyler Erle and John Crowley contributed massively to the process of enabling and encouraging OpenStreetMap use in disaster response after the Haiti earthquake. See them together (along with Jeffrey Johnson) in this inspiring Where 2.0 presentation in 2010 (14 min video) as they look back at how OSM helped in Haiti.
The new board were elected (as announced on the HOT mailing list) along with accepting nominations for new HOT members, on Wed 14th December. Thankyou to all those who took part!
Photo of Schuyler Erle is CC-BY-SA-2 Chris Fleming
CloudMade has found a way to add value for people playing mobile games, generate additional revenue for game developers and create a very compelling way for brands to drive retail traffic and revenue.
Mobile games are a huge marketing opportunity
Millions of users are having fun playing mobile games every day. This creates a huge marketing opportunity for consumer brands.
Banners are not the perfect solution
The only way brands and game developers are really taking advantage of this huge marketing opportunity is through banner ads adopted from the web. The banners do add revenue for game developers, but most gamers don’t like seeing banners in the middle of their game. In addition, the banners are rarely the right tool for consumer brands and retailers to achieve positive branding, retail traffic and an increase in revenue.
We need to find a tool that ensures that game developers make money and that brands get a great marketing tool in a way that people playing mobile games love.
CloudMade has solved this challenge by talking to leading brands and top game developers and building a service that meets their specific needs. The solution we have come up with is Sponsored Locations.
We look forward to talking to you too
If you are a game developer or a consumer brand that we have not yet talked to about tailoring Sponsored Locations to YOUR needs, we look forward to connecting with you soon. In the meantime, you can learn more about Sponsored Locations here.
The World Health Organization (WHO) has asked HOT and the Stand By Task Force (SBTF) to activate to map health facilities in in Libya.
This is not an emergency activation, but still very important. The basic point is to get a final Health Facility Registry GIS layer for Libya, with the location type, status and name of health centers across the country. This helps WHO as the starting point for return and increase of crucial health services in Libya; geographic information is crucial for the Libyan health system to manage its services for benefit of all citizens.
Volunteers are asked to collect all available information on the net that can be used to create the registry. This Activation will end on January 15. Much of the work will take place in a shared Google Doc to start, with import into OSM after completion. This makes it easier to coordite with SBTF volunteers.
1) Email myself (mikel_maron yahoo com) and request to join the activation. Include your Skype id, if you’d like to join the public Skype chat. I’ll respond with links to the spreadsheets we’re using.
2) Fill out a GForm with your information. You need to fill this form only the first time you start working, you do not need to do this every time.
3) Go to the GSpreadsheet and start looking for Health Facilities in Libya. You DO NOT need to fill in all the information if you do not find them: try to add as many info as you can find but do not get crazy if you cannot find one of the requested info. Other volunteers will look for it. The information already contained in the spreadsheet needs to be completed and checked.
Under the License/Origin column, you will see different data sets with different licenses for any individual point. Some are ok to use with OSM (VRAM_WHO, ICRC), and some are clearly not (gmapmaker). OSM, of course, wants data thats free and open license wise. So as HOT, we can also contribute by finding an open source for places already mapped in un-free sources.
Otherwise, the task should be clear? We can discuss on the HOT list here, but please do not share the spreadsheets publicly for now.
As previously announced, we are working with the UK’s Department for Transport to make advanced cycling data attributes available for incorporation into OpenStreetMap.
Rather than organising this along the lines of a bulk import, we are taking advantage of new technologies in Potlatch 2 and have commissioned Andy Allan, creator of OpenCycleMap, to develop new features to allow volunteers to collaborate on inspecting and merging the information into OSM.
This merging tool will also be of use for other external data that could be manually inspected and merged into OpenStreetMap.Background
The DfT commissioned survey work in various cities around the UK for their Transport Direct product. In 2011 they released the results of the surveys as Open Data, in a complex GML format based on Ordnance Survey ITN data – unsuitable for use with OSM. However, in addition, they have funded work to convert the survey data to be based on OSM geometries suitable for incorporation. This has been done through brilliant work by Ralph of CCG.
The kinds of things surveyed include cycle routes, cycle parking, cycle lanes and their widths, surfaces widths and lighting conditions of cycleable paths, and so on. We are working to add support for these attributes into CycleStreets, so that routes are further improved.
In the UK wide areas of the cycling infrastructure have been mapped in OpenStreetMap, often more recently than the data from the DfT. Also, with the development of Vector Background layers in Potlatch 2, there was an opportunity to create an improved process for dealing with external datasets.
Further background information is available in blogs and on the mailing lists.
We’re pleased to announce that a demo is now available, and we’d like people to test it.
A demo is now available. It contains sample data for Nottingham and Cambridge, but it’s deliberately unable to save the data back to the main OSM server. When the final version of the data conversion is complete and available, this will be updated and fully able to work.
Two test areas are currently loaded:
The merging process works as follows:
Feel free to play around with this – the snapshot data is being reloaded from time to time as we get better imports, although we think we’re almost at a stage where the data conversion is fairly bug-free.
The merging tool is currently a beta and further improvements are planned. See the main Merging tool page on the OSM wiki.
The first screenshot shows the thick gray line (DfT data, as a background layer) highlighted. It shows the attributes it has:
The second screenshot shows what happens if we now control-click (or cmd+click on a Mac) on the OSM line – we now get a merging interface where we can accept/reject each attribute, and click the button at the end to accept all the changes:
Feedback on the data
We would really welcome feedback as to any errors you spot in the data conversion. The aim is that the data is pre-processed and snapped to the OSM geometry as effectively as possible, so that merging is merely a case of manual confirmation of each attribute according to your local knowledge.
Issues we have fed back so far on are:
A number of software components are used to make all this work
The data is expected to be released under the Open Government License. We have been seeking an early letter of confirmation from the DfT on this and will update this page and the OSM Wiki accordingly. (The ITN-referenced dataset is released under the OGL already.)Feedback
We’d really appreciate it if you could try out the beta and add comments below, or contact Andy Allan with any feedback you have. Did you figure out how to use the tool? Did you manage to merge some data? What doesn’t work? How could the tool be improved?
If you have local knowledge of the areas in question, it would be great to hear back from you on the datasets themselves – do they match reality? Are the tags appropriate?
Soon after the earthquake near Van, Turkey in October OpenStreetMap volunteers began mapping. Without new satellite imagery available this task was not going to be very effective. Fortunately Suha Ulgen contacted put me in touch with Sean Lowery from DigitalGlobe. There was a lot of imagery available for the area effected by the earthquake, just OpenStreetMap did not have access to it. That is where DigitalGlobe’s FirstLook service came in.
In support of the Turkish Earthquake response, DigitalGlobe donated access to its FirstLook product soon after the event. The FirstLook service, was served online to OpenStreetMap editors through a WMS service. DigitalGlobe describes the service as “The FirstLook product leverages DigitalGlobe’s constellation of three very high resolution satellites to minimize any wait time for post event data. FirstLook has delivered coverage of over 70 events of high public interest worldwide to date in 2011.”Allow those volunteering for the response to see highly accurate and detailed both pre and post event satellite imagery. The Humanitarian OpenStreetMap Team created areas of interest utilizing the Tasking Manager to direct people to areas where imagery was available and digitizing concentration was needed.
This was not the first time that DigitalGlobe has donated access to the FirstLook service in response to an event. The first was after the tsunami in the Sendai area of Japan earlier this year. Without access to high resolution imagery OpenStreetMap would not be able to assist in response to these events. Thanks DigitalGlobe for providing it in response to the October earthquake in Turkey and thank you to Suha for putting us together for this response.To see fully how much data was added checkout this visualization from ITOWorld, depicting edits over the past 90 days.
Contribute to the OpenStreetMap developer community by getting involved in the Top Ten Tasks!
OpenStreetMap is huge, with an extensive and varied community. Our data is used in applications for specialty and general audiences, for devices common and rare. The infrastructure that we rely upon, as members of the OpenStreetMap community is continually improved in ways more- or less-visible and with more or less celebration.
As an example, we’ve improved our friends recently. This is a small, visible improvement in the OSM web site. You might just think, “Wow, how did we go so long without this function?” If you are logged into the web site, now you can view the recent changesets by the contributors that you have added as your friends.
This new feature sprang from a discussion on the talk@ list two weeks ago. Toby Murray made the suggestion. Mikel Maron liked the idea so much that he wrote some code and Tom Hughes refined Mikel’s patch then merged it into the rails port so that we can all use it.
Photo ©R.Weait, used by permission.
November 28th, 2011 – December 12th, 2011
A summary of all the things happening in the OpenStreetMap (OSM) world.
Did we miss something? You can contact us via email@example.com
MapQuest's rendering of national parks around Cape Canaveral.
We’ve had a busy year, and we’d like to thank everyone who has helped out with the CycleStreets project.
As you can see, since our launch in March 2009, usage has grown very considerably this year.
We’ve planned almost 1.4 million cycle journeys, covering some 23 million km as the crow flies, equivalent to cycling to the moon 60 times!
It’s been incredibly heartening to see so many great comments from users of CycleStreets on how it’s helped them find better routes or start cycling.
The year for us has been dominated by a number of funded major projects.
Work for the bicycle industry to create a Leisure Routing mode for the great cycle satnav app, Bike Hub (which uses our routing). Leisure routing is a new kind of routing, exclusive to the app, which enables you to plan a circular route lasting a specified time or distance. Read more about this great new addition to an already brilliant app.
Work with the Department for Transport to open up their cycling data. We’ve been helping the DfT with their very laudable objective to open up the cycling data they collected as part of their Transport Direct system. This detailed cycling attribute data has the potential to enhance the quality of cycle routing, by adding things like surface quality, cycle lane widths, missing pieces of data. We’ve funded some development work on amerging tool for the main OpenStreetMap editor, Potlatch 2, which we hope will be of wider benefit for other datasets also. The data is now almost ready for release following a lot excellent work by the DfT and its main contractor – stay tuned.
Cyclescape, a comprehensive online campaigning toolkit to assist cycle campaign groups around the UK. We’re able to create this thanks to funding from GeoVation. Earlier this year we won a Dragon’s Den -style contest to create a website that would build on our Photomap of 30,000 images, and provide a way for cycle campaign groups around the UK to work more effectively to solve cycling infastructure problems in their area. The website will be launched shortly, and we’re moving into beta with it after months of solid work. You can read more about this on the Cyclescape blog.
Work with Cycling Scotland to improve cycle journey planning in Scotland. We’ve been undertaking a range of projects to help motivate improved data collection and usage. This work will be finished shortly, and will see a journey planner on the Cycling Scotland website and a new guide to assist local communities with data collection for OpenStreetMap.
Hosted journey planners for London Cycling Campaign, West Sussex County Council, Bike Hub and others soon to launch! Read more about our offering of customised cycle journey planner sites and see the brochure for Local Authorities we created.Other projects
As well as these funded projects, we’ve been working on several other areas of our core offering:
Continual improvements in route quality: We’re continually tweaking the routing engine to make the routes better and better, partly in response to the great feedback we get from users. In the summer we added partial support for surface quality data (something we’ll be building on soon), and earlier we undertook some work to make routes more ‘sticky’ to the Sustrans and Local Authority cycle networks where they exist.
Faster routing: The routing engine is now much faster than a year ago. Even when we’ve had a load spike, the hosting has barely flinched!
Mobile apps on all the major platforms have been released. As well as our free iPhone app, which has seen many improvements over the year, we’ve released a well-received and fully-featured Android app and a Mobile web site which works on Blackberry and other platforms. All of these are open source projects, and we encourage people to join our brilliant volunteers who have been working on the apps so far.
A lot of work on resilience: Simon has worked extremely hard since the summer to improve the resilience of the system in many ways: Backup, fallback, easier importing of data, scripting various administrative tasks and so on.
Points of Interest: We’ve got a new set of webpages and an API for points of interest (shops, cash machines, leisure locations, etc.). We hope to integrate this much more closely with other parts of our offerings in coming months.
Lots of other little changes: Addition of CO2 and calorie counts, an increase in the journey planning limit to 200 miles, little usability fixes, reworking of the Photomap pages, railway station codes in searches, new Photomap listing modes, promotional flyers you can order, clickable icons for the London Cycle Hire website, a ‘cycle to us’ link facility, an integrated map editor installed, etc.
The main areas we’re going to be working on are:
CycleStreets wouldn’t have been possible without the brilliant OpenStreetMap (OSM) project, whose data we use. OSM has gone from strength to strength, with many areas of the country seeing far more detailed data than this time last year.
It’s a volunteer project which anyone can contribute to. So whether there’s a newly-added cycle lane near where you live, a bike shop has opened, or whether the surface of a path is unsuitable for cycling, you can contribute that information – just edit away.
If you’ve appreciated the work of the people who’ve created the map, please donate to OpenStreetMap to help keep it running fast.
Yesterday I pushed a bunch of changes to my OSM/Aerial Imagery Hybrid Style, see the (demo). I spent a bit of time on suburb and city labels. They are a really important feature of a map aimed at non-experts, in that when you are looking at the whole of Sydney and want to find a suburb
Actually making a non-trivial style like this has defiantly made me realise the difficulties.
There are a lot of improvements that can be made (either by modifying or making new components) to the OSM Mapping conventions, osm2pgsql/imposm, mapnik library, carto language stack. That said, the current form is still great and you can still make great maps. But,I would like to be able to but this needs have labels for large bays at lower zooms than small bays need bays mapped as closed ways covering their area, rather than a point in the centre define a linear function for the size of icons. i.e. at z10 the icon is 10px, at z20 the icon is 20px, now linearly interpolate all sizes for zooms in between either build this functionality into the carto language, or make another higher level macro like language which you can code this in which is then compiled into carto render spread text inside a riverbank needs functionality in the mapnik rendering engine Static OSM Tiles
I’m still yet to find a tile server which is fast and works well with lighttpd (nginx would probably suffice too). As an experiment I decided to pre-render a bunch of tiles for my hybrid style sheet. This tile layer doesn’t need to be minutely updated, anyone who needs that can use the normal mapnik layer. Also static tiles server straight from the webserver should be pretty fast (maybe on memcached tiles would be faster) and I wanted my tiles to be fast.
Next up how can I generate these static tiles? There is the popular generate_tiles.py, but that won’t render meta tiles, seems like such a waste to render every tile with a buffer of 128px when I would render a 5 by 5 meta tile of the same buffer for only 36% of the total pixels rendered. The larger the meta tiles the larger the latency, but if I’m pre-rendering them all than latency doesn’t matter any more.
So I wrote a C++ program as my replacement for generate_tiles.py. I also programmed it to render from a list of meta tiles rather than a bbox. This means for my demo I can only render high zooms where there is nearmap coverage. This is where https://github.com/andrewharvey/OSMTileListFromGeometry/ came in, which pulls nearmap coverage areas from an osm2pgsql database, and generates a list of meta tiles.
Using this method I rendered up to and including zoom 17, composed of 22444 5 by 5 meta tiles or 561100 regular tiles in a time of,real 198m17.021s user 103m49.985s sys 37m36.117s
with disk usage,<1M 0-8 2.0M 9 6.8M 10 26M 11 98M 12 13M 13 36M 14 114M 15 411M 16 1.5G 17 ======== 2.2G total
I think that time could still be sped up with,
Also a small note on git which I’ve been using a lot (thanks to the free hosting by github) and I think git rebase is awesome. For example I did git commit -ammend instead of git commit --amend (I knew there was a duplicate somewhere but because I seem to have a tint of dyslexia I confused the duplicate — with mm).
Of course this resulting in committing all files which had changed with a message of “mend” as a new commit. I did this twice, and only noticed after I had already made a bunch of correct commit after. With git rebase I could pop some commits of the commit stack, remove the two “mend” commits fix the commit which I should have amended to, and pop my other commits back on top of the stack.
Thanks to http://stackoverflow.com/a/180085.Next Up