OSM metadata description: the data behind the data

By Monday July 24th, 2017Data

At Oslandia, we like working with Open Source tool projects and handling Open (geospatial) Data. In this article series, we will play with the OpenStreetMap (OSM) map and subsequent data. Here comes the fifth article of this series, dedicated to the extraction of OSM metadata starting from the OSM history data.

1 Extract OSM data history

1.1 An air of “deja vu”

In previous articles, dedicated to OSM chronological evolution and OSM tag set analysis, we saw examples of OSM data parsing. Here we will use a similar parser to get the OSM history, i.e. every versions of every objects for a given area within the OSM API.

import osmium as osm
import pandas as pd

class TimelineHandler(osm.SimpleHandler):
    def __init__(self):
        osm.SimpleHandler.__init__(self)
        self.elements = []

    def add_elements(self, e, elem_type):
        self.elements.append([elem_type,
                              e.id,
                              e.version,
                              e.visible,
                              pd.Timestamp(e.timestamp),
                              e.uid,
                              e.changeset])

    def node(self, n):
        self.add_elements(n, 'node')

    def way(self, w):
        self.add_elements(w, 'way')

    def relation(self, r):
        self.add_elements(r, 'relation')

tlhandler = TimelineHandler()
tlhandler.apply_file("../src/data/raw/bordeaux-metropole.osh.pbf")
colnames = ['type', 'id', 'version', 'visible', 'ts', 'uid', 'chgset']
elements = pd.DataFrame(tlhandler.elements, columns=colnames)
elements = elements.sort_values(by=['type', 'id', 'ts'])

If you read the previous article, this parsing class should be familiar to you. Let’s briefly recall its meaning: we read a native OSM file bordeaux-metropole.osh.pbf with the help of the pyosmium library, we store elements and basic information like versions, timestamps, user ids (and so on…) into a simple Python structure. In the end, using a Pandas DataFrame is even clearer. This structure can be saved on the disk, or handled directly within the Python workspace.

1.2 Towards the metadata

Here we have the OSM data history corresponding to the area of Bordeaux. It is largely descriptive, however it says nothing about data quality in itself.

We could compare the OSM data with some alternative data source, however there is no real consensus about the perfectness of any alternative data set. Knowing that, we choose to analyse metadata associated with OSM elements instead of data themselves.

Some data agreggation operations are needed in this way, to better describe the OSM object evolutions.

2 What do we mean by metadata?

Here we plan to go deeper than just considering the geometric data: some additional information is available if we check the history of contributions, and if we focus on different changesets, users, or even on meta-information about OSM elements themselves.

2.1 Element metadata

That’s sound quite evident on the first sight, but why don’t we just look at the OSM elements? Two major drawbacks prevent us from working with such objects:

  • in reality, no real add-on is made in terms of quality assessment: variables associated with elements are number of versions, of tags, of contributors and so on… It provides very interesting pictures of OSM data, but stays largely descriptive.
  • it’s fairly intensive in terms of computing resources! We got around 3 millions OSM elements only for the Bordeaux area, we let you imagine for a whole country, a continent or even for the entire world…

2.2 Changeset metadata

Investigate on changesets could provide an additional information source. We know that elements are modified in the context of an opened changeset.

We can count the number of modifications, assessing if some new elements have been created during a given changeset, or count the amount of elements that have been modified or even deleted.

Classify changesets is possible: we could hypothesize that there are “productive” changesets, where a large amount of nodes, ways and/or relations are modified, and where modifications are durable from one hand; and less productive ones (we don’t say “useless”!) with less creations, less modifications and less durable elements.

However the information may be gathered more efficiently by considering those who produce changesets: the contributors themselves.

2.3 User metadata

We can hypothesize that a user who contributes a lot, on every kind of OSM elements, and whose contributions stay valid for a long time (or even: are still valid!) is an experienced user; and the elements on which he has contributed are well-represented.

The link between users and elements is more natural than between changesets and elements : it is possible to characterize OSM data quality by considering which type of user contributes the most to each element. In a more simple way, we can consider the most experienced user who’ve contributed on an element as a flag about the element quality. The quality of an element may also be indicated by the type (more or less experienced) of its last contributor.

This last hypothesis will be our central thread in the next section (and… *spoiler warning* in the next articles!).

3 Extract user metadata

We decide to go deeper into the analysis of user contributions. As a reminder, we’ve extracted OSM data history in the first section of this blog article. We can pick one element randomly as an example:

elements.sample().T
                             61781
type                          node
id                       260953178
version                          3
visible                       True
ts       2010-07-31 18:23:54+00:00
uid                          53048
chgset                     5364764

Here comes the time to consider each user.

3.1 Time-related features

We begin with the time they’ve spent on OSM.

user_md = (elements.groupby('uid')['ts']
            .agg(["min", "max"])
            .reset_index())
user_md.columns = ['uid', 'first_at', 'last_at']
user_md['lifespan'] = ((user_md.last_at - user_md.first_at)
                        / pd.Timedelta('1d'))
extraction_date = elements.ts.max()
user_md['n_inscription_days'] = ((extraction_date - user_md.first_at)
                                  / pd.Timedelta('1d'))
elements['ts_round'] = elements.ts.apply(lambda x: x.round('d'))
user_md['n_activity_days'] = (elements
                              .groupby('uid')['ts_round']
                              .nunique()
                              .reset_index())['ts_round']
user_md.sort_values(by=['first_at'])
user_md.query('uid == 4074141').T
                                         1960
uid                                   4074141
first_at            2016-06-06 14:25:01+00:00
last_at             2016-06-09 12:39:47+00:00
lifespan                              2.92692
n_inscription_days                    258.186
n_activity_days                             2

With these short code lines, we have gathered some temporal features in order to know how each user contributes through time. In the provided example, the user with the ID=4074141 is registered as an OSM contributor for 258 days; his lifespan on the OSM website is almost 3 days; he (or she!) made modifications at two different days.

3.2 Change-set-related features

Then we can focus on change-set-related information. By definition, each user has opened at least one changeset (yes, even you, if you’ve contributed!)! Let’s construct a small changeset metadata DataFrame:

chgset_md = (elements.groupby('chgset')['ts']
              .agg(["min", "max"])
              .reset_index())
chgset_md.columns = ['chgset', 'first_at', 'last_at']
chgset_md['duration'] = ((chgset_md.last_at - chgset_md.first_at)
                          / pd.Timedelta('1m'))
chgset_md = pd.merge(chgset_md,
                     elements[['chgset','uid']].drop_duplicates(),
                     on=['chgset'])
chgset_md.sample().T
                              33265
chgset                     44138773
first_at  2016-12-03 15:02:58+00:00
last_at   2016-12-03 15:02:58+00:00
duration                          0
uid                         4867435

Each changeset is associated with its starting and ending time, its duration (in minutes) and the responsible user. We then may associate a changeset quantity and mean duration time for each user.

user_md['n_chgset'] = (chgset_md.groupby('uid')['chgset']
                       .count()
                       .reset_index())['chgset']
user_md['dmean_chgset'] = (chgset_md.groupby('uid')['duration']
                           .mean()
                           .reset_index())['duration']
user_md.query('uid == 4074141').T
                                         1960
uid                                   4074141
first_at            2016-06-06 14:25:01+00:00
last_at             2016-06-09 12:39:47+00:00
lifespan                              2.92692
n_inscription_days                    258.186
n_activity_days                             2
n_chgset                                    3
dmean_chgset                          22.2778

Wow, there are some new interesting information there: we know that user 4074141 had produced three changesets during its lifespan, and the mean duration of these changesets is around 22 minutes.

3.3 Contribution intensity

Then we observed on some preliminary observation that some users were so productive that they modified some elements several times; a typical bot-like behavior if this amount is large, or simple auto-corrections? We can add this information as follows:

contrib_byelem = (elements.groupby(['type', 'id', 'uid'])['version']
                  .count()
                  .reset_index())
user_md['nmean_modif_byelem'] = (contrib_byelem.groupby('uid')['version']
                                 .mean()
                                 .reset_index())['version']
user_md.query('uid == 4074141').T
                                         1960
uid                                   4074141
first_at            2016-06-06 14:25:01+00:00
last_at             2016-06-09 12:39:47+00:00
lifespan                              2.92692
n_inscription_days                    258.186
n_activity_days                             2
n_chgset                                    3
dmean_chgset                          22.2778
nmean_modif_byelem                    2.94061

Oh-oh… Our nice user 4074141 seems to modify OSM elements almost three times. That’s quite few to conclude to his bot nature, however he seems quite unsure about the contributions he makes…

3.4 Element-related features

In order to characterize how the user contributes, a lot of additional features are still missing. The most important ones are related to the amount of modifications.

newfeature = (elements.groupby(['uid'])['id']
              .count()
              .reset_index()
              .fillna(0))
newfeature.columns = ['uid', "n_total_modif"]
user_md = pd.merge(user_md, newfeature, on='uid', how="outer").fillna(0)
newfeature = (elements.query('type == "node"').groupby(['uid'])['id']
              .count()
              .reset_index()
              .fillna(0))
newfeature.columns = ['uid', "n_total_modif_node"]
user_md = pd.merge(user_md, newfeature, on='uid', how="outer").fillna(0)
newfeature = (elements.query('type == "way"').groupby(['uid'])['id']
              .count()
              .reset_index()
              .fillna(0))
newfeature.columns = ['uid', "n_total_modif_way"]
user_md = pd.merge(user_md, newfeature, on='uid', how="outer").fillna(0)
newfeature = (elements.query('type == "relation"').groupby(['uid'])['id']
              .count()
              .reset_index()
              .fillna(0))
newfeature.columns = ['uid', "n_total_modif_relation"]
user_md = pd.merge(user_md, newfeature, on='uid', how="outer").fillna(0)

user_md.query('uid==4074141').T
                                             1960
uid                                       4074141
first_at                2016-06-06 14:25:01+00:00
last_at                 2016-06-09 12:39:47+00:00
lifespan                                  2.92692
n_inscription_days                        258.186
n_activity_days                                 2
n_chgset                                        3
dmean_chgset                              22.2778
nmean_modif_byelem                        2.94061
n_total_modif                                1832
n_total_modif_node                           1783
n_total_modif_way                              46
n_total_modif_relation                          3

Ok! This user is very active to map the Bordeaux area! He proposed 1832 modifications, amongst which 1783, 46 and 3 were respectively dedicated to nodes, ways and relations. However the amount of modified elements should be smaller, as this user made several contributions per element, on average.

Here you can see a difference between the number of changesets and the number of element modifications. As a reminder, a changeset can include several modifications, see [[https://wiki.openstreetmap.org/wiki/Changeset][the description of OSM changesets]] on the official OSM wiki.

The number of modifications can be described even more finely! Why don’t we consider if modifications are still valid, or if other modifications arise after the user action? What about elements that have been deleted since (we consider that working on a useless element is not so valuable for the community)?

3.5 Modification-related features

Actually it is possible, however we need associating a little bit more features to OSM elements. Is the current version an initialization of the object? Is it the up-to-date version? Will it be corrected (by an alternative user or the current user himself?)

import numpy as np

osmelem_versioning = (elements.groupby(['type', 'id'])['version']
            .agg(["first", "last"])
            .reset_index())
osmelem_versioning.columns = ['type', 'id', 'vmin', 'vmax']

elements = pd.merge(elements, osmelem_versioning, on=['type', 'id'])
elements['init'] = elements.version == elements.vmin
elements['up_to_date'] = elements.version == elements.vmax
# note that 'elements' is sorted by type, id and ts
elements['willbe_corr'] = np.logical_and(elements.id.diff(-1)==0,
                                          elements.uid.diff(-1)!=0)
elements['willbe_autocorr'] = np.logical_and(elements.id.diff(-1)==0,
                                                 elements.uid
                                                 .diff(-1)==0)

elements.query("id == 1751399951").T
                                   1620248
type                                  node
id                              1751399951
version                                  1
visible                               True
ts               2012-05-13 17:53:14+00:00
uid                                 260584
chgset                            11588470
ts_round         2012-05-14 00:00:00+00:00
vmin                                     1
vmax                                     1
init                                  True
up_to_date                            True
willbe_corr                          False
willbe_autocorr                      False

Here is a short example of new enriched element definition: now we know that the node of ID 1751399951 has only one version, so this version of course corresponds to its initialization and is up-to-date. As there is no second version (until the extraction date!), the element is not (auto-)corrected yet. These features help to describe more precisely the user contributions:

def create_count_features(metadata, element_type, data, grp_feat, res_feat, feature_suffix):
    feature_name = 'n_'+ element_type + '_modif' + feature_suffix
    newfeature = (data.groupby([grp_feat])[res_feat]
                  .count()
                  .reset_index()
                  .fillna(0))
    newfeature.columns = [grp_feat, feature_name]
    metadata = pd.merge(metadata, newfeature, on=grp_feat, how="outer").fillna(0)
    return metadata

def extract_modif_features(metadata, data, element_type):
    typed_data = data.query('type==@element_type')
    metadata = create_count_features(metadata, element_type, typed_data,
                               'uid', 'id', '')
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("init"),
                               'uid', 'id', "_cr")
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("not init and visible"),
                               'uid', 'id', "_imp")
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("not init and not visible"),
                               'uid', 'id', "_del")
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("up_to_date"),
                               'uid', 'id', "_utd")
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("willbe_corr"),
                               'uid', 'id', "_cor")
    metadata = create_count_features(metadata, element_type,
                               typed_data.query("willbe_autocorr"),
                               'uid', 'id', "_autocor")
    return metadata

user_md = extract_modif_features(user_md, elements, 'node')
user_md = extract_modif_features(user_md, elements, 'way')
user_md = extract_modif_features(user_md, elements, 'relation')
user_md = user_md.set_index('uid')
user_md.query("uid == 4074141").T
uid                                         4074141
first_at                  2016-06-06 14:25:01+00:00
last_at                   2016-06-09 12:39:47+00:00
lifespan                                    2.92692
n_inscription_days                          258.186
n_activity_days                                   2
n_chgset                                          3
dmean_chgset                                22.2778
nmean_modif_byelem                          2.94061
n_total_modif                                  1832
n_total_modif_node                             1783
n_total_modif_way                                46
n_total_modif_relation                            3
n_node_modif                                   1783
n_node_modif_cr                                   0
n_node_modif_imp                               1783
n_node_modif_del                                  0
n_node_modif_utd                                  0
n_node_modif_cor                                598
n_node_modif_autocor                           1185
n_way_modif                                      46
n_way_modif_cr                                    0
n_way_modif_imp                                  46
n_way_modif_del                                   0
n_way_modif_utd                                   0
n_way_modif_cor                                  23
n_way_modif_autocor                              23
n_relation_modif                                  3
n_relation_modif_cr                               0
n_relation_modif_imp                              3
n_relation_modif_del                              0
n_relation_modif_utd                              0
n_relation_modif_cor                              2
n_relation_modif_autocor                          1

That’s a complete picture of the 4074141 user contribution, isn’t it? Amongst the 1783 modifications on node, there are…1783 improvements (so, no creation, no deletion). 598 of these modifications have been corrected by other users, and 1185 of them refer to auto-corrections; but no modification result in up-to-date nodes! We can draw a comparable picture for ways and relations. As a result, we have identified a user that contributes a lot to improve OSM elements; however his contributions are never enough to complete the element representation.

By considering every single user that has contributed on a given area, we can easily imagine that some groups could arise.

4 Conclusion

In this new blog post, we have presented some generic information about OSM contribution history. We’ve seen that user metadata can be easily built by some aggregation operations starting from OSM data history. We have proposed a bunch of features to characterize as well as possible the way people contributes to OSM. Of course a lot of other variables can be designed. We encourage you to think about it if interested in the topic!

In the next blog post, we will see how to use this new information to group OSM users, with the help of some machine learning well-known procedures.