Accessing the OpenStreetMap API with Python directly

Posted on 2024-04-16, by Racum. Tags: GIS OpenStreetMap JSON Python Shapely

Table of Contents

On a previous article called "OpenStreetMap to GeoJSON", I explained how to extract Polygons from OSM using an online tool provided by from OSM France. And, on this article, I’ll explain how do that yourself directly using the OSM API and Python.

This also explains the basic concepts I used to write the osmexp tool, to export OSM elements into GeoJSON from the command-line.

The OpenStreetMap API

Elements

There are 3 core geometric elements on OSM’s API (described on "Elements" page):

Node: a point on the surface.
Way: an open or enclosed line.
Relation: a logical grouping of the elements above.

Notice that don’t align directly with the Shapely primitives:

Point: this is the only straightforward one: a point can be mapped as a node.
LineString: a way, usually open-ended, but they could be closed as well.
Polygon: a closed way (first and last nodes are the same), but not always; there is a tag area=yes that helps disambiguate it agaist just closed ring lines.
MultiPoint: no direct translation, but can be represented as a relation with many nodes.
MultiLineString: no direct translation, but can be represented as a relation with many ways.
MultiPolygon: a relation with self-enclosed ways, sometimes marked with the tag type=multipolygon.

As you can see, the fit is not perfect, every “translation” of formats need to based on the interpretation of the data and its conventions.

Endpoints

The full list of endpoints is very comprehensive, it can manage versions, users, gps tracking, maps, etc; but, to get the geometric data, I’ll keep the scope on the GET endpoints for the main elements (node, way and relation). Also, OSM provides both XML and JSON formats, but I’ll stick with JSON because it is more Python-friendly.

Endpoint	Example	Description
Node	`.../node/2386873668.json`	Single node
Way	`.../way/331158718/full.json`	Way and its nodes
Relation	`.../relation/9596872/full.json`	Relation with its nodes and ways

There are also “single” versions of way and relation (without /full), but they only have their respective root elements.

Payloads

Base structure of all endpoints looks like this:

{
    "version": "0.6",
    "generator": "CGImap 0.9.1 (75342 spike-06.openstreetmap.org)",
    "copyright": "OpenStreetMap and contributors",
    "attribution": "http://www.openstreetmap.org/copyright",
    "license": "http://opendatacommons.org/licenses/odbl/1-0/",
    "elements": [
        ...
    ]
}

The only important field here is the elements list, everything else is just metadata. The structure of the objects listed in this field are described below:

Node Element

Notice the type = node, its numerical id, and its coordinate values lat and lon:

{
    "type": "node",
    "id": 2386873668,
    "lat": -33.8579291,
    "lon": 151.2153359,
    "timestamp": "2022-11-30T12:15:26Z",
    "version": 4,
    "changeset": 129559504,
    "user": "kurisubrooks",
    "uid": 14680934
}

Way Element

Notice the type = way, its numerical id, a list of node references on nodes, and a key-value set of tags:

{
    "type": "way",
    "id": 331158718,
    "timestamp": "2019-03-30T00:32:24Z",
    "version": 3,
    "changeset": 68683269,
    "user": "Warin61",
    "uid": 1830192,
    "nodes": [
        2386873668,
        ...
    ],
    "tags": {
        "highway": "footway",
        "surface": "concrete"
    }
}

Relation Element

Notice the type = relation, its numerical id, a list of nodes and way references on members, and a key-value set of tags:

{
    "type": "relation",
    "id": 9596872,
    "timestamp": "2023-11-15T20:48:50Z",
    "version": 24,
    "changeset": 144068429,
    "user": "pofewi7086",
    "uid": 20210316,
    "members": [
        {
            "type": "way",
            "ref": 331158718,
            "role": "outer"
        },
        ...
    ],
    "tags": {
        "addr:city": "Sydney",
        "amenity": "arts_centre",
        ...
    }
}

Python Fetching and Parsing

We’ll need to install 2 Python modules: requests to call the API and shapely to handle the geometric data:

$ pip install requests
$ pip install shapely

Code

First, lets prepare our imports and constants:

import itertools
import requests
from shapely.geometry import Point, LineString, Polygon, GeometryCollection
from shapely import get_parts, polygonize_full

API_URL = 'https://www.openstreetmap.org/api/0.6'

Before we call the API, lets prepare a function to parse nodes and ways from the “elements” list; thanks to the JSON format and the Python’s comprehensions, this function can be written in a very compact way:

def parse_nodes_ways(elements: list[dict]) -> tuple[list[Point], list[LineString]]:
    nodes = {e['id']: Point(e['lon'], e['lat']) for e in elements if e['type'] == 'node'}
    ways = [LineString([nodes[n] for n in e['nodes']]) for e in elements if e['type'] == 'way']
    return list(nodes.values()), ways

Because parse_nodes_ways() does most of the heavy lifting, this function is very small: we just call the API, parse the points (ignoring the ways with _) and return the first (and only) Point:

def fetch_node(node_id: int) -> Point:
    response = requests.get(f'{API_URL}/node/{node_id}.json')
    points, _ = parse_nodes_ways(response.json()['elements'])
    return points[0]

Fetching a way is very similar: the only difference is that this time we ignore the points (also with _) and check if the line is closed or not, to define the type to be returned (LineString or Polygon):

def fetch_way(way_id: int) -> LineString | Polygon:
    response = requests.get(f'{API_URL}/way/{way_id}/full.json')
    _, ways = parse_nodes_ways(response.json()['elements'])
    way = ways[0]
    return Polygon(way) if way.is_closed else way

The relation case is a bit more involved, we have to:

Parse the ways (ignore nodes),
Run polygonize_full() from Shapely, this returns 4 GeometryCollections matching the 4 possible cases (polygons, cuts, dangles and invalid),
Check if a case not empty and split them into their individual geometries (get_parts()),
Flatten the cases "list of lists" into a single list of geometries with itertools.chain(),
Re-combine them into a GeometryCollection().

def fetch_relation(relation_id: int) -> GeometryCollection:
    response = requests.get(f'{API_URL}/relation/{relation_id}/full.json')
    _, ways = parse_nodes_ways(response.json()['elements'])
    cases = [get_parts(p) for p in polygonize_full(ways) if p]
    parts = list(itertools.chain(*cases))
    return GeometryCollection(parts)

Of course, this is a naive approach: we are not checking the tags set by convention to disambiguate LineString, Polygons and MultiPolygons described earlier, but this is beyond the introductory scope of this article.

Usage

Here they are: the OSM geometry types, as Shapely objects, ready to be used inside Python:

>>> fetch_node(2386873668)
<POINT (151.215 -33.858)>

>>> fetch_way(331158718)
<LINESTRING (151.215 -33.858, 151.215 -33.858, 151.215 -33.858, 151.214 -33....>

>>> fetch_relation(9596872)
<GEOMETRYCOLLECTION (POLYGON ((151.215 -33.858, 151.215 -33.858, 151.215 -33...>

If you are not sure how to get those IDs, please check the related section on the previous article.

Conclusion

I hope this can be useful! OpenStreetMap is an amazing free and open repository of geographical data maintained by very enthusiastic volunteers, and creating applications that integrate with it enhances the whole ecosystem. And, combining all this greatness with the power and simplicity of Python and its environment is a huge win! Now, go build something awesome!