Missing record_id in buildings from Microsoft ML Buildings

[StrandedKitty]

I was going to rebuild our tileset from the latest Overture release (2025-04-23.0) for use at shadowmap.org, but I encountered a breaking regression while testing the data.

Some building features sourced from Microsoft ML Buildings are missing record_id values. For example, see https://explore.overturemaps.org/#20.85/39.985028/-3.7695136

I can't pinpoint exactly which release made record_id vanish, but right now we're using tiles generated from the 2025-02-19.0 release, and this building mentioned above has record_id=bb7e61a3d450.

So something must've broken in one of these releases:

• 2025-03-19.0
• 2025-03-19.1
• 2025-04-23.0 (latest)

In our particular case it's a breaking change because we rely on record_ids instead of ids for building identification to stay provider-agnostic (i.e. to be able to migrate away from Overture in the future).

[Jennings Anderson (jenningsanderson)]

Hi StrandedKitty, great question that actually highlights the value of id over record_id in this case (though I apologize for breaking your data pipeline, I hope you'll find this valuable):

The id for buildings is part of the Global Entity Reference System, which means Overture performs the necessary spatial conflation with each release to ensure that id remains as stable as possible for that building entity, regardless of the original data source.

Buildings sourced from the Microsoft ML Buildings Footprints dataset don't actually have an original ID that is meaningful. The record_id that was being populated before was part of our ingestion pipeline, which is why we removed it. For reference, the value bb7e61a3d450 was just the first 12 hex characters of xxhash64(geometry) (You could actually regenerate that based on geometry, if you need to, but please read on first).

By using the id field (GERS ID) instead, you will actually be more provider agnostic because the larger GERS ecosystem can use this id as a common foreign key across future datasets. We will have more examples in the coming months, but your provider agnostic scenario is exactly what we are designing for with our bridge files.

Unfortunately, in this particular case, since there is no meaningful original record id for Microsoft buildings, we do not populate this field nor produce a bridge file for Microsoft ML buildings. However, for your scenario, consider buildings from Esri Community Maps, which do contain a meaningful record_id:

You can always find the latest mapping of GERS ID <> Esri Community Maps ID in the Esri Community Maps Bridge File. This is even more valuable than relying on record_id in the release data because only about 50% of the buildings from Esri Community Maps actually make it into the Overture data release – the rest are conflated out because they overlap with a building from OpenStreetMap, which takes precedence in the conflation pipeline.

The Bridge file, however, contains all 29.9 M rows for GERS ID <> Esri Community Maps ID whereas there are only 14.6 buildings with an Esri Community Maps record_id in the actual Overture data release.

In this manner, Bridge files are Overture's method of sharing our conflation results for input datasets in the case where data doesn't make it all the way to release, which actually enables data consumers to be more provider agnostic.

[StrandedKitty]

Thanks, that's a great explanation. It feels like it should've been mentioned in the release notes.

Just FYI, our current tile generator creates custom IDs for Overture buildings based on dataset and record_id values, so we end up with IDs like google_13603052032910230 and ms_5d457c23c6af. I don't remember the specific rationale behind it, but most likely I just wanted something a bit more human-readable, and I wasn't sure whether your IDs persisted between releases.

Luckily, we don't have that many references to non-OSM buildings in our DB, so it should be reasonably easy to migrate to GERS IDs without building a complicated migration tool.