[Base] Would OvertureMaps be open to also storing in another format.

[地図の神様 (CraigglesO)]

Category Feedback

Request

Would OvertureMaps be open to or interested in also storing each dataset with all the properties intact with something a bit simpler and more accessible to parse like a very basic Protobuf schema or Shapefile?

Explanation

I'm extremely interested in OvertureMaps data. I've been studying it on and off for a while but never really dove in yet.

I've been hard at work on a project that makes as many GIS data formats accessible as possible and if desired also builds vector/raster/grid tiles for use via a renderer.

The project is dependency free and still manages to have a fairly comprehensive list of geospatial data formats it can parse. I know OvertureMaps has committed to geoparquet, and I understand the reasoning behind this for data analysis. But Parquet is an extremely complicated database with encoding and compression mechanics that make it hard to decode.

The creator of the geoparquet format himself admitted to this complexity and noted ports to other languages have actually failed. I also didn't like the shear complexity and size of the wasm build, nor is it actually that performant as I found a geojson replica of the example data to out-pace from parsing to rendering via the mapbox-gl renderer.

Dependency with other categories, if any.

No response

[Dana Bauer (danabauer)]

Hi 地図の神様 (@CraigglesO)! Thanks for reaching out and sharing your project. Right now Overture has no plans to release datasets as anything other than parquet. We do generate PMTiles every month, and you can use our Explorer tool to download a viewport's worth of data as geojson files. There are also a few data mirrors out there that are not maintained by Overture: our friends at Carto have made Overture data available in BigQuery, Snowflake, and Databricks; and the team at Fused hosts Overture data on their platform.

[Dana Bauer (danabauer)]

Our latest tiles are available at s3://overturemaps-tiles-us-west-2-beta/2025-02-19/ or https://overturemaps-tiles-us-west-2-beta.s3.us-west-2.amazonaws.com/2025-02-19/

[地図の神様 (CraigglesO)]

I appreciate the response!

That's a shame but I understand. Tiles aren't useful as it splits up the data.

Does OvertureMaps (docs, script, or otherwise) have a pipeline to parse full features of each schema after downloading an entire block of data?

For instance, I am interested in downloading the entire buildings dataset, then iterating over every feature. The docs aren't very clear or I'm navigating the docs wrong.

If this is the wrong place to ask this, nevermind.

[Matt Travis (mtravis)]

地図の神様 (@CraigglesO) you can extract all of the buildings data using this command.

aws s3 cp --no-sign-request s3://overturemaps-us-west-2/release/2025-02-19.0/theme=buildings/type=building/ . --recursive

This will download all the parquets to a local drive. The total size is 230GB

You can then use duckdb to loop over the parquet files and convert to a spatial format of your choice using the spatial plugin. That uses GDAL so you could export to pbf.

Here's a bash script I use that could easily be converted to python.

for f in *.parquet; 
do duckdb -c \
"install spatial; 
load spatial; 
COPY (SELECT id, st_geomfromwkb(geometry) from read_parquet('$f')) 
to '$f.fgb' WITH (FORMAT GDAL, DRIVER 'flatgeobuf');"; 
done

[地図の神様 (CraigglesO)]

Matt Travis (@mtravis) - appreciate it, that helped tremendously.