Visualise cogs geoparquet post force push

[nf-s]

Replaces

• Visualise cogs geoparquet #35

After force push to main

[willjnz]

This example shows how to visualise many COGs using a client-side mosaic https://github.com/developmentseed/deck.gl-raster/tree/main/examples/naip-mosaic
https://github.com/developmentseed/deck.gl-raster/blob/main/examples/naip-mosaic/src/App.tsx
https://developmentseed.org/deck.gl-raster/api/deck-gl-geotiff/classes/MosaicLayer/

[willjnz]

We have 7 datasets:
https://csdr.dev.oceandevelopmentdata.org/console/dataset

• 5 are STAC-Geoparquet (GMW v3 and v4, ACE, DEP Seagrass, DEP Mangrove). These can be visualised as a mosaic with https://www.npmjs.com/package/@developmentseed/deck.gl-raster.

• 2 are Parquet (ACA Reef, VIDA Buildings). VIDA Buildings have a PMTiles file so can be visualised. ACA Reef does not?

I am no longer changing to write in arrow.

ACA Reef is the last one to plan. It is 500MB. Should the pipeline write to PMTiles or is there another way?

[willjnz]

@nf-s @alexgleith visualising COGs is working well. Here is the DEP Seagrass data straight from S3 (no band or arrow changes needed):

This covers 5/7 of the datasets. I need to debug why ACE COG is not displaying (it can't be loaded into QGIS either).

The remaining 2 are vector/parquet.

• VIDA Buildings: polygon. Visualising the preexisting PMTiles (hosted by SourceCoop) works fine (despite 240GB file size!). I need to add PMTiles URL as an optional attribute to a dataset.
• ACA Reef: polygons. Doesn't have PMTiles. There are 2 options for this:

1. Create PMTiles in pipeline. Requires customisation and duplication of data.
2. Read existing (partitioned) parquets with duckdb-wasm to filter for map viewport and use range requests. I am not sure how performant this will be. It means we can use the data as-is. This makes the visualisation more dynamic and less standardised.

What do you think?

[alexgleith]

why ACE COG is not displaying

This is a big file, right, but only has small regions of data. You sure it doesn't work in QGIS?

[willjnz]

ACE for one year is only 100MB. The examples show >1GB COGs loading. I will figure out what else could be blocking it.

[alexgleith]

Regarding ACA, I vote for:

Create PMTiles in pipeline.

I think we already have the code to make them, and they're most performant... shame about duplication, but let's do it anyway...

THoughts @nf-s?

[willjnz]

ACE issue was CORs. Able to get around it with a custom proxy route.

[willjnz]

@nf-s Related to the above, I would have to add a PMTiles URL field to the dataset run (just like the geometries runs already have).

[Copilot]

Pull request overview

Adds dataset-run map visualization support for GeoParquet/STAC-GeoParquet (PMTiles + COG mosaics), including schema + DB plumbing for a PMTiles URL.

Changes:

• Add Deck.gl + GeoTIFF/COG rendering and Parquet (Arrow/WASM) parsing dependencies to the web app.
• Introduce a new DatasetRunMap client UI for visualizing PMTiles (GeoParquet) and COG mosaics (STAC-GeoParquet).
• Add dataPmtilesUrl to dataset-run schemas, DB table, API query selection, and a Drizzle migration.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Locks new visualization/parquet parsing dependencies and their transitive graph.
packages/schemas/src/crud.ts	Extends dataset-run response schema with dataPmtilesUrl.
apps/web/package.json	Adds Deck.gl, deck.gl-geotiff, apache-arrow, parquet-wasm dependencies.
apps/web/next.config.mjs	Enables webpack top-level await + async WASM to support parquet/geotiff deps.
apps/web/app/console/dataset/run/[datasetRunId]/client.tsx	Renders the new dataset-run map in the dataset run detail page.
apps/web/app/console/dataset/_components/dataset-run-map.tsx	New map component: PMTiles rendering + STAC-Geoparquet → COG mosaic visualization.
apps/server/src/schemas/db.ts	Adds data_pmtiles_url column to dataset_run.
apps/server/src/routes/datasetRun.ts	Includes dataPmtilesUrl in dataset-run query column selection.
apps/server/drizzle/meta/_journal.json	Registers the new migration entry.
apps/server/drizzle/0034_lowly_wolf_cub.sql	Adds the data_pmtiles_url column via SQL migration.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nf-s]

Looks great, thanks Will!

Do you have plan for making the symbology and legends configurable? As currently they seem to be hard coded. I think these should be attached to the dataset model (or perhaps even the dataset run, not sure)

Also, can you please remove uses of as any or other typecasting unless its really needed?

[willjnz]

@nf-s thank you.

Right now I have hardcoded the colour maps for the known datasets but have helpful fallbacks so any dataset will visualise.
Currently the user sees the available assets and years for STAC-Geoparquet datasets. I could allow them to select which of these they want to visualise and then show them the values that they could map to colours and labels. This could then be saved to the dataset run. Similar could be done for PMTiles although that data is simpler.

@alexgleith what do you think? How much customisation is wanted?

I will clean up the types.

[alexgleith]

How much customisation is wanted?

I think if we can capture the configuration into the dataset model, that's probably a good idea. Just needs to be as simple as possible... it would be better if there wasn't a huge form with every option. Just something minimal, somehow...

[willjnz]

Ok I can add this as a field to the dataset table. Then when users add a new dataset with this info, the UI will be straight forward.

We need:

1. Asset name (“mangroves” for GMW). Some datasets have many assets so we need to narrow it down.
2. Mapping of asset values to colours and labels (GMW would be {1: {“label”: “mangroves”, “color”: “#000”}} (we only have categorical data in COGs so far).

PMTiles could just have a label and a color. There are no categories for the datasets we currently have.

I will make this visualisation field nullable and the styling logic have nice fallbacks (if null or errors). I will keep just showing the last year of data but letting users know the available years.

[willjnz]

I have updated the map to be styled from the dataset table. The user can update and see these in the UI. Here are the styles I will add to the prod datasets, once the DB is migrated. Any new dataset falls back to a safe default style if this field is null.

GMW v3 and v4
{"asset":"mangrove","type":"raster","display":"categorical","values":{"1":{"color":"rgba(46, 139, 87, 1)","label":"Mangrove"}}}

DEP Seagrass
{"asset":"seagrass","type":"raster","display":"categorical","values":{"1":{"color":"rgba(0, 190, 196, 1)","label":"Seagrass"}}}

ACE
{"asset":"classification","type":"raster","display":"categorical","values":{"2":{"label":"Intertidal","color":"rgba(148, 65, 14, 1)"},"3":{"label":"Mangrove","color":"rgba(41, 223, 58, 1)"},"4":{"label":"Saltmarsh","color":"rgba(219, 228, 54, 1)"},"5":{"label":"Seagrass","color":"rgba(14, 131, 78, 1)"}}}

DEP Mangrove
{"asset":"mangroves","type":"raster","display":"categorical","values":{ "1": { "color": "rgba(86, 173, 60, 1)", "label": "Mangrove (Open)" }, "2": { "color": "rgba(46, 139, 87, 1)", "label": "Mangrove (Closed)" }}}

ACA Reef
{"type":"vector-polygon","display":"simple","color": "rgba(209, 255, 93, 1)", "label": "Reef"}

VIDA Buildings
{"type":"vector-polygon","display":"simple","color": "rgba(142, 78, 32, 1)", "label": "Buildings"}

[nf-s]

Looks great! ✅