Skip to content

Spatial Postgres backend - #15

Merged
Solcany merged 134 commits into
mainfrom
staging
Jul 23, 2026
Merged

Spatial Postgres backend#15
Solcany merged 134 commits into
mainfrom
staging

Conversation

@Solcany

@Solcany Solcany commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator
  1. Spatial Postgres backend — the data layer moved to Postgres + PostGIS (replacing the earlier binary-file approach)
    and now ships as a custom image bundling the pg_roaringbitmap extension.

  2. Roaring-bitmap precompute rollup — heatmap and histogram counts are precomputed into a cell_features index at
    rebuild time, cutting per-request query time from ~13s to under a second while staying exact at any resolution or
    filter.

  3. Multi-dataset Linked-Data ingestion — ingestion loads streets, neighbourhoods, districts, and addresses from
    Adamlink TTL, plus the Beeldbank, Joods Monument, and Delpher record datasets, each linked to canonical places.

  4. Point/line/polygon place geometry — places gained a dedicated geometry model supporting lines and polygons (not
    just points), with historical names and validity dates.

  5. Spatially accurate heatmap — lines and polygons are rasterized into grid cells via PostGIS, and cells are
    reprojected from the Dutch RD grid to WGS84 with proj4 so they're square and grid-aligned.

  6. Interactive filtering and browsing — the map gained filtering by dataset, record type, and geometry/place type, a
    time slider, and a side panel for browsing a cell's features.

  7. Production deployment pipeline — Dockerized self-hosted and existing-Postgres setups, CI workflows that build and
    publish images to GHCR, and schema/ETL run from the image itself.

  8. Non-blocking frontend rendering — heatmap and histogram are fetched client-side and cancellably so the map shell
    renders instantly, with sparse-heatmap rendering to keep payloads small.

  9. Log-scale density normalization — heatmap color and histogram height are driven by spatial and temporal
    frequencies normalized on logarithmic scales, so relative density reads correctly across the city and timeline.

  10. Schema consolidation — datasets and organisations tables, an Entity JSONB column, and URI-namespaced feature IDs
    for cleaner, deduplicated data.

  11. Caching and connection resilience — a TTL query cache and pooled, timeout-guarded database connections.

  12. Dynamic timeline — time slices are generated from the data's actual date range at index time rather than
    hardcoded.

  13. Basemap provider switch — tiles moved from MapTiler to OpenFreeMap, dropping the API-key dependency.

  14. Documentation — a README (architecture, ingestion, indexing, deployment) plus an mdsvex-rendered
    About page, LICENSE, and CITATION.

Solcany added 29 commits June 22, 2026 18:29
Split Place geometry into its own table
use full adamlink uri for linked point IDs
…equest place_cells join up into roaring bitmap buckets at rebuild-index time
…-vs-display units

  silence melt state_referenced_locally warnings and re-seed Pagination when the dataset changes
  port the Toaster to next-melt's runes builder with a bottom-right fade-only layout
  bump db and app dependencies to their latest minor/patch versions
  upgrade maplibre-gl to v6 (namespace import for the now-ESM-only build)
@Solcany
Solcany merged commit c105210 into main Jul 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant