diff --git a/docs/architecture-and-infrastructure/data-pipeline-architecture/Key-Concepts/Provision-Quality.md b/docs/architecture-and-infrastructure/data-pipeline-architecture/Key-Concepts/Provision-Quality.md new file mode 100644 index 0000000..208cc73 --- /dev/null +++ b/docs/architecture-and-infrastructure/data-pipeline-architecture/Key-Concepts/Provision-Quality.md @@ -0,0 +1,149 @@ +--- +title: Provision Quality +--- + +## What is provision quality? + +**Provision quality** describes *who provides data for each dataset on the platform, and how good that data is*. It answers questions like: how many organisations feed a dataset, are they the authoritative source for their data or not, and which datasets or organisations have gaps. + +It is produced by a nightly pipeline task (`provision-quality`) that writes three related tables as [Delta Lake](https://delta.io/) tables (Parquet) into the platform's `parquet-datasets` S3 bucket — the same output pattern as the [task table](./Tasks.md). From there they are surfaced to the serving database so the [digital-land.info](https://www.planning.data.gov.uk) front-end can show a provider/quality breakdown on each dataset page. + + +## Two lenses: providers and owners + +The core subtlety is that an organisation can relate to a dataset in **two different ways**, and they don't always line up: + +- **Provider** — an organisation that *submits* data. It has an active **endpoint** feeding the dataset (a non-empty `endpoint` in `source` / `source_pipeline`). *"Who feeds data in."* +- **Owner** — an organisation that *owns entities* on the platform, recorded by the `organisation_entity` field on each entity. *"Who owns the records that are actually there."* + +These usually overlap, but the mismatches are exactly what this work makes visible. For example, when a national body does a bulk upload ("seeding") of conservation areas on behalf of many councils: + +- the **provider** is the national body (it holds the endpoint), but +- the **owners** are the individual councils (the entities are attributed to them). + +Conversely, a council may **own** entities (loaded historically or seeded on its behalf) while no longer **submitting** anything itself. + +## Authoritative vs some + +Within a dataset, each entity carries a `quality` value that tells us whether the data came from the right place: + +- **`authoritative`** — the data is from the organisation that is the authoritative source for it (a council's own conservation-area data, owned by that council). +- **`some`** — the data is present but *not* from the authoritative source (for example, seeded by another body). + +An organisation's provision quality for a dataset is `authoritative` if it owns at least one authoritative entity, otherwise `some`. + +## The three tables + +Everything derives from a single base grain — **provision** = one `(dataset, organisation)` pair — which is then rolled up two ways. + +### 1. `provision_quality` (base) + +One row per `(dataset, organisation)`. A row exists for **any organisation that has an active endpoint *or* owns entities** for the dataset — i.e. both lenses combined. Nothing is dropped; flag columns record what each row *is*, so downstream consumers can filter (e.g. to only formally-designated providers) as needed. + +| Column | Description | +|---|---| +| `dataset` | The dataset (e.g. `conservation-area`) | +| `organisation` | The organisation reference (e.g. `local-authority:ADU`) | +| `organisation_name` | Human-readable organisation name | +| `has_active_endpoint` | `true` if the org submits data via a non-empty endpoint (provider lens) | +| `owns_entities` | `true` if the org owns at least one entity (owner lens) | +| `is_designated_provider` | `true` if the org has an allocated entity range in `entity-organisation.csv` for this dataset — i.e. this `(dataset, org)` is a *formally designated provision* | +| `quality` | `authoritative` or `some` | +| `entity_count` | Number of entities owned by the org (or seeded on its behalf, for seeders with no directly-owned entities) | +| `quality_score` | *Reserved — empty in v1* | + +The `is_designated_provider` flag is what distinguishes formally-provisioned data from data that is simply *present* in the system (Owen's "endpoints/data that doesn't belong to a provision"). Both are retained; the flag lets the front-end include or exclude the latter without recomputation. + +### 2. `dataset_quality` (rollup per dataset) + +One row per dataset, aggregated from `provision_quality`. + +| Column | Description | +|---|---| +| `dataset` | The dataset | +| `authoritative_organisations` | Count of orgs classified `authoritative` for this dataset | +| `some_organisations` | Count of orgs classified `some` | +| `total_organisations` | Total contributing organisations | +| `total_entities` | Total entities in the dataset | +| `quality_score` | *Reserved — empty in v1* | + +This is the natural extension of the existing "N data providers" figure shown on dataset pages today (the `publisher_count` on `dataset_publication`). The provider-vs-owner split is available by filtering `provision_quality` on the flag columns; `dataset_quality` gives the headline counts. + +### 3. `organisation_quality` (rollup per organisation) + +One row per organisation, aggregated from `provision_quality` **across all datasets**. This view did not exist in the original analysis. + +| Column | Description | +|---|---| +| `organisation` | The organisation reference | +| `organisation_name` | Human-readable organisation name | +| `authoritative_datasets` | Number of datasets the org provides authoritatively | +| `some_datasets` | Number of datasets the org provides at `some` quality | +| `total_datasets` | Total datasets the org contributes to | +| `total_entities_owned` | Total entities owned across all datasets | +| `quality_score` | *Reserved — empty in v1* | + +## How the classification works + +The classification logic is reused from the original Data Design analysis. In outline, per dataset: + +1. **Providers** are the organisations with a non-empty endpoint in `source` ⋈ `source_pipeline`. +2. **Owners and quality** come from each dataset's `entity` table — `organisation_entity` (who owns each entity) and `quality` (`authoritative` / `some`). +3. **Designated provisions** come from `entity-organisation.csv` — the organisations allocated an entity range for the dataset. +4. **Seeder detection** uses `lookup.csv`: an active organisation that seeded entities it does not own (and is not the designated provider) is credited as a `some`-quality contributor. To avoid crediting dissolved/reorganised bodies with stale lookup entries, local-authority-type organisations must have seeded entities for **more than one** owner to count; other org types need at least one. +5. **Active organisations only** — organisations with an `end_date` in the `organisation` table are excluded from `some`-quality seeder classification. + +> **Deviation from the original analysis (deliberate):** the prototype *dropped* two categories of row — designated providers whose data has stalled (`some`-only with an allocated range), and `some` providers with zero seeded entities. In line with the "keep all data, flag it" approach, `provision_quality` **retains** these rows and distinguishes them via the flag columns and `entity_count`, rather than removing them. Consumers filter as needed. + +## Where it runs + +`provision-quality` is a new task in the [build-digital-land-builder DAG](https://github.com/digital-land/airflow-dags/blob/main/dags/digital_land_builder.py), running nightly. + +- **It runs in parallel with `build-digital-land-builder`.** All of its inputs live in the `collection-data` S3 bucket (see [Data sources](#data-sources)), independent of the built `digital-land.sqlite3`, so it does not need to wait for the build. Like `assemble-tasks`, it can hang off `configure-dag` and run alongside both the build and the `digital-land-postgres-loader`. This keeps it off the critical path to the 8am publish deadline. +- **The single task writes both outputs itself** — the Delta tables to S3 *and* the rows to serving Postgres — rather than splitting the Postgres load into a separate DAG step. Everything the pipeline does for these tables lives in one place, so it can be followed through end to end. +- It is implemented in **Python** using **[Polars](https://pola.rs/)**. At this data scale plain Python is sufficient; +- The CloudFront cache **must be invalidated after this task completes** + +### Data sources + +The productionised task reads everything from the **`collection-data` S3 bucket** — it does **not** hit Datasette or GitHub-raw at runtime (avoiding both the Datasette load and the GitHub rate-limiting seen in overnight runs), and it does **not** depend on the built `digital-land.sqlite3`. + +| Data | Source | +|---|---| +| Provider mapping — `endpoint`, `organisation`, dataset(s), `end_date` | Each collection's `collection/source.csv`, globbed across `s3://{env}-collection-data/*-collection/collection/source.csv`. The `pipelines` column carries the dataset(s) an endpoint serves — the same information as Datasette's `source` ⋈ `source_pipeline`. | +| `organisation` — names, entity IDs, active/inactive (`end_date`) | `s3://{env}-collection-data/organisation-collection/dataset/organisation.csv` | +| `entity` (with `organisation_entity`, `quality`), per dataset | The per-dataset `{dataset}.sqlite3` in `s3://{env}-collection-data/{collection}-collection/dataset/` | +| `entity-organisation.csv`, `lookup.csv`, per collection | `s3://{env}-collection-data/config/pipeline/{collection}/` | + +Because all of these are per-collection outputs already present in `collection-data` when this DAG runs, the task has no dependency on the build step — hence it can run in parallel with it. + +> **Entity source — sqlite for v1, Delta/parquet later.** The entity + quality read is the task's dominant cost (the big datasets — title-boundary, listed-building — have millions of entities), and the per-dataset sqlites carry geometry inline, so the whole file is downloaded even though only three columns are needed. A columnar entity table would be far cheaper. For v1 we read the **sqlites**, because they are the only source that currently exists for every dataset — the `parquet-datasets` bucket holds entity tables for just the handful of datasets already migrated to the PySpark pipeline. Once the collection pipelines are fully on PySpark and emit per-dataset Delta tables (`dataset=/`-partitioned), the entity read switches to those. To keep that switch cheap, the entity read is isolated behind a single loader function so it can be swapped without touching the classification logic. + +### Output + +The task writes **both** of its outputs itself, in the one task: + +1. **Delta Lake tables (Parquet)** to the platform's Parquet datasets bucket — `s3://{env}-parquet-datasets/{table}/`, one each for `provision_quality`, `dataset_quality`, and `organisation_quality`. This mirrors how the [task table](./Tasks.md) is produced by the `assemble-tasks` pipeline: the Delta table is the canonical, re-queryable artifact. This is the primary output. +2. **Serving Postgres** — the same three tables are written to the serving database, alongside `dataset_publication`, so [digital-land.info](https://www.planning.data.gov.uk) can serve them on dataset pages. The `total_organisations` figure in `dataset_quality` should reconcile with the existing `publisher_count` on `dataset_publication`. + +Both writes happen inside the `provision-quality` task rather than being split across separate DAG steps, so everything the pipeline does for these tables can be read in one place. + +### Cache invalidation + +The dataset pages that serve these tables sit behind CloudFront, so the cache must be invalidated once the new data is written. So invalidate-cloudfront-cache task will run immediatly after provision-quality task to ensure this data is available even if it takes longer than build-digital-land-builder + +## Worked example: conservation area + +Using the seeding scenario above, three representative rows in `provision_quality` for `conservation-area`: + +| organisation | has_active_endpoint | owns_entities | is_designated_provider | quality | entity_count | +|---|---|---|---|---|---| +| Adur DC | ✅ | ✅ | ✅ | authoritative | 50 | +| Lewes DC | ❌ | ✅ | ✅ | some | 30 | +| Historic England | ✅ | ❌ | ❌ | some | 3 | + +- **Adur** submits its own data and owns authoritative entities — the healthy case. +- **Lewes** never submitted, but owns 30 entities seeded on its behalf, so `some` quality. +- **Historic England** holds the endpoint and seeded data, but owns no entities itself — a provider that is not an owner. + +`dataset_quality` for `conservation-area` then rolls these (and all other orgs) up into authoritative / some / total organisation counts; `organisation_quality` rolls the same rows up per organisation across every dataset each org touches.