Tech debt: migrate specification fetch from GitHub-raw to S3/CDN across remaining repos

[tombrooks248]

Background

makerules main now fetches the specification CSVs from the environment's S3 datastore bucket (falling back to the prod CDN), instead of the old hard-coded raw.githubusercontent.com/digital-land/specification/main/... per-file curl. See makerules PR feat: specification-from-s3.

The task repos that pull makerules from main at runtime (collection-task, digital-land-builder-task) already pick this up automatically, and their vendored makerules/makerules.mk copies have been synced to match makefiles tidy-up ticket. config has been handled separately because it does not regenerate makerules at runtime — its committed copy is the one actually used.

Several other repos still carry the old GitHub-raw spec fetch in their makerules/makerules.mk.

Problem

Repos still on the old fetch:

• always pull spec from the specification repo's main branch, identically
  for every environment (no per-env isolation possible);
• depend on GitHub raw availability + rate limits at pipeline runtime;
• are inconsistent with the rest of the stack, which now fetches reference
  data (spec, organisation.csv) from S3/CDN.

Why this is worth doing

1. Reliability — removes a hard runtime dependency on GitHub raw; pulls
   from our own S3/CDN (faster, no rate limits).
2. Consistency — same mechanism the rest of the stack already uses for
   organisation.csv and config.
3. Unlocks per-environment spec testing — the S3 fetch is the consumer
   half of being able to test a spec change in DEV/Staging in isolation
   without merging to specification main (see follow-on below).

Scope

In scope

• Audit all repos (org-wide, not just locally checked out) with a
  vendored makerules/makerules.mk and identify which still use the old
  per-file GitHub-raw spec fetch.
• For each, sync the vendored makerules/makerules.mk to makerules main
  (or, for repos that regenerate it at runtime via make makerules, confirm
  no committed copy needs touching).
• Verify each migrated repo's CI / runtime still goes green (the new fetch
  uses the env bucket if COLLECTION_DATASET_BUCKET_NAME is set, else the
  prod CDN).

Known repos to start from (local scan — NOT exhaustive)

• config (in progress under the makefiles tidy-up)
• digital-land-python — assess: library, confirm whether it actually
  runs the spec target at runtime before changing
• specification — special case: this is the spec source repo and
  builds spec from its own specification.mk; the makerules.mk fetch
  block is likely inert here. Confirm before touching / probably exclude.
• (org-wide audit will add the remaining task/collection repos)

Out of scope (tracked as follow-on, see below)

• Building the per-environment spec publish workflow.

Acceptance criteria

• An audit list exists of every repo using the old spec fetch.
• Each in-scope repo's vendored makerules.mk matches makerules main
  (0 diff) OR is confirmed to be runtime-regenerated and needs no change.
• CI green on each migrated repo.
• No repo still references
  raw.githubusercontent.com/digital-land/specification/main/... for the
  spec fetch.

Follow-on (separate ticket): per-environment spec publish

The S3 fetch is only the consumer half. To actually test spec changes in
DEV/Staging in isolation, the specification repo needs a publish
workflow that aws s3 syncs the built spec CSVs to a chosen environment's
<bucket>/specification/ prefix — a direct copy of config's existing
per-env S3 sync pattern (its evening pipeline's detect-environments +
upload-s3 matrix). Until that exists, the benefit here is reliability +
consistency, with per-env testing latent but enabled.

Notes / caveats

• Per-env isolation only applies to tasks that have
  COLLECTION_DATASET_BUCKET_NAME set; the else fallback is the prod
  CDN, so unset = prod spec.
• A per-env spec is a partial fork (pipeline's view only) — docs site and
  the public spec API still reflect specification main.

Glossary

• Vendored — a copy of a file that really lives in another repo
  (here, makerules.mk from the makerules repo) checked into this repo as
  well, so it has its own committed copy rather than pulling it live.