Load Spec into S3 - Async uses S3 instead

[Ben-Hodgkiss]

Initial tech idea. In tasks.py the before_task() celery function could look at the latest commit in Specification Repo, if this is not the same as what it currently stores, it updates the specification files stored.

This would be the same process as what Make Init currently does, but saves the pain of having to reboot the async service to pull in new specification files.

[pooleycodes]

Specification Freshness — Implementation Plan

Status: Approach updated after discussion with Owen and Joseph — see revised approach below.

Problem

Specification CSV files (dataset-field.csv, field.csv, issue-type.csv, etc.) are
downloaded from GitHub at Docker image build time via make init → make specification.
They are never updated while the container is running, so a long-lived worker will use
stale specs as the upstream digital-land/specification repository evolves.

Revised Approach — S3-backed Specification

Rather than polling GitHub from within the worker, move the spec files to S3 and update
the Specification class to read from S3 directly via cloudpathlib. This keeps the
spec source-of-truth separate from the image and leverages authentication the worker
already has.

Upstream spec change
  └─ CI/CD sync job (Ernest)
       └─ pushes latest CSVs → s3://spec-bucket/specification/
            └─ worker reads via cloudpathlib (UCPath)
                 └─ local cache hit on subsequent tasks → no extra latency

Work Items

1. Update Specification class to accept an S3 path

Use cloudpathlib so the class can be pointed at an s3:// URI instead of a local
directory. cloudpathlib's UCPath caches files locally after the first download and
only re-fetches when the remote object changes, keeping per-task overhead low.

2. Populate and maintain the S3 spec bucket

A sync process (to be set up by Ernest) must push the latest spec CSVs from
digital-land/specification into the target S3 bucket whenever upstream changes.

3. Remove spec download from make init / Docker build

The make specification step in makerules/makerules.mk downloads CSVs at image build
time. Once the worker reads from S3 at runtime, this step can be removed — the image no
longer needs to bake in spec files.

4. Verify worker S3 authentication

The async worker already has IAM access to S3 (for REQUEST_FILES_BUCKET_NAME). Confirm
the same role/policy covers the spec bucket, or extend it accordingly.

Open Question

Will this add latency per task?

cloudpathlib caches files locally after the first fetch and only re-downloads on
change, so steady-state overhead should be minimal. Worth benchmarking in staging before
rolling out to production.