[EPAC-1917]: publish statistics artifacts to S3

[riddim-developer-bot]

Why

EPAC-1917 migrates the eight backend statistics pipelines off local/stdout-only publishing paths so they can emit CDN-ready S3 artifacts as part of the Aurora retirement project.

What changed

• Added a shared Python statistics artifact publisher that writes statistics/v1/<pipeline-name>/<dataset>.json objects with content-hash-sha256 metadata.
• Wired CPI, Fiscal Monitor, CPP/OAS, EI, VAC, student finance, corrections, and transport safety pipelines to --s3-publish while preserving existing stdout/file output modes.
• Added fake-S3 unit coverage for each pipeline's publish path and enabled the existing CPP/OAS unittest suite in CI.
• Added each statistics pipeline to publish-artifacts.yml with monthly CPI/Fiscal Monitor cadence and quarterly cadence for the other six pipelines.
• Documented pipeline schedules/artifact keys in READMEs and updated the use-case catalog/S3 migration plan.

Trade-offs not taken

• Did not delete or change backend/loader; inspection shows it handles members, speeches, and PBO only, not these eight statistics pipelines. Loader removal remains downstream as stated in the issue.
• Kept the existing full all.json payloads and added slice artifacts where current payloads expose national/province datasets, avoiding a consumer-breaking schema rewrite in this PR.
• Did not run live S3 publishing locally; local verification uses fake S3 clients and workflow validation because the actual publish path is credentialed GitHub Actions infrastructure.

Test plan

• for pipeline in cpi-statistics fiscal-monitor cpp-oas-statistics ei-statistics vac-statistics student-finance-statistics corrections-statistics transport-safety-statistics; do (cd "backend/$pipeline" && python3 -m unittest) || exit 1; done
• actionlint .github/workflows/publish-artifacts.yml .github/workflows/backend-python.yml
• scripts/check-boundaries.sh

External validation still required after deploy: human verifies the relevant iOS statistics dashboards render the most recent reference period correctly.

Reviewer-Boundary: review-only

Resolves EPAC-1917

[ghost]

ReviewAutonomousPR

• Verdict: approve
• Reviewer boundary: review_only
• Acceptance criteria coverage: covered=5, missing=0, unclear=1

Summary

PR is acceptable as-is. The eight statistics pipelines have S3 publish paths, workflow integration, docs, and focused fake-S3 tests. Local verification passed for all changed Python unittest suites, actionlint on both changed workflows, compileall on changed Python paths, and scripts/check-boundaries.sh.

Actionable findings

• None.

Acceptance criteria coverage

• covered — Each of the eight pipelines, when run with a new --s3-publish flag (or env-var equivalent), writes its output to statistics/v1//.json in the artifacts bucket instead of stdout / Postgres.
  • Each pipeline exposes --s3-publish/--s3-bucket and uses the shared statistics_artifacts publisher for statistics/v1 keys with S3 put_object metadata.
  • Actionability: none
  • Evidence: Reviewed backend/statistics_artifacts.py plus all eight pipeline main/publish_payload functions; unittest fake-S3 coverage passed for all eight pipeline suites.
• covered — All eight are added as steps to .github/workflows/publish-artifacts.yml (per-pipeline cadence: monthly for CPI / fiscal-monitor; quarterly for others; doc the schedule in each pipeline README).
  • publish-artifacts.yml includes all eight pipeline steps, with monthly gating for CPI/Fiscal Monitor and quarterly gating for the other six; README schedule sections were added.
  • Actionability: none
  • Evidence: actionlint passed for .github/workflows/publish-artifacts.yml and .github/workflows/backend-python.yml.
• covered — The loader Lambda's responsibilities for these eight pipelines are removed (or the Lambda is deleted entirely if no other pipelines use it — verify in PR description).
  • Loader inspection shows it still handles members, speeches, and PBO, not these eight statistics pipelines; the PR description documents that loader deletion is downstream.
  • Actionability: none
  • Evidence: Reviewed backend/loader/main.go and repo references for these statistics pipelines.
• covered — Each pipeline's test_*.py continues to pass; new tests cover the S3-publish path (fixture S3 client; assert written object key + content-hash).
  • All eight pipeline unittest suites passed locally from a clean archive of the PR head; each contains fake-S3 assertions for object keys and content-hash metadata.
  • Actionability: none
  • Evidence: Command passed: for pipeline in cpi-statistics fiscal-monitor cpp-oas-statistics ei-statistics vac-statistics student-finance-statistics corrections-statistics transport-safety-statistics; do (cd backend/$pipeline && python3 -m unittest) || exit 1; done
• covered — Aurora statistics_* tables are dropped after a one-week observation window (post-cutover) — tracked in the Aurora teardown issue, not here.
  • No table-drop work was added here, matching the stated out-of-scope/deferred teardown requirement.
  • Actionability: none
  • Evidence: Diff review found no Aurora teardown migration or loader deletion in this PR.
• unclear — Human verifies each statistics dashboard in the iOS app still renders correct values for the most-recent reference period after deploy.
  • This is explicitly a post-deploy visual QA gate and should not block merge of the code PR.
  • Actionability: external_gate
  • Evidence: External validation gate is listed in the issue and PR body.
  • Suggested follow-up: Post-deploy visual QA for migrated statistics dashboards